Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......................................XX..............] duration: 5.890201553s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:14:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.890231379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.615142625s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:14:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.615171681s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 313.112467ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:14:20 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 313.139899ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.788003105s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:14:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.788025847s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 326.810982ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:13:59 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 326.83611ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......................................XX.............] duration: 5.852830361s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:13:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.852862512s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 53.079032ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3425d218113d21b1 FAIL (2025-11-14 02:13:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59112->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.115603ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.72394253s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.723971345s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 77.453065ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3425d218113d21b1 FAIL (2025-11-14 02:13:25 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59112->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.484385ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 322.330208ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:13:18 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 322.35759ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.747153049s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:13:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.747187846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.......................................XX............] duration: 5.870620375s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:13:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.870650344s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 317.027633ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:12:43 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 317.052109ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 56.6643ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3425d218113d21b1 FAIL (2025-11-14 02:12:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59112->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.694085ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 230.300477ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3425d218113d21b1 FAIL (2025-11-14 02:12:17 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59112->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 230.322207ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.741663066s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.741690919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......................................XX...........] duration: 5.987383944s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:12:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.987661151s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 354.676023ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:12:02 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 354.7011ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......................................XX..........] duration: 6.203968836s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:11:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.203985777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.934747286s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:11:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.934776381s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 236.889806ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:11:32 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 236.916756ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......................................XX.........] duration: 5.983115456s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:11:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.983145763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.852830446s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:10:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.852917422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......................................XX........] duration: 5.970553337s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:10:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.970583693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.99775557s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:10:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.997856521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.826512824s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:09:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.826539456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......................................XX.......] duration: 5.957531639s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.957566005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.846447786s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:09:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.846478213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......................................XX......] duration: 6.21808886s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.218150667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.754269352s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.754298348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......................................XX.....] duration: 6.153181637s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.153223286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................................XX....] duration: 5.899846748s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.899874341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.671870037s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:07:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.67190279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................................XX...] duration: 6.041085457s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:07:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.041114172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................................XX..] duration: 6.209464564s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:07:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.209494111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.934729689s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:06:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.934757704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................................XX.] duration: 5.953150279s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.953169726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.909890367s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.909915674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.750068855s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:05:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.750098063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................................XX] duration: 30.25776742s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.257797938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......................................X] duration: 30.003262146s trace: https://jaeger.dev.storj.io/trace/68ff175ee56693e7 FAIL (2025-11-14 02:04:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Nov 14 02:04:03.568279553 (114h0m2.119750014s) [upload=100317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 02:04:03.822521908 (114h0m2.373992369s) [seg_pos={0 0}, upload=100317]: Begin upload segment... Nov 14 02:04:03.822580901 (114h0m2.374051362s) [seg_pos={0 0}, upload=100317]: Done begin upload segment. Nov 14 02:04:03.822610066 (114h0m2.374080527s) [noise=true , address=45.91.116.12:4227 , piece_id=CNK2TCAS4ME446IW7CQ7F6JLNVHMIO7ICDYBSVUQI52CV5MM5QDQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.822767496 (114h0m2.374237957s) [noise=true , address=45.91.116.8:4047 , piece_id=JNEDNBE5SB4XFRYYH6AMGJTGBHMYC2H2UHOKES23GBWBJSR53MPQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.823803789 (114h0m2.37527425s) [noise=true , address=45.91.116.11:4119 , piece_id=7GWSC3VZO3AAQGM4BGWWFMML2KO4ZOIGTS6V2NOMQQGIWQLG3N2A, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.824264676 (114h0m2.375735148s) [noise=true , address=145.79.227.6:4258 , piece_id=2F7TH3DXYJ3W3MU2RXLT6AFVLVDPNVW56YA5EYCFCSLBJFJOS3EA, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.825076192 (114h0m2.376546653s) [noise=true , address=145.79.227.23:4547 , piece_id=WR4BK7NV4PHL3MKN4ZUNQRWJCIEPY4FBTB46JYHHOGYY6ZK2JLIQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.825571776 (114h0m2.377042237s) [noise=true , address=145.79.227.9:4366 , piece_id=NVYTNXTTRZEGLOAEHFF266AWF5AYWPTYEQVTJOJ64XPI2OW3RC7Q, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.826080063 (114h0m2.377550524s) [noise=true , address=145.79.227.11:4732 , piece_id=2ZJRRRXCTERAIT4LTGPWHS767NCMXSRP72GI6SECIB4YQFUWCBHQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.828205712 (114h0m2.379676173s) [noise=true , address=45.91.116.7:4011 , piece_id=YQ7VTDBUUB2L3MYUDDH7C6JO72WBPIKEW4JVT247LZHYG5HEXAMQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.828491526 (114h0m2.379961987s) [noise=true , address=145.79.227.10:4401 , piece_id=RK4XNTHVLHDI64XCSPNYVFQMWLGJME2VS7DPVCGBKFEHE2MJ6O4A, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.829662615 (114h0m2.381133076s) [noise=true , address=145.79.227.10:4403 , piece_id=SX536OC66HDHHJTSPKI3QEHBXLUDV6AKPSK3WO6LY6FJ7KNFD72Q, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.832947791 (114h0m2.384418252s) [noise=true , address=145.79.227.20:4438 , piece_id=XDNCWWYBNQK4WUXMUCS5TV4KALX4OF76XHWTJNIFIRSB6RRABS3A, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.836454337 (114h0m2.387924799s) [noise=true , address=145.79.227.15:4875 , piece_id=XIZPGQAKGE36WFGQPMB24SDXG6YCGN7JR2U2KFDQ54ULLFLPLDEQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.840800845 (114h0m2.392271306s) [noise=true , address=145.79.227.8:4331 , piece_id=PCZCMGASPSPISTJIQBQLMW3ESQVAGYIH5AH72IUX7VUTDFSKLRIA, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.842962601 (114h0m2.394433052s) [noise=true , address=145.79.227.24:4583 , piece_id=7ZINC4COQC27RVVYNLWUD5YBWEDMCEUJ7GI6UH5TCGJRUAIMWQMA, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.852338695 (114h0m2.403809186s) [noise=true , address=77.90.31.21:4878 , piece_id=536BAKNEURXEAOVUQ2WJSXYK3XRVEEDQH5FS3NMRNOSXSGN6WCQQ, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:03.854679713 (114h0m2.406150164s) [noise=true , address=45.91.116.9:4263 , piece_id=ODHGOX5UXEL6VRBUXJ3LJKEER7FSELGHIYL7G2UVAUGEEOAYLHRA, seg_pos={0 0}, upload=100317]: Uploading piece... Nov 14 02:04:05.585133930 (114h0m4.136604391s) [upload=100317]: Next returned nil segment Nov 14 02:04:05.585170010 (114h0m4.136640471s) [upload=100317]: Waiting for error group managing segments... Nov 14 02:04:05.914416828 (114h0m4.465887279s) [noise=true , address=45.91.116.8:4047 , piece_id=JNEDNBE5SB4XFRYYH6AMGJTGBHMYC2H2UHOKES23GBWBJSR53MPQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:05.965166604 (114h0m4.516637065s) [noise=true , address=45.91.116.7:4011 , piece_id=YQ7VTDBUUB2L3MYUDDH7C6JO72WBPIKEW4JVT247LZHYG5HEXAMQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:05.982612283 (114h0m4.534082735s) [noise=true , address=45.91.116.11:4119 , piece_id=7GWSC3VZO3AAQGM4BGWWFMML2KO4ZOIGTS6V2NOMQQGIWQLG3N2A, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:05.989472708 (114h0m4.540943169s) [noise=true , address=45.91.116.9:4263 , piece_id=ODHGOX5UXEL6VRBUXJ3LJKEER7FSELGHIYL7G2UVAUGEEOAYLHRA, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:06.039050463 (114h0m4.590520934s) [noise=true , address=77.90.31.21:4878 , piece_id=536BAKNEURXEAOVUQ2WJSXYK3XRVEEDQH5FS3NMRNOSXSGN6WCQQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:06.046771906 (114h0m4.598242357s) [noise=true , address=45.91.116.12:4227 , piece_id=CNK2TCAS4ME446IW7CQ7F6JLNVHMIO7ICDYBSVUQI52CV5MM5QDQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:09.582281157 (114h0m8.133751619s) [noise=true , address=145.79.227.23:4547 , piece_id=WR4BK7NV4PHL3MKN4ZUNQRWJCIEPY4FBTB46JYHHOGYY6ZK2JLIQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:09.828230171 (114h0m8.379700632s) [noise=true , address=145.79.227.9:4366 , piece_id=NVYTNXTTRZEGLOAEHFF266AWF5AYWPTYEQVTJOJ64XPI2OW3RC7Q, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:11.080947688 (114h0m9.632418149s) [noise=true , address=145.79.227.10:4403 , piece_id=SX536OC66HDHHJTSPKI3QEHBXLUDV6AKPSK3WO6LY6FJ7KNFD72Q, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:11.781981298 (114h0m10.333451759s) [noise=true , address=145.79.227.20:4438 , piece_id=XDNCWWYBNQK4WUXMUCS5TV4KALX4OF76XHWTJNIFIRSB6RRABS3A, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:14.390382332 (114h0m12.941852793s) [noise=true , address=145.79.227.6:4258 , piece_id=2F7TH3DXYJ3W3MU2RXLT6AFVLVDPNVW56YA5EYCFCSLBJFJOS3EA, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:18.804898995 (114h0m17.356369456s) [noise=true , address=145.79.227.10:4401 , piece_id=RK4XNTHVLHDI64XCSPNYVFQMWLGJME2VS7DPVCGBKFEHE2MJ6O4A, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:19.561335070 (114h0m18.112805531s) [noise=true , address=145.79.227.24:4583 , piece_id=7ZINC4COQC27RVVYNLWUD5YBWEDMCEUJ7GI6UH5TCGJRUAIMWQMA, seg_pos={0 0}, upload=100317]: Done uploading piece. err: <nil> Nov 14 02:04:19.561355689 (114h0m18.11282615s) [seg_pos={0 0}, upload=100317]: Segment reached optimal threshold of 13 pieces. Nov 14 02:04:19.561606276 (114h0m18.113076737s) [noise=true , address=145.79.227.15:4875 , piece_id=XIZPGQAKGE36WFGQPMB24SDXG6YCGN7JR2U2KFDQ54ULLFLPLDEQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4875: use of closed network connection Nov 14 02:04:19.561613700 (114h0m18.113084161s) [noise=true , address=145.79.227.15:4875 , piece_id=XIZPGQAKGE36WFGQPMB24SDXG6YCGN7JR2U2KFDQ54ULLFLPLDEQ, seg_pos={0 0}, upload=100317]: Long tail cancellation due to optimal threshold Nov 14 02:04:19.561782331 (114h0m18.113252783s) [noise=true , address=145.79.227.11:4732 , piece_id=2ZJRRRXCTERAIT4LTGPWHS767NCMXSRP72GI6SECIB4YQFUWCBHQ, seg_pos={0 0}, upload=100317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:35012->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35012->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35012->145.79.227.11:4732: use of closed network connection Nov 14 02:04:19.561792440 (114h0m18.113262891s) [noise=true , address=145.79.227.11:4732 , piece_id=2ZJRRRXCTERAIT4LTGPWHS767NCMXSRP72GI6SECIB4YQFUWCBHQ, seg_pos={0 0}, upload=100317]: Long tail cancellation due to optimal threshold Nov 14 02:04:19.561861983 (114h0m18.113332434s) [noise=true , address=145.79.227.8:4331 , piece_id=PCZCMGASPSPISTJIQBQLMW3ESQVAGYIH5AH72IUX7VUTDFSKLRIA, seg_pos={0 0}, upload=100317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:45232->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:45232->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45232->145.79.227.8:4331: use of closed network connection Nov 14 02:04:19.561868867 (114h0m18.113339328s) [noise=true , address=145.79.227.8:4331 , piece_id=PCZCMGASPSPISTJIQBQLMW3ESQVAGYIH5AH72IUX7VUTDFSKLRIA, seg_pos={0 0}, upload=100317]: Long tail cancellation due to optimal threshold Nov 14 02:04:19.561883685 (114h0m18.113354146s) [seg_pos={0 0}, upload=100317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 02:04:19.562051364 (114h0m18.113521825s) [upload=100317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:04:19.562060722 (114h0m18.113531173s) [upload=100317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:04:19.855269065 (114h0m18.406739546s) [upload=100317]: Done uploading segments... Nov 14 02:04:19.855914765 (114h0m18.407385236s) [upload=100317]: Upload finished. err: <nil> Duration: 30.00328551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.270116362s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:04:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.270145749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.822337589s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:03:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.822368908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................X.......................................] duration: 5.962082463s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 02:03:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.962104243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.635227158s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.635254068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................X......................................] duration: 5.958322543s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 02:02:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.958357109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.745175741s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:02:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.745205037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.782072356s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:01:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.782102722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................X.....................................] duration: 5.838846867s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 02:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.838887735s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 323.184313ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1fc7c26ef222e679 FAIL (2025-11-14 02:01:09 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:53548->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 323.278362ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)