Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.......X............X..X............................] duration: 5.855794587s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:56:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.855823634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.......X............X..X...........................] duration: 5.702793097s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:56:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.702818766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.......X............X..X..........................] duration: 5.842790636s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.842905765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.......X............X..X.........................] duration: 5.999331963s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.999360648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.......X............X..X........................] duration: 5.706901796s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.706925181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.......X............X..X.......................] duration: 5.661955944s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:53:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.661981814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.......X............X..X......................] duration: 5.78205584s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:52:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.782085746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.......X............X..X.....................] duration: 5.745689676s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:52:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.745716766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.......X............X..X....................] duration: 5.715463012s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:51:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.715484953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.......X............X..X...................] duration: 5.778494461s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:50:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.778523336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.......X............X..X..................] duration: 5.754914224s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:49:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.754943209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.......X............X..X.................] duration: 5.854704398s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:48:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854728534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.......X............X..X................] duration: 5.776521558s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:47:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.776540084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.......X............X..X...............] duration: 5.720740233s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:47:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.720768096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.......X............X..X..............] duration: 5.844660693s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:46:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.844747509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....................X.......X............X..X.............] duration: 5.702886068s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.702931543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.....................X.......X............X..X............] duration: 5.73588497s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:45:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.73591653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.718494386s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:44:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.718523523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.....................X.......X............X..X...........] duration: 5.832153595s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:44:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832184875s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.87098581s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:44:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.871014285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.....................X.......X............X..X..........] duration: 5.648960792s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:43:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.648993133s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.821239999s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:43:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.821269665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.633658113s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:43:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.63367184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.....................X.......X............X..X.........] duration: 5.713890581s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:43:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713914085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.054198511s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:43:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.054227566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.803788202s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:42:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.803804994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.865789797s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:42:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.865813592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....................X.......X............X..X........] duration: 5.769491003s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.769516771s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.92743861s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:41:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.927471404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.....................X.......X............X..X.......] duration: 5.762813065s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:41:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.762860046s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.839492631s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.839524632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.....................X.......X............X..X......] duration: 5.786744487s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786774534s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.609423332s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:40:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.609473948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X......XX.....................X.......X............X..X.....] duration: 5.97813303s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:39:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.978162847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X......XX.....................X.......X............X..X....] duration: 5.736209519s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:39:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.736237111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.902469801s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:39:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.902504297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.765376635s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:38:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.765419496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X......XX.....................X.......X............X..X...] duration: 5.667528635s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:38:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.667563s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.90296244s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:37:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.902991104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X......XX.....................X.......X............X..X..] duration: 5.716501462s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:37:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.716547168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.982902398s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:36:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.982929489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X......XX.....................X.......X............X..X.] duration: 5.789357437s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.789381902s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.931453773s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:36:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.931475063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X......XX.....................X.......X............X..X] duration: 30.241501454s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:35:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241524708s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.816691001s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.817136379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X......XX.....................X.......X............X..] duration: 9.487772709s trace: https://jaeger.dev.storj.io/trace/6ad6e2fd5c5c0de4 FAIL (2025-11-27 21:34:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 2:23.385593390 (445h28m21.937063851s) [upload=392276]: Got next segment. Inline: false Nov 27 21:32:23.385653484 (445h28m21.937123945s) [upload=392276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:32:23.385661519 (445h28m21.93713198s) [upload=392276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:32:23.631516519 (445h28m22.18298698s) [seg_pos={0 0}, upload=392276]: Begin upload segment... Nov 27 21:32:23.631562828 (445h28m22.183033279s) [seg_pos={0 0}, upload=392276]: Done begin upload segment. Nov 27 21:32:23.631597754 (445h28m22.183068215s) [noise=true , address=145.79.227.43:4157 , piece_id=V6RXBXNNNQU4TARHQ3ELVPM32Y4RUJQGG2TQHHZXVM6ZKDA2FT3Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.631638702 (445h28m22.183109163s) [noise=true , address=145.79.227.6:4258 , piece_id=4QOYHD6GJBR5LEUARBQKQWPH5GY6TDGSLPYHZTFPR7OW6G2XPIJQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632110109 (445h28m22.18358057s) [noise=true , address=145.79.227.4:4187 , piece_id=6Z6GBKP7HVWFTC3TKG2OYNABIF3EPVO5CK74LEKHKHMIJ56PI54A, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632459214 (445h28m22.183929675s) [noise=true , address=145.79.227.17:4972 , piece_id=GG3E6Z4KWATDZAIBZU5EGY4NZP3JUGC7FZKEWJPIAOV2A6C2Z6TA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632675386 (445h28m22.184145847s) [noise=true , address=145.79.227.38:4978 , piece_id=3RHAEBD3G2IRJFBZQRHUWJGWGBC7MJKVQGOIPO23MY7RT7FVNNIA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633034119 (445h28m22.18450458s) [noise=true , address=145.79.227.13:4804 , piece_id=WEM7YBWHBIG2I7JB27EWSQC3Z2ADNHF4T5LRICME4U7AZJISYC2Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633414303 (445h28m22.184884754s) [noise=true , address=145.79.227.29:4942 , piece_id=YTWL37K4B25FKSQNF3DUW5UDQT36ROKTMZWP35L3YWRDEMMKOVYQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633771403 (445h28m22.185241864s) [noise=true , address=145.79.227.41:4086 , piece_id=7XJXASDCJXLRXUXXTWAEIG7RDRUYQM7RVSHKJMHTRN2ZWLRQDFLA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633990780 (445h28m22.185461241s) [noise=true , address=145.79.227.34:4905 , piece_id=6TSD55YCIHGYPYPSW54SZQUNYGB5RIWQZG7OISE4BAGIT32DUMPA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634313115 (445h28m22.185783576s) [noise=true , address=145.79.227.8:4329 , piece_id=PIOHQLZ5SV4TKEWQ3YY7BITKKHJ6L6QGQ5UDLWFDC2MA6OLQUA2Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634388598 (445h28m22.185859059s) [noise=true , address=145.79.227.22:4510 , piece_id=QSCG5ZXL4JG5UCW7TKN7B46KJ44F5THVR6O5BMTLN5GNYJBP62QQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634695303 (445h28m22.186165764s) [noise=true , address=145.79.227.14:4838 , piece_id=LXFSAWG6HD6DLYWZ2VBUQNMUEYFHY2EJPKFHA3CH3RD7Y7GYNL7Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634753563 (445h28m22.186224024s) [noise=true , address=145.79.227.44:4194 , piece_id=3RIBVK2TC36OQZN6LDGYIEJAFQWH7IXBZ4KIBFSBXIPX36FUIHJQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634996505 (445h28m22.186466956s) [noise=true , address=145.79.227.27:4690 , piece_id=HUFKMPAMA73QSUE5O2MX32GSMLCV7ZOM7SAUO4GY5XD7C43B4J6A, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.635302238 (445h28m22.186772689s) [noise=true , address=145.79.227.42:4122 , piece_id=HKOBEOK4O6OIPHR2J6UGCLJ4PT27SUHH7IFA2553YQK7RCRB2BLA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.637155707 (445h28m22.188626168s) [noise=true , address=77.90.31.14:4663 , piece_id=WUG32AV5P24XSKRPECDM2GCL4V3CJKUJZYRR7JMBEU6D6DWDO6SQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:25.322503861 (445h28m23.873974322s) [upload=392276]: Next returned nil segment Nov 27 21:32:25.322517897 (445h28m23.873988358s) [upload=392276]: Waiting for error group managing segments... Nov 27 21:32:25.612265686 (445h28m24.163736147s) [noise=true , address=145.79.227.8:4329 , piece_id=PIOHQLZ5SV4TKEWQ3YY7BITKKHJ6L6QGQ5UDLWFDC2MA6OLQUA2Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.620730926 (445h28m24.172201387s) [noise=true , address=145.79.227.38:4978 , piece_id=3RHAEBD3G2IRJFBZQRHUWJGWGBC7MJKVQGOIPO23MY7RT7FVNNIA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.635241897 (445h28m24.186712358s) [noise=true , address=145.79.227.6:4258 , piece_id=4QOYHD6GJBR5LEUARBQKQWPH5GY6TDGSLPYHZTFPR7OW6G2XPIJQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.639621297 (445h28m24.191091768s) [noise=true , address=145.79.227.4:4187 , piece_id=6Z6GBKP7HVWFTC3TKG2OYNABIF3EPVO5CK74LEKHKHMIJ56PI54A, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.640311451 (445h28m24.191781912s) [noise=true , address=145.79.227.17:4972 , piece_id=GG3E6Z4KWATDZAIBZU5EGY4NZP3JUGC7FZKEWJPIAOV2A6C2Z6TA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.644747788 (445h28m24.196218239s) [noise=true , address=145.79.227.34:4905 , piece_id=6TSD55YCIHGYPYPSW54SZQUNYGB5RIWQZG7OISE4BAGIT32DUMPA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.645778691 (445h28m24.197249163s) [noise=true , address=145.79.227.29:4942 , piece_id=YTWL37K4B25FKSQNF3DUW5UDQT36ROKTMZWP35L3YWRDEMMKOVYQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.651444260 (445h28m24.202914711s) [noise=true , address=145.79.227.42:4122 , piece_id=HKOBEOK4O6OIPHR2J6UGCLJ4PT27SUHH7IFA2553YQK7RCRB2BLA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.658670090 (445h28m24.210140551s) [noise=true , address=145.79.227.13:4804 , piece_id=WEM7YBWHBIG2I7JB27EWSQC3Z2ADNHF4T5LRICME4U7AZJISYC2Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.664727253 (445h28m24.216197704s) [noise=true , address=145.79.227.14:4838 , piece_id=LXFSAWG6HD6DLYWZ2VBUQNMUEYFHY2EJPKFHA3CH3RD7Y7GYNL7Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.743764887 (445h28m24.295235338s) [noise=true , address=77.90.31.14:4663 , piece_id=WUG32AV5P24XSKRPECDM2GCL4V3CJKUJZYRR7JMBEU6D6DWDO6SQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:52.689036006 (445h28m51.240506457s) [noise=true , address=145.79.227.43:4157 , piece_id=V6RXBXNNNQU4TARHQ3ELVPM32Y4RUJQGG2TQHHZXVM6ZKDA2FT3Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection Nov 27 21:32:52.689172816 (445h28m51.240643267s) [noise=true , address=145.79.227.44:4194 , piece_id=3RIBVK2TC36OQZN6LDGYIEJAFQWH7IXBZ4KIBFSBXIPX36FUIHJQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection Nov 27 21:32:52.689258620 (445h28m51.240729081s) [noise=true , address=145.79.227.41:4086 , piece_id=7XJXASDCJXLRXUXXTWAEIG7RDRUYQM7RVSHKJMHTRN2ZWLRQDFLA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection Nov 27 21:32:52.689433132 (445h28m51.240903593s) [noise=true , address=145.79.227.27:4690 , piece_id=HUFKMPAMA73QSUE5O2MX32GSMLCV7ZOM7SAUO4GY5XD7C43B4J6A, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection Nov 27 21:32:52.689524266 (445h28m51.240994727s) [noise=true , address=145.79.227.22:4510 , piece_id=QSCG5ZXL4JG5UCW7TKN7B46KJ44F5THVR6O5BMTLN5GNYJBP62QQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection Nov 27 21:32:52.689556186 (445h28m51.241026657s) [seg_pos={0 0}, upload=392276]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:32:52.932668881 (445h28m51.484139332s) [upload=392276]: Done uploading segments... Nov 27 21:32:52.932703698 (445h28m51.484174159s) [upload=392276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.487801974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.925043542s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:34:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.925078948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.634167398s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:33:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.634194952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X......XX.....................X.......X............X.] duration: 5.832311903s trace: https://jaeger.dev.storj.io/trace/6ad6e2fd5c5c0de4 FAIL (2025-11-27 21:33:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 2:23.385593390 (445h28m21.937063851s) [upload=392276]: Got next segment. Inline: false Nov 27 21:32:23.385653484 (445h28m21.937123945s) [upload=392276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:32:23.385661519 (445h28m21.93713198s) [upload=392276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:32:23.631516519 (445h28m22.18298698s) [seg_pos={0 0}, upload=392276]: Begin upload segment... Nov 27 21:32:23.631562828 (445h28m22.183033279s) [seg_pos={0 0}, upload=392276]: Done begin upload segment. Nov 27 21:32:23.631597754 (445h28m22.183068215s) [noise=true , address=145.79.227.43:4157 , piece_id=V6RXBXNNNQU4TARHQ3ELVPM32Y4RUJQGG2TQHHZXVM6ZKDA2FT3Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.631638702 (445h28m22.183109163s) [noise=true , address=145.79.227.6:4258 , piece_id=4QOYHD6GJBR5LEUARBQKQWPH5GY6TDGSLPYHZTFPR7OW6G2XPIJQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632110109 (445h28m22.18358057s) [noise=true , address=145.79.227.4:4187 , piece_id=6Z6GBKP7HVWFTC3TKG2OYNABIF3EPVO5CK74LEKHKHMIJ56PI54A, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632459214 (445h28m22.183929675s) [noise=true , address=145.79.227.17:4972 , piece_id=GG3E6Z4KWATDZAIBZU5EGY4NZP3JUGC7FZKEWJPIAOV2A6C2Z6TA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.632675386 (445h28m22.184145847s) [noise=true , address=145.79.227.38:4978 , piece_id=3RHAEBD3G2IRJFBZQRHUWJGWGBC7MJKVQGOIPO23MY7RT7FVNNIA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633034119 (445h28m22.18450458s) [noise=true , address=145.79.227.13:4804 , piece_id=WEM7YBWHBIG2I7JB27EWSQC3Z2ADNHF4T5LRICME4U7AZJISYC2Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633414303 (445h28m22.184884754s) [noise=true , address=145.79.227.29:4942 , piece_id=YTWL37K4B25FKSQNF3DUW5UDQT36ROKTMZWP35L3YWRDEMMKOVYQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633771403 (445h28m22.185241864s) [noise=true , address=145.79.227.41:4086 , piece_id=7XJXASDCJXLRXUXXTWAEIG7RDRUYQM7RVSHKJMHTRN2ZWLRQDFLA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.633990780 (445h28m22.185461241s) [noise=true , address=145.79.227.34:4905 , piece_id=6TSD55YCIHGYPYPSW54SZQUNYGB5RIWQZG7OISE4BAGIT32DUMPA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634313115 (445h28m22.185783576s) [noise=true , address=145.79.227.8:4329 , piece_id=PIOHQLZ5SV4TKEWQ3YY7BITKKHJ6L6QGQ5UDLWFDC2MA6OLQUA2Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634388598 (445h28m22.185859059s) [noise=true , address=145.79.227.22:4510 , piece_id=QSCG5ZXL4JG5UCW7TKN7B46KJ44F5THVR6O5BMTLN5GNYJBP62QQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634695303 (445h28m22.186165764s) [noise=true , address=145.79.227.14:4838 , piece_id=LXFSAWG6HD6DLYWZ2VBUQNMUEYFHY2EJPKFHA3CH3RD7Y7GYNL7Q, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634753563 (445h28m22.186224024s) [noise=true , address=145.79.227.44:4194 , piece_id=3RIBVK2TC36OQZN6LDGYIEJAFQWH7IXBZ4KIBFSBXIPX36FUIHJQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.634996505 (445h28m22.186466956s) [noise=true , address=145.79.227.27:4690 , piece_id=HUFKMPAMA73QSUE5O2MX32GSMLCV7ZOM7SAUO4GY5XD7C43B4J6A, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.635302238 (445h28m22.186772689s) [noise=true , address=145.79.227.42:4122 , piece_id=HKOBEOK4O6OIPHR2J6UGCLJ4PT27SUHH7IFA2553YQK7RCRB2BLA, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:23.637155707 (445h28m22.188626168s) [noise=true , address=77.90.31.14:4663 , piece_id=WUG32AV5P24XSKRPECDM2GCL4V3CJKUJZYRR7JMBEU6D6DWDO6SQ, seg_pos={0 0}, upload=392276]: Uploading piece... Nov 27 21:32:25.322503861 (445h28m23.873974322s) [upload=392276]: Next returned nil segment Nov 27 21:32:25.322517897 (445h28m23.873988358s) [upload=392276]: Waiting for error group managing segments... Nov 27 21:32:25.612265686 (445h28m24.163736147s) [noise=true , address=145.79.227.8:4329 , piece_id=PIOHQLZ5SV4TKEWQ3YY7BITKKHJ6L6QGQ5UDLWFDC2MA6OLQUA2Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.620730926 (445h28m24.172201387s) [noise=true , address=145.79.227.38:4978 , piece_id=3RHAEBD3G2IRJFBZQRHUWJGWGBC7MJKVQGOIPO23MY7RT7FVNNIA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.635241897 (445h28m24.186712358s) [noise=true , address=145.79.227.6:4258 , piece_id=4QOYHD6GJBR5LEUARBQKQWPH5GY6TDGSLPYHZTFPR7OW6G2XPIJQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.639621297 (445h28m24.191091768s) [noise=true , address=145.79.227.4:4187 , piece_id=6Z6GBKP7HVWFTC3TKG2OYNABIF3EPVO5CK74LEKHKHMIJ56PI54A, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.640311451 (445h28m24.191781912s) [noise=true , address=145.79.227.17:4972 , piece_id=GG3E6Z4KWATDZAIBZU5EGY4NZP3JUGC7FZKEWJPIAOV2A6C2Z6TA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.644747788 (445h28m24.196218239s) [noise=true , address=145.79.227.34:4905 , piece_id=6TSD55YCIHGYPYPSW54SZQUNYGB5RIWQZG7OISE4BAGIT32DUMPA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.645778691 (445h28m24.197249163s) [noise=true , address=145.79.227.29:4942 , piece_id=YTWL37K4B25FKSQNF3DUW5UDQT36ROKTMZWP35L3YWRDEMMKOVYQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.651444260 (445h28m24.202914711s) [noise=true , address=145.79.227.42:4122 , piece_id=HKOBEOK4O6OIPHR2J6UGCLJ4PT27SUHH7IFA2553YQK7RCRB2BLA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.658670090 (445h28m24.210140551s) [noise=true , address=145.79.227.13:4804 , piece_id=WEM7YBWHBIG2I7JB27EWSQC3Z2ADNHF4T5LRICME4U7AZJISYC2Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.664727253 (445h28m24.216197704s) [noise=true , address=145.79.227.14:4838 , piece_id=LXFSAWG6HD6DLYWZ2VBUQNMUEYFHY2EJPKFHA3CH3RD7Y7GYNL7Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:25.743764887 (445h28m24.295235338s) [noise=true , address=77.90.31.14:4663 , piece_id=WUG32AV5P24XSKRPECDM2GCL4V3CJKUJZYRR7JMBEU6D6DWDO6SQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: <nil> Nov 27 21:32:52.689036006 (445h28m51.240506457s) [noise=true , address=145.79.227.43:4157 , piece_id=V6RXBXNNNQU4TARHQ3ELVPM32Y4RUJQGG2TQHHZXVM6ZKDA2FT3Q, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.43:4157: use of closed network connection Nov 27 21:32:52.689172816 (445h28m51.240643267s) [noise=true , address=145.79.227.44:4194 , piece_id=3RIBVK2TC36OQZN6LDGYIEJAFQWH7IXBZ4KIBFSBXIPX36FUIHJQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->145.79.227.44:4194: use of closed network connection Nov 27 21:32:52.689258620 (445h28m51.240729081s) [noise=true , address=145.79.227.41:4086 , piece_id=7XJXASDCJXLRXUXXTWAEIG7RDRUYQM7RVSHKJMHTRN2ZWLRQDFLA, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57432->145.79.227.41:4086: use of closed network connection Nov 27 21:32:52.689433132 (445h28m51.240903593s) [noise=true , address=145.79.227.27:4690 , piece_id=HUFKMPAMA73QSUE5O2MX32GSMLCV7ZOM7SAUO4GY5XD7C43B4J6A, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40400->145.79.227.27:4690: use of closed network connection Nov 27 21:32:52.689524266 (445h28m51.240994727s) [noise=true , address=145.79.227.22:4510 , piece_id=QSCG5ZXL4JG5UCW7TKN7B46KJ44F5THVR6O5BMTLN5GNYJBP62QQ, seg_pos={0 0}, upload=392276]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33358->145.79.227.22:4510: use of closed network connection Nov 27 21:32:52.689556186 (445h28m51.241026657s) [seg_pos={0 0}, upload=392276]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:32:52.932668881 (445h28m51.484139332s) [upload=392276]: Done uploading segments... Nov 27 21:32:52.932703698 (445h28m51.484174159s) [upload=392276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.832343292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.894358255s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:33:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.894386808s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)