Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.100205827s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.10023344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.907753732s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:53:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.907784199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.229242628s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.229271393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.902847891s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:52:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.903242652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.976368888s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:51:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.976401951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.937624062s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.937653778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.98154851s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.981579289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.863306963s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.863337611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.847034728s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.847061811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.121343045s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.121376039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.014587122s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:48:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.014619474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.975913955s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:47:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.975944493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.948177814s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.948209213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.172222448s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.172248459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.061586867s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:46:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.061621594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.918986338s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:45:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.919065399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.034816672s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:44:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.034848163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.9331075s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:43:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.933201129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.053880142s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:42:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.053935499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.038556686s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.038573929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.969789347s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:40:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.969822592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.176707137s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:40:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.176734979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.966271286s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:38:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.966302956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.266995244s trace: https://jaeger.dev.storj.io/trace/1c5c8003a8e83f0c FAIL (2025-11-16 05:38:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 346694111s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795241233 (165h33m41.346711694s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.795891822 (165h33m41.347362283s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796363521 (165h33m41.347833982s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.796748143 (165h33m41.348218594s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.797977784 (165h33m41.349448235s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.798441968 (165h33m41.349912429s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.799817197 (165h33m41.351287658s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.801401924 (165h33m41.352872375s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.803308675 (165h33m41.354779136s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805300477 (165h33m41.356770938s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.805869841 (165h33m41.357340292s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.807385568 (165h33m41.358856029s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808902126 (165h33m41.360372587s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.808967159 (165h33m41.36043761s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:42.810886034 (165h33m41.362356495s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Uploading piece... Nov 16 05:37:44.597630157 (165h33m43.149100628s) [upload=145754]: Next returned nil segment Nov 16 05:37:44.597647540 (165h33m43.149117991s) [upload=145754]: Waiting for error group managing segments... Nov 16 05:37:44.894167352 (165h33m43.445637813s) [noise=true , address=77.90.31.19:4843 , piece_id=EQ7LBQSQN4G3EV633X3X5EPTET4SOIX6ZJH4Z3CHJL3HK6NONSVA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.942246573 (165h33m43.493717024s) [noise=true , address=45.91.116.4:4939 , piece_id=IDU5QP2GIPSHVRY3C7J4BNKFQZLIX2W3JP3ZBDF7UMOGWG6T2ASQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.961667925 (165h33m43.513138386s) [noise=true , address=45.91.116.8:4047 , piece_id=DBE55EUY32CVYV57K465S5FRDUIRWZ76IGJVHWJWMTOHLCZQLA3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:44.962619406 (165h33m43.514089867s) [noise=true , address=45.91.116.6:4975 , piece_id=XQ4E2RYLHCOJYULU6P3A4WVIR7VIQZD5I5XOGY6KUS44P7W2CSPA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.897162707 (165h33m46.448633169s) [noise=true , address=145.79.227.20:4439 , piece_id=XKZNLOMKWDVIK6F5QHJWTHC5IULNCRIG62EBFPBHDUOYDI6XMDLQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:47.943675556 (165h33m46.495146027s) [noise=true , address=145.79.227.30:4761 , piece_id=S75RHZ67KQIFN4W43ZC4TRZW4DUNEC6SBLO5I275ZXCP774BQC3Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:49.676860212 (165h33m48.228330663s) [noise=true , address=145.79.227.16:4910 , piece_id=UOGU5G4KY45GVMPRK6CTNPFU6HB2XTY6LFRGULVOT6HDB2BSFGGQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:50.415196234 (165h33m48.966666695s) [noise=true , address=145.79.227.31:4797 , piece_id=VKAOHHO3W5JVW6J3S7ACLCVHHHS4PSX4UWZ3FDS2QCFYC3MXZGQA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:37:51.545355703 (165h33m50.096826154s) [noise=true , address=145.79.227.32:4835 , piece_id=TUS6JOJ5XRI6AMPTIX6JRXKZWU6GYYFLPWLNVHK5BIVQG46T4EKA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: <nil> Nov 16 05:38:11.814213177 (165h34m10.365683648s) [noise=true , address=145.79.227.17:4972 , piece_id=NZCFIJT2ELGKTYSE7NJOVWUHTFQKXYCZWMEOV2U3AZAWWHONDECA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->145.79.227.17:4972: use of closed network connection Nov 16 05:38:11.814455238 (165h34m10.365925699s) [noise=true , address=145.79.227.22:4511 , piece_id=U7QVXFWMRZFPW3IAII3S4JKP64WNFKYSWEGOTYATUZV4YMISE25Q, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42840->145.79.227.22:4511: use of closed network connection Nov 16 05:38:11.814703351 (165h34m10.366173812s) [noise=true , address=145.79.227.9:4367 , piece_id=EMQXCHZJJMHWBCWASSLOVTHUGD66SAMBNLCVK6JUSUZXIH5TINTA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54226->145.79.227.9:4367: use of closed network connection Nov 16 05:38:11.814730582 (165h34m10.366201053s) [noise=true , address=145.79.227.10:4402 , piece_id=A3VFB3XVL7TKSI7KA34LL7WKT35C2YA7ERN4DBIGY6FU6OBHWH3A, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 05:38:11.814850971 (165h34m10.366321433s) [noise=true , address=145.79.227.15:4874 , piece_id=FSBBTJ2QQKTRN6WKU5YSX5P64FGJNL7VKX6EGKIJTMNI3GM5ENEA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38566->145.79.227.15:4874: use of closed network connection Nov 16 05:38:11.814992701 (165h34m10.366463172s) [noise=true , address=145.79.227.25:4619 , piece_id=F3XDVMW2LYV6ZGW2K3NDP2ANYWMZIPXWXZK63LOJGCQMIU4CGDDQ, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59882->145.79.227.25:4619: use of closed network connection Nov 16 05:38:11.815142095 (165h34m10.366612557s) [noise=true , address=145.79.227.27:4690 , piece_id=HIMKFN3K5EXHO6ZAA3BJMWWHRGBDDZINJ5QPBLW5XV7WMD767XYA, seg_pos={0 0}, upload=145754]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49478->145.79.227.27:4690: use of closed network connection Nov 16 05:38:11.815174357 (165h34m10.366644818s) [seg_pos={0 0}, upload=145754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 16 05:38:12.078838527 (165h34m10.630308988s) [upload=145754]: Done uploading segments... Nov 16 05:38:12.078891778 (165h34m10.630362239s) [upload=145754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.267022116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.063114963s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:20:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.063148987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.082110583s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:20:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.082140159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.910047522s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:19:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.910073601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.164730677s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:19:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.164782806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.070784267s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:18:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.070815706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.244346485s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:17:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.244372244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.114708576s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:17:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.114737782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.53923468s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539262765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.676149092s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:16:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.676177387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.952191873s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:16:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.952223422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.018606772s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:15:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.018639995s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.681243382s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:15:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681270334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.473555207s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:15:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473592778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.98015543s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:14:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.980171671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.776875025s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:14:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.776906014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.535714599s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:14:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535744867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.152437148s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:13:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.152477315s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.429411218s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:13:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429438248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.981650178s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:12:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.981668955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.572220353s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572338007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.451950306s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451985202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.96758126s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:11:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.967607701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.584115804s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584150901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.112276121s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.112297641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.496626388s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:10:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496652386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.315620133s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:10:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.315870469s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)