Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.84934019s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:14:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 7.849371671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 27.471976444s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:13:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 27.472006352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 13.110630201s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:12:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 13.110662384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 15.568388014s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:12:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 15.568418983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 15.097367428s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 15.097387327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 19.961865636s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:11:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 19.9618984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 22.1154065s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:10:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 22.115448171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 14.807003507s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:09:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 14.807036328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 9.26844454s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 9.268478974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 19.062654456s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:08:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 19.062685777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.943960092s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:07:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 8.943992654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.533348693s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 12.533379031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.774247099s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 7.774289249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.754598278s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:05:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 7.754653234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 12.308108315s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:04:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 12.308149474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 10.317865461s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 10.317898112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.848691605s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:02:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 7.848724367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.844634089s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:02:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 9.844678905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.412852334s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 8.41290327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.241754346s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 13:00:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 8.241780775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 11.314077762s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 12:59:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 11.314116216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.543790067s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 12:58:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 6.543815516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 16.944974323s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 12:58:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 16.94499868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002665549s trace: https://jaeger.dev.storj.io/trace/27722b6774b4dd96 FAIL (2025-11-08 12:57:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ) context deadline exceeded ======LOGS====== 780106 (217h16m51.956628859s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 12:57:20.534743978 (217h16m52.230592741s) [seg_pos={0 0}, upload=191366]: Begin upload segment... Nov 8 12:57:20.534808550 (217h16m52.230657303s) [seg_pos={0 0}, upload=191366]: Done begin upload segment. Nov 8 12:57:20.534850541 (217h16m52.230699294s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.536244956 (217h16m52.232093709s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537044028 (217h16m52.232892781s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.537631847 (217h16m52.2334806s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538151096 (217h16m52.233999849s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.538685223 (217h16m52.234534026s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.539601547 (217h16m52.2354503s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540409517 (217h16m52.23625827s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.540579130 (217h16m52.236427883s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541022383 (217h16m52.236871146s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541251630 (217h16m52.237100373s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.541819301 (217h16m52.237668054s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542270370 (217h16m52.238119123s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.542838793 (217h16m52.238687546s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543299660 (217h16m52.239148413s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:20.543749436 (217h16m52.23959819s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Uploading piece... Nov 8 12:57:22.365505731 (217h16m54.061354494s) [upload=191366]: Next returned nil segment Nov 8 12:57:22.365569842 (217h16m54.061418585s) [upload=191366]: Waiting for error group managing segments... Nov 8 12:57:22.674976246 (217h16m54.370824999s) [noise=true , address=145.79.227.13:4803 , piece_id=7HZRN2SKXDZLNFQ4ZO4RDIHKHDM5P6AXX4H7NSFOL2YUU2RYESLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.676115686 (217h16m54.371964449s) [noise=true , address=145.79.227.14:4838 , piece_id=QJL2PRJLYTPJFT2LXHD7WT6YBC7OJTTM2P32NCOOD4PHR2KKKABQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.678526637 (217h16m54.37437539s) [noise=true , address=145.79.227.27:4690 , piece_id=2J4KUVZRVXPL2F2Z5N7PXMSQ47QAZ7DTQZCSH5IWGU4Z346YSJDQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683596190 (217h16m54.379444943s) [noise=true , address=145.79.227.30:4762 , piece_id=7NBZIASQQPKBYPKYI2KUVMO2LWLQYPENEADBZVLDKPFLX2J63FPQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.683718052 (217h16m54.379566806s) [noise=true , address=145.79.227.20:4439 , piece_id=EN4ODDFAMJKOMVIWNNFBAH2S6ZAZFXCOACDE57TNDOB7N2AVUNRQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.703710282 (217h16m54.399559025s) [noise=true , address=145.79.227.31:4797 , piece_id=QR6YIBKV6EXT2N3APPCPPFUQ5VTVPL6GOKVA3UE52RJJ2NTQ33DA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.735576934 (217h16m54.431425697s) [noise=true , address=145.79.227.18:4005 , piece_id=PJ6OMFVWK6IZTVZ3CB65MH32OM3CTG6JNO3G7TJ5QWBBRDH3HUTQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.744982524 (217h16m54.440831277s) [noise=true , address=145.79.227.11:4730 , piece_id=LNUYAI7JB53C5EIHPDXEAURB5OVEBENJQG5M52G75SSO3XJPQYLQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.817928431 (217h16m54.513777184s) [noise=true , address=45.91.116.5:4299 , piece_id=L5MMPAGPF3TDM4GKZ5YUO56NUWBIGN54DWCGTDBHMDUT7LVWWNFQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.851561456 (217h16m54.547410209s) [noise=true , address=145.79.227.17:4972 , piece_id=IVFM6YPNZSOO7WLFYOTOFNE3F5AR6F2FJWKBJPDSWUGMIYF2UMDA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.853649993 (217h16m54.549498746s) [noise=true , address=45.91.116.9:4263 , piece_id=TJ6EZ5DEBRWLGVTDGORSPYJGUCIU2TJNYA5AAYXPCIWEPDVC7ZJQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:22.916528539 (217h16m54.612377302s) [noise=true , address=45.91.116.4:4940 , piece_id=QBDQ2XX5YCBIFFTGM3W6XTQJZDXXHGSXFQEI7G44OJQIWETRRORA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138235268 (217h16m54.834084031s) [noise=true , address=45.91.116.6:4975 , piece_id=YHQA3OWGLFQYXL5AUPQJ4R5JLIEGI7I7VBVBYA7K5J23GQIRKOEQ, seg_pos={0 0}, upload=191366]: Done uploading piece. err: <nil> Nov 8 12:57:23.138266778 (217h16m54.834115521s) [seg_pos={0 0}, upload=191366]: Segment reached optimal threshold of 13 pieces. Nov 8 12:57:23.138867291 (217h16m54.834716044s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42922->45.91.116.14:4191: use of closed network connection Nov 8 12:57:23.138877281 (217h16m54.834726024s) [noise=true , address=45.91.116.14:4191 , piece_id=IHX2UOGL27ZWBZGHVBVVCHAL2YCMWL7UWT7DSQVR3YVHMILR6FMA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.138885937 (217h16m54.83473469s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41698->145.79.227.8:4331: use of closed network connection Nov 8 12:57:23.138896798 (217h16m54.834745551s) [noise=true , address=145.79.227.8:4331 , piece_id=WAPJR3SEQWLTQZSA24Z36MKXWW4CUBBD6GK6RNIFXT42A2TCAYFA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139029901 (217h16m54.834878654s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55254->145.79.227.16:4910: use of closed network connection Nov 8 12:57:23.139061431 (217h16m54.834910184s) [noise=true , address=145.79.227.16:4910 , piece_id=G72YSGGTOY5VGCFWJTUJBPSPPAH64WGYBM5F4L5ILKFRDIC24WGA, seg_pos={0 0}, upload=191366]: Long tail cancellation due to optimal threshold Nov 8 12:57:23.139092330 (217h16m54.834941083s) [seg_pos={0 0}, upload=191366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 12:57:23.139187622 (217h16m54.835036375s) [upload=191366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 12:57:23.139194866 (217h16m54.835043619s) [upload=191366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 12:57:23.769489093 (217h16m55.465337846s) [upload=191366]: Done uploading segments... Nov 8 12:57:23.769512277 (217h16m55.46536102s) [upload=191366]: Upload finished. err: <nil> Duration:* 30.002690185s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.387742727s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:15:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.387781772s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.221595733s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:14:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.221629467s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.26212433s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:13:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.262158715s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.31250521s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:13:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.312539595s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.188510366s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:11:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.188537257s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.119375589s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:10:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.11940715s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.316520755s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:10:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.31654958s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.474714057s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:10:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.474815089s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.373218618s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:09:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.373245508s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.277737986s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:09:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.277798914s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.450763948s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:09:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.450815597s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.450957673s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:08:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.451073001s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.063970329s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.064012639s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.324114704s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:08:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.324168096s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.447265519s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:07:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.447293822s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.241638905s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.24167318s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.164218869s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.164260887s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.305738277s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.305773704s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.560198636s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:06:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.560255104s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.898896989s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:06:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.898971381s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.180285155s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:05:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.180315845s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.274956771s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:05:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.274989943s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.483026794s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:05:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.48306274s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.26918026s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:04:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.269212551s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.377958015s trace: https://jaeger.dev.storj.io/trace/2f4f0eb203f8dfd3 FAIL (2025-11-08 04:04:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.377989856s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.256270332s trace: https://jaeger.dev.storj.io/trace/471c5be2a49ec9c1 FAIL (2025-11-08 04:03:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.256289478s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)