New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does this tool support chinese? #54

Open

RayShark opened this issue Sep 8, 2021 · 1 comment

RayShark commented Sep 8, 2021

I just wondering whether this tool support chinese corpus.

For example, do i suppose to use Jieba or other chinese tokenizer ? And is there interface reserved for chinese tokenizer...

Thanks a lot.

gaojunyu21 commented Apr 30, 2024

是的，只需要将文件读取改为utf-8的格式读取即可

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment