Switch from pyvi to underthesea for Vietnamese word tokenization

[BLKSerene]

Hi, spaCy currently uses pyvi for Vietnamese word tokenization, have you ever considered Underthesea as a better alternative?

[polm]

Thanks for the suggestion! I can see that Underthesea looks like an active and well-maintained project, but I don't understand Vietnamese, and it's not clear to me what's actually different about the tokenization. Are the differences in tokenization approach or quality documented somewhere, or could you explain them?

[BLKSerene]

I do not speak Vietnamese either. The only difference should be the tokenization accuracy, but the author does not provide any information about it.

[polm]

OK, in that case I think we'll stay as we are for now. Underthesea has some things that aren't attractive to us, like a GPL 3 license and potentially heavyweight dependencies.

If anyone later wants to chime in on accuracy we'd be happy to hear about it.