This repository has been archived by the owner on Apr 23, 2024. It is now read-only.
Controlling word tokenization #73
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I would like to know if there is any possibility to control the splitting of a word into tokens, besides setting up bpe dropout?
For example: "Best" can be tokenized into ['▁Best'] or ['▁Be', 'st'] and etc. Can I choose which one tokens to use?
The text was updated successfully, but these errors were encountered: