You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Apr 23, 2024. It is now read-only.
In several languages there are graphemes that consist of several characters - typically it's a base followed by one or many combining characters. For example: a + ◌̈ = ä.
Youtokentome assumes that every character is a valid grapheme and generates tokens that may start with a combining character.
If would be beneficial to train and encode with an option to pre-merge all combining characters to their base characters before running the actual BPE.
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
In several languages there are graphemes that consist of several characters - typically it's a base followed by one or many combining characters. For example:
a
+◌̈
=ä
.Youtokentome assumes that every character is a valid grapheme and generates tokens that may start with a combining character.
If would be beneficial to train and encode with an option to pre-merge all combining characters to their base characters before running the actual BPE.
The text was updated successfully, but these errors were encountered: