New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

diacritic opt-in for frequencies #19

Open

paceaux opened this issue Feb 19, 2024 · 1 comment

Labels

Owner

paceaux commented Feb 19, 2024 •

edited

Loading

right now n-grams work by stripping diacritics; all frequencies/ results take away accent marks.

We should at least offer the ability to opt-in to diacritics.
Seeing confusing results on the demo:

in portuguese, ão is listed, but we can't highlight (a problem with the demo, but still)

With French, we have éèê to account for and it may be useful to see this separate.

paceaux added the enhancement label

Owner Author

paceaux commented Feb 27, 2024

This should probably be a few pieces:

There should be an ability to identify characters with diacritics and keep them in a separate place
There should be an ability to match characters with diacritics to ones without (i.e. knowing that èéê are all still e)
Frequency analysis with diacritics on/off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment