Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect Toki Pona in non-latin text #3

Open
gregdan3 opened this issue Jun 25, 2024 · 0 comments
Open

Detect Toki Pona in non-latin text #3

gregdan3 opened this issue Jun 25, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@gregdan3
Copy link
Owner

Currently, the library can only detect Toki Pona in latin-alphabet text and UCSUR; all text in other writing systems is considered to not be Toki Pona, even though it is perfectly reasonable to render Toki Pona in almost any writing system.

To do this as fully as my preferential config for Latin alphabet text, I would need the following per script:

  • List of words in the dictionary rendered in the target script (each Dictionary filter)
  • A regex which matches words rendered with appropriate syllables (Syllabic filter)
  • A list of all the characters in the language which may be used to render (Alphabetic filter)

While the alphabetic filter specifically would be relatively easy (even though it would be improperly named for, say, Japanese), the dictionary and syllabic filters would be challenging for languages which have multiple ways to write approximately the same sound in Toki Pona. For example, I was provided this list for Greek by jan Niwe (@Nerd1729 on Discord):

α = /a/
ε = αι = /e/
η = ι = υ = ει = οι = υι = /i/
γη = γι = γυ = γει = γοι = γυι = /j/
κ = /k/
λ = /l/
μ = /m/
ν = /n/
ο = ω = /o/
π = /p/
σ = /s/
τ = /t/
ου = ȣ = /u/
β = γου = /w/
@gregdan3 gregdan3 added the enhancement New feature or request label Jun 25, 2024
@gregdan3 gregdan3 changed the title [Feature] Detecting Toki Pona in non-latin text Detect Toki Pona in non-latin text Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant