roadmap of cleaning perseus

[jankounchained]

Perseus idiosyncrasies

Lemmas
Root and suffix are sometimes dash-separated.
This is happens only for VERBs.
Learn the reason for this & remove dashes (possibly also suffixes if it makes sense).

Beta code encoding errors
Make sure that every character is a valid ancient greek letter.
For example, ἀλλ̓ should be ἀλλ’

Compatability with Proiel

XPOS
XPOS tags just contain the combination of UPOS & FEATS, no new information is introduced. Our model should not learn them / use them as labels

Morpohological features
Proiel has both more features & possible values.
How do we not learn proiel-specific features when training on their data?

POS
Perseus doesn't have PROPN,
while proiel doesn't have PUNCT

• Does training on Proiel make our senter worse?
• Should we convert PROPN to NOUN? Or are we learning PROPN well enough to keep it?