Distinguish between sister languages

[mamoit]

Languages should be organized in such a way that would make it easy to inherit common language rules, but distinguish them from each language.

For eg, Portuguese from Portugal pt-pt and from Brazil pt-br are quite different, and their rules differ in way more than just one place.
For eg, the use of words such as Vietnã and Vietname both mean Vietnam, the first in pt-br and the second in pt-pt.
I know this is just a subject alternation, but I am not coming up with any better examples at the moment.

I think it would be nice to create a base language pt, which pt-pt and pt-br would inherit from.
This would also be applicable to languages such as Spanish.

[ines]

If you're adding your own Language class, this should already be possible in theory – and probably even easier in v2.0. You could, for example, overwrite certain parts of the PortugueseDefaults in a Language subclass.

By design, the Language classes mostly favour composition over inheritance. This means that you don't really need a subclass to reuse the data or logic – you can split things up with different variables and functions however you like, and reuse components between the different variations of the language.

We still haven't decided on an "official" way of solving this problem going forward, but something like this might be a good solution:

• Separate tokenizer exceptions and other specific language data to allow pt-pt and pt-br to only use the relevant rules.
• Have one "general-purpose" Portuguese class and normalise differences between pt and br using the new norm exceptions. This would also make it easier to train models if you're working with mixed inputs (i.e. text using both variations), as it'd ensure that Vietnã and Vietname both receive similar representations – even if one of them is much less frequent in your training data.

Aside from the vocabulary, are there other significant differences in pt-pt and pt-br that affect the morphological rules or tag map? And if so, would there be an easy way to "unify" those to create a universal Portuguese class that could handle both at the same time? In general, this would always be our preferred solution, as we'd like spaCy to be able to handle all variations of a language within the same processing pipeline.

[mamoit]

This article from Oxford Dictionaries seems to illustrate the differences pretty thoroughly.

[ines]

Merging this with #2338!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.