Add support for Kabyle (kab)

[teolemon]

What

• Add support for Kabyle (kab) (as requested by a translator which has started work on it on Launchpad)

Part of

• 🌍 Make Open Food Facts work well in all countries in the world #5690

[hangy]

This problem is at least twofold.

• Update taxonomy to contain languages that don't have a language_code_2 value.
• Update Product Opener to support different versions of language ISO codes. Right now, only two-letter codes are supported. To do this right, it would probably make sense to treat deu and de as equivalent and redirect to the latter, and only allow the language_code_3 to be used, if there is no language_code_2. The $lc variable and it's friends are used everywhere in Product Opener, often with the assumption that it should always contain two characters.

[hangy]

https://crowdin.com/project/openfoodfacts/kab

[hangy]

I was wrong. The issues I mentioned in February still persist.

Also, there is apparently an inconsistency

openfoodfacts-server/lib/ProductOpener/Display.pm

Line 275 in e771134

if ((not defined $lc) or (($lc !~ /^\w\w(_|-)\w\w$/) and (length($lc) != 2) )) {

openfoodfacts-server/lib/ProductOpener/Display.pm

Line 302 in e771134

if ((defined param('lc')) and (defined $language_codes{param('lc')})) {

The former checks that the $lc is only two chars, but the latter does not.

[teolemon]

This is causing issues for Portuguese i think. We need full standard local support for many languages we now have.

[hangy]

We need full standard local support for many languages we now have.

• Support for ISO 639-3 (kab, fra) in addition to ISO 639-1 (fr)
• Support for more complex IETF language codes

Anything missing? 😉

[aleene]

I have been using the language codes used by wikidata, which seem sometimes their own invention and I saw some other issues.

• the current taxonomies sometimes have a rs-entry for croatia. The correct ISO 693-1(?) code is sr. There is a similar issue for norway and norwegian;
• off needs to support the three letter language codes. These are supported by wikidata and crowdin;
• sometimes a language has multiple scripts variants. For instance sr can be written in latin and cyrillic script. Wikidata has encoded this as sr-ec and sr-el. Their default entry for sr can be in either script, although cyrillic seems to preferred. There are other countries with this issue as well: Kazakh and Tatar come to mind;
• the same language can have country variants. In wikidata I saw those for english (canada, great britain), portuguese (brasil), german (switzerland, austria), kazakh (turkey, kazakhstan, china). However there is not much difference between these variants (low impact).
• chinese is a special case with two variants (han traditional and han simplified). China, Singapore and Malaysia use simplified. Traditional is used by Hongkong, Macao and Taiwan. In vert few cases Hongkong and Taiwan have there own variant. The default is in wikidata very badly defined, sometimes traditional, sometimes simplified.
• Wikidata has various other deviations. Belarus in old script for instance. (Can make a list).
• Interestingly Crowdin has other languages that I have yet to see in the wikidata. So we have to think about priorities.