Is Spacy developed English models specific to USA only? No support to India city, currency unit

[Rammurthy5]

How to reproduce the behaviour

import spacy
nlp = spacy.load('en_core_web_sm')
for i in nlp(u'Apple is gonna build a factory in Bangalore. Bengaluru worth ₹565 , another one 6546342₹').ents:
    print(i, i.label_)

Output:

Apple ORG
Bangalore ORG
Bengaluru ORG
₹565 NORP
one CARDINAL

Your Environment

• Operating System: macOS
• Python Version Used: 3.7.7
• spaCy Version Used: 2.0.16
• Environment Information: jupyter notebook

[BramVanroy]

The English spacy models were trained on OntoNotes. You can have a look here to see what sources that includes.

[Rammurthy5]

Thanks for the swift response. The model is able to identify some Indian cities, whilst failing at some. Doesn’t it sound like a bug, if I m not wrong?

…

On 16-May-2020, at 1:32 PM, Bram Vanroy ***@***.***> wrote: The English spacy models were trained on OntoNotes. You can have a look here <https://catalog.ldc.upenn.edu/LDC2013T19>to see what sources that includes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/explosion/spaCy/issues/5443#issuecomment-629606220>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJLGQSANJJKXYSDQVRMVBADRRZCB7ANCNFSM4NCZRCVQ>.

[BramVanroy]

Don't think so. It can simply mean that those cities were not part of the training data, so the model never learned. If you have a lot of data you can train your own spaCy model.

[honnibal]

You should probably try updating the model to recognise the cities you're missing. You have two options for this:

1. Train a new model
2. Try to update the existing model by adding examples
3. Use matcher rules to fix prominent errors

Updating the model is always a bit fiddly, because it's hard to reason about the learning process. But the library does support it -- you can do nlp.resume_training() and then make calls to nlp.update().

Matcher often work really well, and are an option you should consider.