Allow intersecting items in NER.

[buriy]

How do I allow entities of different types to intersect in NER?
For example, if I have "Address": "Moscow, Frunze str., 5", then I also have tags "GEO/LOC" ("Moscow"), and "PER" tag ("Frunze" is also a person last name, and is a street name as well).
Currently, I have two options:

1. different NER models in the same pipeline, but they can't have multiple tags of the simplified markup in token.entity_type, they share lemmatization, but still don't share tok2vec NN.
2. different pipelines, but then they can't share vectors and lemmatization, and processing speed becomes even slower. Also saved models all store the same vectors, multiplying the amount of the memory used.

( Here's speed comparison between NLP frameworks on Russian datasets: https://github.com/natasha/naeval#ner , there are also POS and DEP comparisons there, and Spacy-RU-NER uses an additional corpus to beat most other models )

[adrianeboyd]

Since the base Token representation doesn't allow overlapping entities, I think the only solution for multiple models in the same pipeline is to have a custom pipeline component between NER models that copies the entity spans to a custom extension on the doc and then resets in the entities in preparation for the next NER model. After all the NER models have been applied, you can access all the entity spans saved in the custom extensions for further processing. It's not going to be particularly elegant.

[buriy]

No, it's ok if it's not elegant now, I'd like to maximize the speed of the current solution because it's already too slow for some of the use cases.
I think I will also need a cache for tok2vec: 8 sentences/second on a single CPU core is very slow.
(I usually think in terms of these 3 stages: make it work - make it work fast - make it elegant)

Do you have plans for v3.0 to support intersecting NER entities?

[svlandeg]

Yes! From v.3 onwards, we want to store the entities on the Doc level instead of on the Token level, so this should then be supported. We have yet to hash out the details for this, but stay tuned ;-)

[MalikRumi]

Similar issue: I wanted to subclass an entity type, i.e., GPE-Nation, GPE-City, etc. Is that possible now? Can you tell me how? If not, is it on the roadmap for v.3? If not, can you add it please? Thanks.

[svlandeg]

Built-in support for subclassing of entity types is not currently on the roadmap. Right now what I would advice is to perform one round of classification using the supertypes, then filter the data for each type and run a second round of classification (one per supertype) to get the more fine-grained types. In 2.x, you'll have to reset the entity annotations because each Token can only be part of one entity, as described above. From 3.0 onwards, you'd be able to store both, but the principle would remain the same.

As it looks like the original question was answered sufficiently, I'll go ahead and close this issue.