is_plural/is_singular are really missing

[hzitoun]

Feature description

That would be nice to have an attribute is_plural associated to Token class.

[adrianeboyd]

Hi, there is support for morphological features under development that will include features like Number_sing and Number_plur.

[hzitoun]

Hi, there is support for morphological features under development that will include features like Number_sing and Number_plur.

Great, I'm eager to use it! Because I'm not at all satisfied with my current way to check if a tag is plural or not!

split_on_number = spacy_token.tag_.split("Number=")
if len(split_on_number) > 1 and split_on_number[1].startswith("Plur"):
  return true

Do you propose any other way? @adrianeboyd

[adrianeboyd]

Right now there are long combined tag+morph tags as Token.tag_ to get around the fact that a real morphological analysis isn't available yet. If the language you're working with uses the combined tags, I'd suggest something more like:

if "Number=Plur" in token.tag_:
    ...

---

And just as a preview (but UNSTABLE and UNDOCUMENTED):

The morphological feature implementation is still under development, so please don't rely on this right now, but the idea is that you will have a Token.morph attribute:

import spacy
nlp = spacy.load('en')
doc = nlp("She walks.")
assert "Number_sing" in doc[1].morph

Some languages have the morphological features from the combined tags incorporated into their tag maps so you can experiment with Token.morph a bit (the example above should work for real), but it's possible that the API will change in the future.

[2ny]

split_on_number = spacy_token.tag_.split("Number=")
if len(split_on_number) > 1 and split_on_number[1].startswith("Plur"):
  return true

Hey @hzitoun
how do you manage to have custom "extra" string into your token.tag_?
I'm trying to train a model with additional semantic information, but I can not find a clean way to add it to the model (as there doesn't seem to exist a way to add custom fields in the model).
So, would it be possible to append some extra info to tag?
For now, when validating my JSON data, SpaCy reports me those errors:

✘ Label 'DT,Number=Plur' not found in tag map for language 'en'

(I've tried "DT__Number=Plur", "DT|Number=Plur" without more success)

[adrianeboyd]

@2ny: Some languages supported by spacy come with combined tag+morph tags, typically ones converted from UD corpora. You can use spacy convert -m to create the combined tags (XPOS + "__" + FEATURES) from a CoNLL-U corpus. The accuracy for the tagger is slightly worse than with a smaller tagset, but often only a little bit worse even though you're going from 10s to 100s of tags in many cases. (Beware that combining closely-related morpho-syntactic tags is probably a very different story from combining semantic tags, though, in terms of performance!)

I hope to make extending the tag map easier with CLI options in the next version of spacy, but for now you need to edit spacy/lang/lg/tag_map.py to add the new tags before training in your local spacy install. It doesn't matter what your tags look like, every single tag just needs to be in this dict.

Here's the preview of how it might work in the future, where you can see how to update the tag map in spacy/cli/train.py on-the-fly: https://github.com/explosion/spaCy/pull/4750/files

[2ny]

thanks @adrianeboyd for your answer.
would having additional fields be possible for the trainer? For semantic informations that are neither DEP, TAG or NER, and that we would retrieve in the ._ Token extensions? Maybe it would help for custom morph analysis, but also for speech (intonation sound for exemple) related stuff, or a lot of other domains knowledge the user may have in his dataset.

[svlandeg]

Hi, to come back to the original question, I just wanted to point out that the current spacy-nightly, which is the Release Candidate for v3, has much better support for such morphological features:

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("I gave them the news paper.")
print(doc[0].morph.get("Number"))
print(doc[2].morph.get("Number"))

will print

['Sing']
['Plur']