How to correctly add lemmas into spacy default lookup table ?

[Pandalei97]

Hello !

I want to add some lemmas in the spacy default lemma lookup table. But it doesn't work as expected.

I did like this :

nlp = spacy.load('fr_core_news_sm')
nlp.select_pipes(enable=["lemmatizer", "morphologizer"])
lemmatizer = nlp.get_pipe("lemmatizer")
lemma_lookup = lemmatizer.lookups.get_table("lemma_lookup")
lemma_lookup.set("rétro-péritonéale", "rétro-péritonéal")
lemma_lookup.set("rétro-péritonéales", "rétro-péritonéal")
doc = nlp("rétro-péritonéale rétro-péritonéales")
tokens = [(t.text, t.lemma_) for t in doc]
# Output
# [('rétro-péritonéale', 'rétro-péritonéale'), ('rétro-péritonéales', 'rétro-péritonéales')]

Did I do something wrong ?

Thanks in advance for your response.

SpaCy version : 3.0.6

[adrianeboyd]

The lookups are only used in the French lemmatizer as a backoff for cases not covered by the rules. There's no detailed documentation for the French lemmatizer, but it should be pretty easy to follow the code to figure out which tables are relevant for the cases you want to modify:

spaCy/spacy/lang/fr/lemmatizer.py

Lines 25 to 80 in 78a88f7

	def rule_lemmatize(self, token: Token) -> List[str]:
	cache_key = (token.orth, token.pos)
	if cache_key in self.cache:
	return self.cache[cache_key]
	string = token.text
	univ_pos = token.pos_.lower()
	if univ_pos in ("", "eol", "space"):
	return [string.lower()]
	elif "lemma_rules" not in self.lookups or univ_pos not in (
	"noun",
	"verb",
	"adj",
	"adp",
	"adv",
	"aux",
	"cconj",
	"det",
	"pron",
	"punct",
	"sconj",
	):
	return self.lookup_lemmatize(token)
	index_table = self.lookups.get_table("lemma_index", {})
	exc_table = self.lookups.get_table("lemma_exc", {})
	rules_table = self.lookups.get_table("lemma_rules", {})
	lookup_table = self.lookups.get_table("lemma_lookup", {})
	index = index_table.get(univ_pos, {})
	exceptions = exc_table.get(univ_pos, {})
	rules = rules_table.get(univ_pos, [])
	string = string.lower()
	forms = []
	if string in index:
	forms.append(string)
	self.cache[cache_key] = forms
	return forms
	forms.extend(exceptions.get(string, []))
	oov_forms = []
	if not forms:
	for old, new in rules:
	if string.endswith(old):
	form = string[: len(string) - len(old)] + new
	if not form:
	pass
	elif form in index or not form.isalpha():
	forms.append(form)
	else:
	oov_forms.append(form)
	if not forms:
	forms.extend(oov_forms)
	if not forms and string in lookup_table.keys():
	forms.append(self.lookup_lemmatize(token)[0])
	if not forms:
	forms.append(string)
	forms = list(set(forms))
	self.cache[cache_key] = forms
	return forms

Things to note:

• you need to keep tok2vec in the pipeline for morphologizer to run correctly
• in the rule-based lemmatizer you probably want to add exceptions to lemma_exc instead
• the rule-based lemmatizer is POS-dependent so short docs like "rétro-péritonéale rétro-péritonéales" where the words are used out of context may not get tagged correctly, which cascades into lemmatizer errors, so you might just need to check token.pos_ first to figure out the source of the error
• you can switch from a rule-based lemmatizer to a lookup lemmatizer (no morphologizer required) instead if that would be useful for your task, see: https://spacy.io/models#design-modify

[Pandalei97]

Thanks for you detailed response !

Indeed I have forgotten to add the tok2vec in the demo code.

Here's another test with a real example by considering your suggestions:

# spacy 3.0.6
import spacy

nlp = spacy.load('fr_core_news_sm')
nlp.select_pipes(enable=["tok2vec", "morphologizer", "lemmatizer"])
lemmatizer = nlp.get_pipe("lemmatizer")
lemma_exc = lemmatizer.lookups.get_table("lemma_exc")
lemma_exc.set("rétro-péritonéale", "rétro-péritonéal")
lemma_exc.set("rétro-péritonéales", "rétro-péritonéal")
lemma_exc.set("rétro-péritonéaux", "rétro-péritonéal")
doc = nlp("une évolution métastatique ganglionnaire rétro-péritonéale.")
tokens = [(t.text, t.lemma_, t.pos_) for t in doc]

Output:

[('une', 'un', 'DET'), 
('évolution', 'évolution', 'NOUN'),
('métastatique', 'métastatique', 'ADJ'),
('ganglionnaire', 'ganglionnaire', 'ADJ'),
('rétro-péritonéale', 'rétro-péritonéale', 'NOUN'), 
('.', '.', 'PUNCT')]

The morphologizer have mispredicted the POS of rétro-péritonéale (NOUN instead of ADJ).

I understand that this may happen. But is there a way to force morphologizer to consider "rétro-péritonéale" as an ADJ ? (rétro-péritonéale is never a noun). In our project, we will often see sentences with "NOUN + ganglionnaire rétro-péritonéal/rétro-péritonéale/rétro-péritonéales/rétro-péritonéaux".

Most of the time the morphologizer works well so we don't want to abandon it.

[adrianeboyd]

The exceptions are also keyed by POS, so that entry will be ignored. There's no way to change the existing morphologizer's behavior, but you can add rule-based exceptions with another component, e.g., an attribute ruler.

See the current lemma_exc table: https://raw.githubusercontent.com/explosion/spacy-lookups-data/master/spacy_lookups_data/data/fr_lemma_exc.json

[Pandalei97]

The attribute_ruler solves the problem. Thanks a lot !!

I post my solution here in case it's useful :

# spacy 3.0.6
import spacy

nlp = spacy.load('fr_core_news_sm')
# Only select the relevant components
nlp.select_pipes(enable=["tok2vec", "attribute_ruler", "morphologizer", "lemmatizer"])
ruler = nlp.get_pipe("attribute_ruler")
patterns = [
    {
        "patterns": [[{"LOWER": "rétro-péritonéale"}], [{"LOWER": "rétro-péritonéales"}], [{"LOWER": "rétro-péritonéaux"}]],
        "attrs": {"LEMMA": "rétro-péritonéal"}
    }
]
ruler.add_patterns(patterns)
doc = nlp("une évolution métastatique ganglionnaire rétro-péritonéale.")
tokens = [(t.text, t.lemma_, t.pos_) for t in doc]

Output:

[('une', 'un', 'DET'), ('évolution', 'évolution', 'NOUN'), ('métastatique', 'métastatique', 'ADJ'), ('ganglionnaire', 'ganglionnaire', 'ADJ'), ('rétro-péritonéale', 'rétro-péritonéal', 'NOUN'), ('.', '.', 'PUNCT')]