Update the logic of misspell identification

[R1j1t]

Is your feature request related to a problem? Please describe.
The current logic of misspelling identification relies on vocab.txt from the transformer model. BERT tokenisers break not such common words into subwords and subsequently store the sub-words in vocab.txt. Hence the original word might not be present in vocab.txt and be identified as misspelt.

Describe the solution you'd like
Still not clear, need to look into some papers on this.

Describe alternatives you've considered
Alternate which I can think of right now will be 2 folds:

• ask user to provide list of such words and append in the vocab.txt from the transformers model
• if the proposed change is ##x then check the editdistance from detokenised form of that word + previous word

Additional context
#30 explosion/spaCy#3994

[letconex]

Concerning the logic: Is this a viable response?

>>> doc = nlp("This is a majour mistaken.")
>>> print(doc._.outcome_spellCheck)
This is a fact mistaken.
>>> doc = nlp("This is a majour mistake.")
>>> print(doc._.outcome_spellCheck)
This is a major mistake.
>>> doc = nlp("This is a majour mistakes.")
>>> print(doc._.outcome_spellCheck)
This is a for mistakes.
>>> doc = nlp("This is a majour misstake.")
>>> print(doc._.outcome_spellCheck)
This is a minor story.

[R1j1t]

That is not the desired response. But it is based on the current logic. If you want to improve accuracy, please try pass the vocab file

contextualSpellCheck/contextualSpellCheck/contextualSpellCheck.py

Lines 34 to 35 in 15b30eb

	vocab_path (str, optional): Vocabulary file path to be used by the
	model . Defaults to "".

This will help model prevent False positives. Feel free to open a PR with a fix!

[kshitij12345]

One side-effect of using the current transformers tokenizer logic is that it would by default support multi-lingual models. Otherwise I am not sure but I think different languages might require different spell-checkers as per the language nuances.

[R1j1t]

As mentioned in the README

This package currently focuses on Out of Vocabulary (OOV) word or non-word error (NWE) correction using BERT model.

So lets say you want to perform spell correction on Japanese sentence:

1. provide Japanese spacy model: This will break the sentence into tokens. Now as this model is trained on Japanese language it knows the nuances (better than english model)
2. Provide the Japanese bert model (from tokenizer models): Which will provide the candidate word for OOV word. Note that vocabulary here is considered of the transformer model and not the spaCy model

Below is some code contributed to the repo for Japanese language:

contextualSpellCheck/examples/ja_example.py

Lines 4 to 13 in f8cbeb8

	nlp = spacy.load("ja_core_news_sm")
	checker = ContextualSpellCheck(
	model_name="cl-tohoku/bert-base-japanese-whole-word-masking",
	max_edit_dist=2,
	)
	nlp.add_pipe(checker)
	doc = nlp("しかし大勢においては、ここような事故はウィキペディアの拡大には影響を及ぼしていない。")
	print(doc._.performed_spellCheck)
	print(doc._.outcome_spellCheck)

contextualSpellCheck examples folder

I hope it answers your question @kshitij12345. Please feel free to provide ideas or reference if you find something I might have missed something here!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.