Spell Checker suggestions

[DonaldTsang]

Having a spell checker that involves:

1. Keyboard typos with adjacent keys from QWERTY and QWERTZ
2. Misspellings, including addition, deletion, repitition and phonetic replacement
3. Out-Of-Vocabulary handling with OOV selection, clustering and misspelling

[DuyguA]

Hello,
Spellchecking is a highly domain (language model) driven task. Mispellings are guessed
such that the LM assigns the maximum probability to the sentence. It makes sense anyway. If you have restaurant reviews, grn is most probably green, if you have movie reviews, it can be grin.

This task involves

• Introducing a LM. Here question is to support which formats, RNNLM and ARPALM, for a start.

• String correction is done by DAGs. Vocabulary is stored in a character or subword based DAG and composed with the LM.
  I play with DAGs a lot: https://arxiv.org/abs/1803.00902 .Please roll until the figure with grün then you'll see the efficiency advantages of DAGs in language computations.
  Current vocabulary object of SpaCy is not a graph like data structure.

In short, this task requires changing a data structure and loads of PyFST code. I wish best of luck to the volunteers :)

[xwiz]

Have you tried Symspell? I just published an extension for Spacy on PyPi implementing Symspell. I have a simple good idea on how the accuracy can be improved using character groups which are like unit morphemes.

[DonaldTsang]

@xwiz explain it in detail please, I am interested

[xwiz]

Basically the root idea is to sense the context and grammatical correctness based on word juxtaposition frequency and potentially grammatical rules for completeness.

For example look at the following sentences:

• that dook now about anything
• what doyon about anyhting

These two sentences are corrections provided by Symspell with the default dictionary of about 500k words for the sentence: 'What doyuoknowabout antyhing'

From those suggestions, it is clear that symspell does not take into account context, grammatical rules and word-with-word commonness (i.e. how words are commonly combined).

Furthermore a spelling corrector should be able to determine randomness as well as incomprehensibility. Most human beings won't be able to understand: 'whas do dou meansad thati am thefull' but most deep learning based correctors would figure out something out of that. The right way to respond to a sentence like that would be 'what' or 'pardon'? It is not completely random but it is incoherent.

The solution would imply

1. Using a pure statistical approach or seq2seq to figure out how words are commonly juxtaposed.
2. Using same method as above to figure out how character groups (in 2s or 3s, example th, shi) are commonly juxtaposed.
3. Using fixed logic or deep learning to figure out and correct grammatical rules (basically how words are juxtaposed based on their POS tag)
4. Providing probability scores which would be used to determine incoherence for any of the above processes.

I will start a repo to test the statistical/non deep learning approach right away.

[R1j1t]

Hi, I created a spell checker which uses context to correct the misspells. If you get time do have a look! I plan to submit it to Spacy Universe

GitHub: https://github.com/R1j1t/contextualSpellCheck
PyPI: https://pypi.org/project/contextualSpellCheck/

There are some features left (like RWE or using cython) for development, but I plan to complete it in the coming days. While developing this, I really liked the Spacy documentation and the architecture. I did not think it would be this easy to add the extensions or pipeline object. Love the Spacy project!

[DonaldTsang]

@R1j1t have you ever compared yourself with all the other major contextual spell checkers? It would be good to see which ones have more features, and others that are the best at integration with other systems, and learn from that.

[xwiz]

Interesting one @R1j1t Did you try it with my example above? i.e.

What doyuoknowabout antyhing?

[R1j1t]

That is a good idea, @DonaldTsang, contributions are always welcome! @xwiz I tried the sentence you quoted, but the model output is not what you or I would expect.

>>> doc = nlp("What doyuoknowabout antyhing?")
>>> doc._.outcome_spellCheck
'What about mean?'

I would not defend the model, but I think it will take time. Just out of curiosity I tried that same input sentence on duckduckgo and google search. On DuckDuckGo, I did not get any suggestion (seemed a bit weird) and on google I was prompted

Did you mean: What do you know about anty hing?

But, like I said, it is still a work in progress and at present, I am focusing on RWE and performance improvement. If you find the work interesting please jump in and help out on some the tasks you mentioned. I would love to see people contributing.

[xwiz]

Yeah Google uses a mostly statistical based approach and it works well. The return value could mean that anty and hing are also common enough in its database and it prioritizes that kind of separation over letter misplacement.

[R1j1t]

okay, but dont you think anything being a much more common word and the edit distance also being low should be the output. This was my assumption, dont really know why google's model proposed it, maybe you are right.