Improving Bengali Stopwords collection and Exception

[Debangan-MishraIIIT]

I want to help in improving the stopword collection and the exception list for Bengali. What should be the correct procedure for doing so? Is it raising an issue, or directly submitting a Pull Request. As I am new to open source contribution, I would like some help.

P.S.: Being interested in linguistics, NLP and open source, I want to contribute to Spacy. Is it possible for open source contribution to a new language (Assamese)?

[svlandeg]

Hi @Debangan-MishraIIIT!

We typically lean on native speakers for language-specific improvements, so your help would be greatly appreciated. Typically, it's easiest to submit a Pull Request directly, so we can review it. You'll want to create a fork of the spaCy repo, make a new branch, and then put your edits there. Once done, you can create the PR and target spaCy's master branch. Some further explanations can be found here.

Another resource to look at, is Ines's excellent post about improving language support here. In the code, have a look at the lang folder that has a separate module for each language - for an entirely new language you'd be adding one there + a corresponding module in the test folder. You can take some inspiration from the existing languages, read about spaCy's tokenization here, and general explanations about language data here. If you're new to spaCy, I also highly recommend our free, interactive course that runs in the browser.

As an example PR, here's the PR that was submitted to add Lower Sorbian support, and you can find similar PRs through this search function.

Feel free to follow-up here with further questions or if you run into any issue!