This is a very simple micromaterial created for the Oxford Summer of Hacks Language Hack Day.
The aim is to give learners practice in doing a very simple NLP task: turning text into its corresponding Part of Speech (POS) tags. These are things like NNS (singular noun) and JJS (comparative adjective).
- what is a POS tag, and how to get them from text
- count the total different types of POS tags in a text
- count the frequency of different POS tags in a text
- calculate the most common n-grams of POS tags in a text
One big skeleton function has already been written, along with the test for it. So to complete the activity, just fill in the functions and run the tests. If the test passes, you did it! If not, try to fix the function so the test passes.
to run the test:
python -m unittest
Here is a list of POS tags used in both nltk and spacy, two popular NLP libraries in python. And this is a good explanation of POS tags and what they're used for (though sketch engine doesn't use the Penn Treebank set of POS tags).
The Wikipedia page has a much more in-depth discussion of these, if you're interested.
For our purposes, it doesn't matter whether you use the NLTK POS tagger (example) or the Spacy POS tagger (example)
Basically, read in your data source, load the models, and pass in the text into the appropriate function. It might be a good idea to write the output to a file for later analysis so you don't have to repeat this step multiple times.
An n-gram means a set of things that show up together, with the "n" part meaning the number of things. For example, if we're looking at POS tags, a 2-gram that shows up a lot might be "IN, DET" ("in a"), and a 3-gram that shows up a lot might be "DT NNS IN" ("the NOUN of"). For a more detailed discussion, see the Wikipedia page.
Follow the steps from #2, and see which 2-grams and 3-grams are the most frequent. If you want to use your own data source (perhaps a web page or document from the file system), possibly even look for 4-grams or 5-grams.