📓 NLP Study Group

Background: We originally started tracking the FastAI course.

But because of a lack of depth, we decided instead to dissect blogs and tutorials.

Week 0

Introductions!

What does your current work look like? How do you plan to use NLP?

---

Week 1

1. What is NLP?

• A changing field
• Resources
• Tools
• Python libraries
• Example applications
• Ethics issues

Notebook: What is NLP?

Video(s):

• FastAI: What is NLP?
• SOTA 2019

---

Week 2

Video(s):

• FastAI: Topic Modeling with SVD & NMF

Notebook:

• Topic Modeling with SVD & NMF

Resources:

• A Tutorial on PCA (closely related to SVD)
• SVD article, explaining intuition behind factorizing to 3 different matrices

Week 3

Video(s):

• FastAI: Logistic Regression & Naive Bayes

Notebook:

• Logistic Regression & Naive Bayes

Resources:

• Baysean Inference

Questions:

• Why is it when you plot the sorted document-term matrix it shows those waves

Week 4

Skipped

Week 5

Video(s):

• FastAI: Transfer Learning

Notebook:

• Transfer Learning

Resources:

• NLP's ImageNet Movement
• Cyclical Learning Rates
• Stop using Word2Vec

Questions:

• When we pick a learning rate off the LR vs. Loss graph, we pick it halway down the descent to the lowest point. Why is that?

Week 6: Word2Vec

Blogs:

• Word Embeddings - Part 1: Intro
• Word Embeddings - Part 2: Approximating SoftMax

Week 7: Word2Vec

• Word Embeddings - Part 3: Secret Ingredients of Word2Vec
• Word Embeddings - Part 5: Future Trends

Optional:

• The illustrated Word2Vec

Week 8: RNNs, LSTMs, GRUs

• The Unreasonable Effectiveness of RNNs
• Illustrated Guide to LSTMs and GRUs
• Understanding LSTM Networks

Week 9: LSTMs, Seq2Seq w/ Attn

• Tutorial on LSTMs: A Computational Perspective
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
• Seq2Seq with Attention and Beam Search

**Week 10: Attention **

• Attention and Augmented Recurrent Neural Networks
• Attention? Attention!
• Attention is all you need

Optional:

• Intuitive Understanding of Attention Mechanism in Deep Learning
• NLP FROM SCRATCH: TRANSLATION WITH A SEQUENCE TO SEQUENCE NETWORK AND ATTENTION
• NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

**Week 11: Implementation of Seq2Seq **

Other Resources

Books

• Deep Learning Book
• FastAI Book

Classes

• Linear Algebra with code examples
• Linear Algebra Class(MIT Opencourseware)
• FastAI: Computational Linear Algebra

Articles+

• A Primer in BERTology: What we know about how BERT works
• Awesome NLP
• The Illustrated Transformer
• The Illustrated BERT, ELMo, and co.

Blogs:

• Distill
• OpenAI
• StichFix
• The Gradient