Skip to content

jfecunha/nlpaug

BranchesTags
 
 

Repository files navigation



Build Code Quality Downloads

nlpaug

This python library helps you with augmenting nlp for your machine learning projects. Visit this introduction to understand about Data Augmentation in NLP. Augmenter is the basic element of augmentation while Flow is a pipeline to orchestra multi augmenter together.

Features

  • Generate synthetic data for improving model performance without manual effort
  • Simple, easy-to-use and lightweight library. Augment data in 3 lines of code
  • Plug and play to any machine leanring/ neural network frameworks (e.g. scikit-learn, PyTorch, TensorFlow)
  • Support textual and audio input

Textual Data Augmentation Example


Acoustic Data Augmentation Example


Section Description
Quick Demo How to use this library
Augmenter Introduce all available augmentation methods
Installation How to install this library
Recent Changes Latest enhancement
Extension Reading More real life examples or researchs
Reference Reference of external resources such as data or model

Quick Demo

Augmenter

Augmenter Target Augmenter Action Description
Textual Character KeyboardAug substitute Simulate keyboard distance error
Textual OcrAug substitute Simulate OCR engine error
Textual RandomAug insert, substitute, swap, delete Apply augmentation randomly
Textual Word AntonymAug substitute Substitute opposite meaning word according to WordNet antonym
Textual ContextualWordEmbsAug insert, substitute Feeding surroundings word to BERT, DistilBERT, RoBERTa or XLNet language model to find out the most suitlabe word for augmentation
Textual RandomWordAug swap, crop, delete Apply augmentation randomly
Textual SpellingAug substitute Substitute word according to spelling mistake dictionary
Textual SplitAug split Split one word to two words randomly
Textual SynonymAug substitute Substitute similar word according to WordNet/ PPDB synonym
Textual TfIdfAug insert, substitute Use TF-IDF to find out how word should be augmented
Textual WordEmbsAug insert, substitute Leverage word2vec, GloVe or fasttext embeddings to apply augmentation
Textual BackTranslationAug substitute Leverage two translation models for augmentation
Textual ReservedAug substitute Replace reserved words
Textual Sentence ContextualWordEmbsForSentenceAug insert Insert sentence according to XLNet, GPT2 or DistilGPT2 prediction
Textual AbstSummAug substitute Summarize article by abstractive summarization method
Textual LambadaAug substitute Using language model to generate text and then using classification model to retain high quality results
Signal Audio CropAug delete Delete audio's segment
Signal LoudnessAug substitute Adjust audio's volume
Signal MaskAug substitute Mask audio's segment
Signal NoiseAug substitute Inject noise
Signal PitchAug substitute Adjust audio's pitch
Signal ShiftAug substitute Shift time dimension forward/ backward
Signal SpeedAug substitute Adjust audio's speed
Signal VtlpAug substitute Change vocal tract
Signal NormalizeAug substitute Normalize audio
Signal PolarityInverseAug substitute Swap positive and negative for audio
Signal Spectrogram FrequencyMaskingAug substitute Set block of values to zero according to frequency dimension
Signal TimeMaskingAug substitute Set block of values to zero according to time dimension
Signal LoudnessAug substitute Adjust volume

Flow

Augmenter Augmenter Description
Pipeline Sequential Apply list of augmentation functions sequentially
Pipeline Sometimes Apply some augmentation functions randomly

Installation

The library supports python 3.5+ in linux and window platform.

To install the library:

pip install numpy requests nlpaug

or install the latest version (include BETA features) from github directly

pip install numpy git+https://github.com/makcedward/nlpaug.git

or install over conda

conda install -c makcedward nlpaug

If you use BackTranslationAug, ContextualWordEmbsAug, ContextualWordEmbsForSentenceAug and AbstSummAug, installing the following dependencies as well

pip install torch>=1.6.0 transformers>=4.0.0 sentencepiece

If you use LambadaAug, installing the following dependencies as well

pip install simpletransformers>=0.61.10

If you use AntonymAug, SynonymAug, installing the following dependencies as well

pip install nltk>=3.4.5

If you use WordEmbsAug (word2vec, glove or fasttext), downloading pre-trained model first

from nlpaug.util.file.download import DownloadUtil
DownloadUtil.download_word2vec(dest_dir='.') # Download word2vec model
DownloadUtil.download_glove(model_name='glove.6B', dest_dir='.') # Download GloVe model
DownloadUtil.download_fasttext(model_name='wiki-news-300d-1M', dest_dir='.') # Download fasttext model

If you use SynonymAug (PPDB), downloading file from the following URI. You may not able to run the augmenter if you get PPDB file from other website

http://paraphrase.org/#/download

If you use PitchAug, SpeedAug and VtlpAug, installing the following dependencies as well

pip install librosa>=0.7.1 matplotlib

Recent Changes

1.1.8dev, Aug, 2021

  • Added RandomSentAug
  • Added skip_check parameter for WordEmbsAug

See changelog for more details.

Extension Reading

Reference

This library uses data (e.g. capturing from internet), research (e.g. following augmenter idea), model (e.g. using pre-trained model) See data source for more details.

Citing

@misc{ma2019nlpaug,
  title={NLP Augmentation},
  author={Edward Ma},
  howpublished={https://github.com/makcedward/nlpaug},
  year={2019}
}

Book cited nlpaug

Research paper cited nlpaug

Project cited nlpaug

Contributions


sakares saengkaew
Binoy Dalal

About

Data augmentation for NLP

makcedward.github.io/

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

22 tags

Packages

No packages published

Languages

  • Jupyter Notebook 61.2%
  • Python 38.7%
  • Shell 0.1%