BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Simple implementation of BERT (TensorFlow 2.0 style) https://arxiv.org/abs/1810.04805

ToDo

• mecab tokenizer
• Multiprocessing (for preprocessing NSP data in parallel)

Requirements

pip install sentencePiece

Future Work

• GPT2

Reference Repositories

• TF 2.0 Style Transformer
• The Annotated Transformer
• Sentencepiece
• Sentencepiece2
• Official BERT
• PyTorch_BERT_Implementation