Simple LSTM - PyTorch version.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Preface"},{"metadata":{},"cell_type":"markdown","source":"This kernel is a PyTorch version of the [Simple LSTM kernel](https://www.kaggle.com/thousandvoices/simple-lstm). All credit for architecture and preprocessing goes to @thousandvoices.\nThere is a lot of discussion whether Keras, PyTorch, Tensorflow or the CUDA C API is best. But specifically between the PyTorch and Keras version of the simple LSTM architecture, there are 2 clear advantages of PyTorch:\n- Speed. The PyTorch version runs about 20 minutes faster.\n- Determinism. The PyTorch version is fully deterministic. Especially when it gets harder to improve your score later in the competition, determinism is very important.\n\nI was surprised to see that PyTorch is that much faster, so I'm not completely sure the steps taken are exactly the same. If you see any difference, we can discuss it in the comments :)\n\nThe most likely reason the score of this kernel is higher than the @thousandvoices version is that the optimizer is not reinitialized after every epoch and thus the parameter-specific learning rates of Adam are not discarded after every epoch. That is the only difference between the kernels that is intended."},{"metadata":{},"cell_type":"markdown","source":"# Imports & Utility functions"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport time\nimport gc\nimport random\nfrom tqdm._tqdm_notebook import tqdm_notebook as tqdm\nfrom keras.preprocessing import text, sequence\nimport torch\nfrom torch import nn\nfrom torch.utils import data\nfrom torch.nn import functional as F","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# disable progress bars when submitting\ndef is_interactive():\n   return 'SHLVL' not in os.environ\n\nif not is_interactive():\n    def nop(it, *a, **k):\n        return it\n\n    tqdm = nop","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed=1234):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CRAWL_EMBEDDING_PATH = '../input/fasttext-crawl-300d-2m/crawl-300d-2M.vec'\nGLOVE_EMBEDDING_PATH = '../input/glove840b300dtxt/glove.840B.300d.txt'\nNUM_MODELS = 2\nLSTM_UNITS = 128\nDENSE_HIDDEN_UNITS = 4 * LSTM_UNITS\nMAX_LEN = 220","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def get_coefs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\n\ndef load_embeddings(path):\n    with open(path) as f:\n        return dict(get_coefs(*line.strip().split(' ')) for line in tqdm(f))\n\ndef build_matrix(word_index, path):\n    embedding_index = load_embeddings(path)\n    embedding_matrix = np.zeros((len(word_index) + 1, 300))\n    unknown_words = []\n    \n    for word, i in word_index.items():\n        try:\n            embedding_matrix[i] = embedding_index[word]\n        except KeyError:\n            unknown_words.append(word)\n    return embedding_matrix, unknown_words","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dict([('a', np.random.randint(1,5, size=(5,))), ('b', np.random.randint(1,5, size=(5,)))])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sigmoid(x):\n    return 1 / (1 + np.exp(-x))\n\ndef train_model(model, train, test, loss_fn, output_dim, lr=0.001,\n                batch_size=512, n_epochs=4,\n                enable_checkpoint_ensemble=True):\n    param_lrs = [{'params': param, 'lr': lr} for param in model.parameters()]\n    optimizer = torch.optim.Adam(param_lrs, lr=lr)\n\n    scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda epoch: 0.6 ** epoch)\n    \n    train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n    test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)\n    all_test_preds = []\n    checkpoint_weights = [2 ** epoch for epoch in range(n_epochs)]\n    \n    for epoch in range(n_epochs):\n        start_time = time.time()\n        \n        scheduler.step()\n        \n        model.train()\n        avg_loss = 0.\n        \n        for data in tqdm(train_loader, disable=False):\n            x_batch = data[:-1]\n            y_batch = data[-1]\n\n            y_pred = model(*x_batch)            \n            loss = loss_fn(y_pred, y_batch)\n\n            optimizer.zero_grad()\n            loss.backward()\n\n            optimizer.step()\n            avg_loss += loss.item() / len(train_loader)\n            \n        model.eval()\n        test_preds = np.zeros((len(test), output_dim))\n    \n        for i, x_batch in enumerate(test_loader):\n            y_pred = sigmoid(model(*x_batch).detach().cpu().numpy())\n\n            test_preds[i * batch_size:(i+1) * batch_size, :] = y_pred\n\n        all_test_preds.append(test_preds)\n        elapsed_time = time.time() - start_time\n        print('Epoch {}/{} \\t loss={:.4f} \\t time={:.2f}s'.format(\n              epoch + 1, n_epochs, avg_loss, elapsed_time))\n\n    if enable_checkpoint_ensemble:\n        test_preds = np.average(all_test_preds, weights=checkpoint_weights, axis=0)    \n    else:\n        test_preds = all_test_preds[-1]\n        \n    return test_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SpatialDropout(nn.Dropout2d):\n    def forward(self, x):\n        x = x.unsqueeze(2)    # (N, T, 1, K)\n        x = x.permute(0, 3, 2, 1)  # (N, K, 1, T)\n        x = super(SpatialDropout, self).forward(x)  # (N, K, 1, T), some features are masked\n        x = x.permute(0, 3, 2, 1)  # (N, T, 1, K)\n        x = x.squeeze(2)  # (N, T, K)\n        return x\n    \nclass NeuralNet(nn.Module):\n    def __init__(self, embedding_matrix, num_aux_targets):\n        super(NeuralNet, self).__init__()\n        embed_size = embedding_matrix.shape[1]\n        \n        self.embedding = nn.Embedding(max_features, embed_size)\n        self.embedding.weight = nn.Parameter(torch.tensor(embedding_matrix, dtype=torch.float32))\n        self.embedding.weight.requires_grad = False\n        self.embedding_dropout = SpatialDropout(0.3)\n        \n        self.lstm1 = nn.LSTM(embed_size, LSTM_UNITS, bidirectional=True, batch_first=True)\n        self.lstm2 = nn.LSTM(LSTM_UNITS * 2, LSTM_UNITS, bidirectional=True, batch_first=True)\n    \n        self.linear1 = nn.Linear(DENSE_HIDDEN_UNITS, DENSE_HIDDEN_UNITS)\n        self.linear2 = nn.Linear(DENSE_HIDDEN_UNITS, DENSE_HIDDEN_UNITS)\n        \n        self.linear_out = nn.Linear(DENSE_HIDDEN_UNITS, 1)\n        self.linear_aux_out = nn.Linear(DENSE_HIDDEN_UNITS, num_aux_targets)\n        \n    def forward(self, x):\n        h_embedding = self.embedding(x)\n        h_embedding = self.embedding_dropout(h_embedding)\n        \n        h_lstm1, _ = self.lstm1(h_embedding)\n        h_lstm2, _ = self.lstm2(h_lstm1)\n        \n        # global average pooling\n        avg_pool = torch.mean(h_lstm2, 1)\n        # global max pooling\n        max_pool, _ = torch.max(h_lstm2, 1)\n        \n        h_conc = torch.cat((max_pool, avg_pool), 1)\n        h_conc_linear1  = F.relu(self.linear1(h_conc))\n        h_conc_linear2  = F.relu(self.linear2(h_conc))\n        \n        hidden = h_conc + h_conc_linear1 + h_conc_linear2\n        \n        result = self.linear_out(hidden)\n        aux_result = self.linear_aux_out(hidden)\n        out = torch.cat([result, aux_result], 1)\n        \n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess(data):\n    '''\n    Credit goes to https://www.kaggle.com/gpreda/jigsaw-fast-compact-solution\n    '''\n    punct = \"/-'?!.,#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~`\" + '\"\"“”’' + '∞θ÷α•à−β∅³π‘₹´°£€\\×™√²—–&'\n    def clean_special_chars(text, punct):\n        for p in punct:\n            text = text.replace(p, ' ')\n        return text\n\n    data = data.astype(str).apply(lambda x: clean_special_chars(x, punct))\n    return data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/jigsaw-unintended-bias-in-toxicity-classification/train.csv')\ntest = pd.read_csv('../input/jigsaw-unintended-bias-in-toxicity-classification/test.csv')\n\nx_train = preprocess(train['comment_text'])\ny_train = np.where(train['target'] >= 0.5, 1, 0)\ny_aux_train = train[['target', 'severe_toxicity', 'obscene', 'identity_attack', 'insult', 'threat']]\nx_test = preprocess(test['comment_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_features = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = text.Tokenizer()\ntokenizer.fit_on_texts(list(x_train) + list(x_test))\n\nx_train = tokenizer.texts_to_sequences(x_train)\nx_test = tokenizer.texts_to_sequences(x_test)\nx_train = sequence.pad_sequences(x_train, maxlen=MAX_LEN)\nx_test = sequence.pad_sequences(x_test, maxlen=MAX_LEN)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_features = max_features or len(tokenizer.word_index) + 1\nmax_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"crawl_matrix, unknown_words_crawl = build_matrix(tokenizer.word_index, CRAWL_EMBEDDING_PATH)\nprint('n unknown words (crawl): ', len(unknown_words_crawl))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"glove_matrix, unknown_words_glove = build_matrix(tokenizer.word_index, GLOVE_EMBEDDING_PATH)\nprint('n unknown words (glove): ', len(unknown_words_glove))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_matrix = np.concatenate([crawl_matrix, glove_matrix], axis=-1)\nembedding_matrix.shape\n\ndel crawl_matrix\ndel glove_matrix\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train_torch = torch.tensor(x_train, dtype=torch.long).cuda()\nx_test_torch = torch.tensor(x_test, dtype=torch.long).cuda()\ny_train_torch = torch.tensor(np.hstack([y_train[:, np.newaxis], y_aux_train]), dtype=torch.float32).cuda()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = data.TensorDataset(x_train_torch, y_train_torch)\ntest_dataset = data.TensorDataset(x_test_torch)\n\nall_test_preds = []\n\nfor model_idx in range(NUM_MODELS):\n    print('Model ', model_idx)\n    seed_everything(1234 + model_idx)\n    \n    model = NeuralNet(embedding_matrix, y_aux_train.shape[-1])\n    model.cuda()\n    \n    test_preds = train_model(model, train_dataset, test_dataset, output_dim=y_train_torch.shape[-1], \n                             loss_fn=nn.BCEWithLogitsLoss(reduction='mean'))\n    all_test_preds.append(test_preds)\n    print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission = pd.DataFrame.from_dict({\n    'id': test['id'],\n    'prediction': np.mean(all_test_preds, axis=0)[:, 0]\n})\n\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Note that the solution is not validated in this kernel. So for tuning anything, you should build a validation framework using e. g. KFold CV. If you just check what works best by submitting, you are very likely to overfit to the public LB."},{"metadata":{},"cell_type":"markdown","source":"# Ways to improve this kernel"},{"metadata":{},"cell_type":"markdown","source":"This kernel is just a simple baseline kernel, so there are many ways to improve it. Some ideas to get you started:\n- Add a contraction mapping. E. g. mapping \"is'nt\" to \"is not\" can help the network because \"not\" is explicitly mentioned. They were very popular in the recent quora competition, see for example [this kernel](https://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing).\n- Try to reduce the number of words that are not found in the embeddings. At the moment, around 170k words are not found. We can take some steps to decrease this amount, for example trying to find a vector for a processed (capitalized, stemmed, ...) version of the word when the vector for the regular word can not be found. See the [3rd place solution](https://www.kaggle.com/wowfattie/3rd-place) of the quora competition for an excellent implementation of this.\n- Try cyclic learning rate (CLR). I have found CLR to almost always improve my network recently compared to the default parameters for Adam. In this case, we are already using a learning rate scheduler, so this might not be the case. But it is still worth to try it out. See for example my [my other PyTorch kernel](https://www.kaggle.com/bminixhofer/deterministic-neural-networks-using-pytorch) for an implementation of CLR in PyTorch.\n- Use sequence bucketing to train faster and fit more networks into the two hours. The winning team of the quora competition successfully used sequence bucketing to drastically reduce the time it took to train RNNs. An excerpt from their [solution summary](https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80568#latest-487092):\n\n> We aimed at combining as many models as possible. To do this, we needed to improve runtime and the most important thing to achieve this was the following. We do not pad sequences to the same length based on the whole data, but just on a batch level. That means we conduct padding and truncation on the data generator level for each batch separately, so that length of the sentences in a batch can vary in size. Additionally, we further improved this by not truncating based on the length of the longest sequence in the batch, but based on the 95% percentile of lengths within the sequence. This improved runtime heavily and kept accuracy quite robust on single model level, and improved it by being able to average more models.\n\n- Try a (weighted) average of embeddings instead of concatenating them. A 600d vector for each word is a lot, it might work better to average them instead. See [this paper](https://www.aclweb.org/anthology/N18-2031) for why this even works.\n- Limit the maximum number of words used to train the NN. At the moment, there is no limit set to the maximum number of words in the tokenizer, so we use every word that occurs in the training data, even if it is only mentioned once. This could lead to overfitting so it might be better to limit the maximum number of words to e. g. 100k.\n\nThanks for reading. Good luck and have fun in this competition!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":1}