cnn_encoder_nmt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EN49CF_D3Yyh"
   },
   "source": [
    "# A Convolutional Encoder Model for Neural Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_6sBRU3pp-4R"
   },
   "source": [
    "In this tutorial we will demonstrate how to implement a state of the art convolutional encoder sequential decoder (conv2seq) architecture (Published recently at ACL'17. [Link To Paper](http://www.aclweb.org/anthology/P/P17/P17-1012.pdf)) for sequence to sequence modeling using Pytorch. While the aim of the tutorial is to make the audience comfortable with pytorch using this tutorial (with a Conv2Seq implementation as an add on), some familiarity with pytorch (or any other deep learning framework) would definitely be a plus. The agenda of this tutorial is as follows:\n",
    "\n",
    "1. Getting Ready with the data \n",
    "2. Network Definition. This includes\n",
    "    * A Convolution Encoder with residual connections\n",
    "    * An attention based RNN decoder \n",
    "3. Training subroutines\n",
    "4. Model testing and Visualizations\n",
    "\n",
    "This tutorial draws its content/design heavily from [this](http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html) pytorch tutorial for attention based sequence to sequence model for translation. We reuse their data selection/filtering methodology. This helps in focussing more on explaining model architecture and it's translation from formulae to code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ke7hDnU38ox0"
   },
   "source": [
    "## Data Preparation\n",
    "\n",
    "While the paper uses the official WMT data, we stick to a relatively smaller dataset for English to French translation released as part of the Tatoeba project \\[3\\] which is present in the \"data\" directory of this project. We will later apply more filtering to restrict our focus on certain type of short sentences. \n",
    "\n",
    "Some examples of English-French pairs available in the data are:\n",
    "\n",
    "    La prochaine fois, je gagnerai la partie. ==> I will win the game next time.\n",
    "\n",
    "    Fouillez la maison ! ==>  Search the house!\n",
    "\n",
    "    Ne vous faites pas de souci ! Je vous couvre. ==> Don't worry. I've got you covered.\n",
    "\n",
    "    Ma famille n'est pas aussi grande que ça. ==> My family is not that large.\n",
    "\n",
    "    Ça va être serré. ==> It's going to be close.\n",
    "\n",
    "\n",
    "  To get started we first import the necessary libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ME1msUuZ85-K"
   },
   "outputs": [],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "from collections import namedtuple\n",
    "import unicodedata\n",
    "import string\n",
    "import re\n",
    "import random\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SSayaNk99AR7"
   },
   "source": [
    "We now define some constants and variables that we will use later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "DRq2oCIu9QHL"
   },
   "outputs": [],
   "source": [
    "use_cuda = torch.cuda.is_available() # To check if GPU is available\n",
    "MAX_LENGTH = 10 # We restrict our experiments to sentences of length 10 or less\n",
    "embedding_size = 256\n",
    "hidden_size_gru = 256\n",
    "attn_units = 256\n",
    "conv_units = 256\n",
    "num_iterations = 750\n",
    "print_every = 100\n",
    "batch_size = 1\n",
    "sample_size = 1000\n",
    "dropout = 0.2\n",
    "encoder_layers = 3\n",
    "SOS_TOKEN = \"<sos>\"\n",
    "EOS_TOKEN = \"<eos>\"\n",
    "\n",
    "eng_prefixes = (\n",
    "    \"i am \", \"i m \",\n",
    "    \"he is\", \"he s \",\n",
    "    \"she is\", \"she s\",\n",
    "    \"you are\", \"you re \",\n",
    "    \"we are\", \"we re \",\n",
    "    \"they are\", \"they re \"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WEUVdHDd9T0O"
   },
   "source": [
    "Note that while the paper uses different model parameters (longer sentences, larger embedding dimensions etc.) we choose smaller values to make the architecture shallow enough for the small data that we are using."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zYNbUYTY-0xJ"
   },
   "source": [
    "Next, we will define (or rather copy from [2]) some helper functions that will prove to be useful later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "KcJ8Xvbx-9ZB"
   },
   "outputs": [],
   "source": [
    "# Function to convert unicdoe string to plain ASCII\n",
    "# Thanks to http://stackoverflow.com/a/518232/2809427\n",
    "\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "T8mrmYkZ_EAR"
   },
   "outputs": [],
   "source": [
    "# Takes all unicode characters, converts them to ascii\n",
    "# Replaces full stop with space full stop (so that Fire!\n",
    "# becomes Fire !)\n",
    "# Removes everything apart from alphabet characters and\n",
    "# stop characters.\n",
    "\n",
    "def normalizeString(s):\n",
    "    s = unicodeToAscii(s.lower().strip())\n",
    "    s = re.sub(r\"([.!?])\", r\" \\1\", s)\n",
    "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "hirS0_Vy_m4r"
   },
   "outputs": [],
   "source": [
    "# Returns the cuda tensor type of a variable if CUDA is available\n",
    "def check_and_convert_to_cuda(var):\n",
    "    return var.cuda() if use_cuda else var"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "69d-i1GXAONE"
   },
   "source": [
    "We now read the dataset and normalize them. To be able to observe the effects of training on our small dataset quickly, we will restrict our dataset to simpler sentences, which begin with phrases like \"i am\", \"he is\", \"she is\" etc. These prefixes which we will be using to filter our dataset have been defined as the variable `eng_prefixes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 851,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 333,
     "status": "error",
     "timestamp": 1510592446783,
     "user": {
      "displayName": "Dushyanta Dhyani",
      "photoUrl": "//lh3.googleusercontent.com/-hbB9NIM00XM/AAAAAAAAAAI/AAAAAAAAAmM/wTcTHmsn9M0/s50-c-k-no/photo.jpg",
      "userId": "106344207329436262036"
     },
     "user_tz": 300
    },
    "id": "s2QkcTKbUTGD",
    "outputId": "af4165e5-f938-4f7a-9978-21e0a71b9ebb"
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/eng-fra.txt', sep='\\t', names=['english', 'french'])\n",
    "data = data[data.english.str.lower().str.startswith(eng_prefixes)].iloc[:sample_size]\n",
    "\n",
    "data['english'] = data.apply(lambda row: normalizeString(row.english), axis=1)\n",
    "data['french'] = data.apply(lambda row: normalizeString(row.french), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZFSzvUTCVvbJ"
   },
   "source": [
    "We now have a list of sentences which are space separated words. Now, we want to convert these individual words to unique numerical ID's so that each unique word in the vocabulary is represented by a particular integer ID. To do this, we first create a function that does this mapping for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Z_tWJC_qW2UR"
   },
   "outputs": [],
   "source": [
    "Vocabulary = namedtuple('Vocabulary', ['word2id', 'id2word']) # A Named tuple representing the vocabulary of a particular language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "C0AfycANWc8P"
   },
   "outputs": [],
   "source": [
    "def construct_vocab(sentences):\n",
    "    word2id = dict()\n",
    "    id2word = dict()\n",
    "    word2id[SOS_TOKEN] = 0\n",
    "    word2id[EOS_TOKEN] = 1\n",
    "    id2word[0] = SOS_TOKEN\n",
    "    id2word[1] = EOS_TOKEN\n",
    "    for sentence in sentences:\n",
    "        for word in sentence.strip().split(' '):\n",
    "            if word not in word2id:\n",
    "                word2id[word] = len(word2id)\n",
    "                id2word[len(word2id)-1] = word\n",
    "    return Vocabulary(word2id, id2word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CPShuTSyXGuB"
   },
   "source": [
    "Now, generating the vocabulary for source/target language is as simple as "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 185,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 197,
     "status": "error",
     "timestamp": 1510593065889,
     "user": {
      "displayName": "Dushyanta Dhyani",
      "photoUrl": "//lh3.googleusercontent.com/-hbB9NIM00XM/AAAAAAAAAAI/AAAAAAAAAmM/wTcTHmsn9M0/s50-c-k-no/photo.jpg",
      "userId": "106344207329436262036"
     },
     "user_tz": 300
    },
    "id": "lL6TY6JnXN0W",
    "outputId": "0a6f6364-6403-4b58-9ceb-c2d1dc1115ff"
   },
   "outputs": [],
   "source": [
    "english_vocab = construct_vocab(data.english)\n",
    "french_vocab = construct_vocab(data.french)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ci2t5jtnXcJC"
   },
   "source": [
    "The next task is to convert each sentence to a list of ID's from the corresponding vocabulary mapping. We create another helper function for it. Note that we also add a special End of Sentence `<eos>` token to mark the end of sentence. ( At decoding time, we keep generating words until the `<eos>` token has been generated )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "mDZLCa8EgcWW"
   },
   "outputs": [],
   "source": [
    "def sent_to_word_id(sentences, vocab, eos=True):\n",
    "    data = []\n",
    "    for sent in sentences:\n",
    "        if eos:\n",
    "            end = [vocab.word2id[EOS_TOKEN]]\n",
    "        else:\n",
    "            end = []\n",
    "        words = sent.strip().split(' ')\n",
    "        \n",
    "        if len(words) < MAX_LENGTH:\n",
    "            data.append([vocab.word2id[w] for w in words] + end)\n",
    "    return data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kpXT0hL5gwSH"
   },
   "source": [
    "And finally use this function to generate sentences with token ID's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "cIclJjNIg3zU"
   },
   "outputs": [],
   "source": [
    "english_data = sent_to_word_id(data.english, english_vocab)\n",
    "french_data = sent_to_word_id(data.french, french_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vk_odElmhR5o"
   },
   "source": [
    "What we have generated now are python lists where each item in itself is a list of ID's. However, Pytorch expects a Tensor object and so we also perform that required transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "d9ilmM_HhlgS"
   },
   "outputs": [],
   "source": [
    "input_dataset = [Variable(torch.LongTensor(sent)) for sent in french_data]\n",
    "output_dataset = [Variable(torch.LongTensor(sent)) for sent in english_data]\n",
    "\n",
    "if use_cuda: # And if cuda is available use the cuda tensor types\n",
    "    input_dataset = [i.cuda() for i in input_dataset]\n",
    "    output_dataset = [i.cuda() for i in output_dataset]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1MgiTEkkh8tx"
   },
   "source": [
    "We are now done with the required data preprocessing that is compatible with the requirements of our Encoder - Decoder architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YdGzs93giJsd"
   },
   "source": [
    "# Encoder - Decoder Architecture\n",
    "\n",
    "At it's core, an encoder-decoder model uses two neural networks. The first takes in the sentence token by token and produces a sentence representation (A vector of given size, say 512). Once we have this representation (presumably containing the entire semantics of the source sentence), we then use this to generate the corresponding sentence in the target language , word by word. Conventionally, recurrent neural networks have been used for both encoder and decoder [4]  as shown in the figure below ([Image Source](http://colah.github.io/posts/2015-01-Visualizing-Representations/))\n",
    "\n",
    "\n",
    "<img src=\"https://colah.github.io/posts/2015-01-Visualizing-Representations/img/Translation2-RepArrow.png\" width=\"600\" height=\"400\" />\n",
    "\n",
    "This however burdens the encoder by asking it to encode the entire representation of the sentence in a single vector. [5] propose a neural attention mechanism that at decoding time, apart from using this sentence representation, also tries peek at the input to get additional help in performing that decoding step.\n",
    "\n",
    "General architecture of how attention is used to selectively focus on a particular part of input to perform decoding\n",
    "\n",
    "<img src=\"http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/12/Screen-Shot-2015-12-30-at-1.16.08-PM.png\" width=\"200\" height=\"200\" />\n",
    "\n",
    "(Image Source: WildML Blog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zmoLWdu4pOfk"
   },
   "source": [
    "This paper replaces the recurrent encoder in the above architecture and replaces it with a Convolutional Encoder. A big benefit from this is that CNN's are highly parallelizable and thus make the training faster (and as shown in the results of the paper with equal or better performance). We now focus on the implementation of the Convolution Encoder (We recommend everyone to also try the previously mentioned RNN Encoder-Decoder tutorial on PyTorch's official tutorial page). As you will see later, the attention formulation is modified from the above architecture when using Convolutional Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ka9rHziAp5I3"
   },
   "source": [
    "# Convolutional Encoder\n",
    "\n",
    "The main components of the convolution encoder are\n",
    "\n",
    "* A multi-layer convolution neural network with one fixed size filter that gathers information for the given context window.\n",
    "\n",
    "* Residual connections that combine the input of a convolution layer to its output followed by a non-linear transformation. (Note that having no pooling layer after convolution layers is essentiall for incorporating residual connections. Moreover, each input to a convolution layer has to be appropriately padded so that the output-size after applying the convolution operation can remain the same as the original input size and can thus be passed to successive convolution layers whithout reducing the input size )\n",
    "\n",
    "* The architecture has two such Convolution networks\n",
    "  * Conv-a - The output of this encoder is used for creating the attention matrix that is used at decoding time.\n",
    "  * Conv-c - The output of this encoder is attended to (the exact formulation is discussed later) using Conv-a and is then passed to the Decoder\n",
    "  \n",
    "  \n",
    "We will now explain the architecture of in convolution encoder in general (without explicitly referring to Conv-a or Conv-c as they are structurally similar)\n",
    "\n",
    "The input to a convolution encoder is combination (addition in this case) of individual word embeddings and their position embeddings which in the paper is given by $e_j = w_j + l_j $. Both these embeddings are learnt during training. Thus for a sentence *\"La prochaine fois je gagnerai la partie\"* the input to the encoder is\n",
    "\n",
    "| Word         | Position  | Representation  |\n",
    "| -------------|:---------:| -----:|\n",
    "|    La        |  1        |  WordEmbeddingFor(*La*) + PositionEmbeddingFor(*1*)|\n",
    "|    prochaine |  2        |  WordEmbeddingFor(*prochaine*) + PositionEmbeddingFor(*2*)  |\n",
    "| fois         |  3        |  WordEmbeddingFor(*fois*) + PositionEmbeddingFor(*3*)   |\n",
    "|   je         |  4        |  WordEmbeddingFor(*je*) + PositionEmbeddingFor(*4*)|\n",
    "|   gagnerai   |  5        |  WordEmbeddingFor(*gagnerai*) + PositionEmbeddingFor(*5*)|\n",
    "|   la         |  6        |  WordEmbeddingFor(*la*) + PositionEmbeddingFor(*6*)|\n",
    "|partie        |  7        |  WordEmbeddingFor(*partie*) + PositionEmbeddingFor(*7*)|\n",
    "\n",
    "\n",
    "We finally begin with our encoder implementation by defining a barebone architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "TB9C_SNxKE6G"
   },
   "outputs": [],
   "source": [
    "%%script false\n",
    "class ConvEncoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_size, dropout=0.2,\n",
    "                 num_channels_attn=512, num_channels_conv=512, max_len=MAX_LENGTH,\n",
    "                 kernel_size=3, num_layers=5):\n",
    "      pass\n",
    "    def forward(self, position_ids, sentence_as_wordids):\n",
    "      # position_ids refer to position of individual words in the sentence \n",
    "      # represented by sentence_as_wordids. \n",
    "      pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E-kzzPEMLI86"
   },
   "source": [
    "Here we have the constructor with the necessary model parameters. The forward() defines the forward pass of your computational graph. Pytorch handles the backward pass of calculating gradients and updating weights on its own. We now incrementally build our encoder in the following steps. (A point worth mentioning is that while the *position_ids* can be dynamically generated on the fly as part of the computation graph, we pass them as input to make the model code minimal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "adQw_D9bMZpr"
   },
   "outputs": [],
   "source": [
    "%%script false\n",
    "class ConvEncoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_size, dropout=0.2,\n",
    "                 num_channels_attn=512, num_channels_conv=512, max_len=MAX_LENGTH,\n",
    "                 kernel_size=3, num_layers=5):\n",
    "      super(ConvEncoder, self).__init__()\n",
    "      # Here we define the required layers that would be used in the forward pass\n",
    "      self.position_embedding = nn.Embedding(max_len, embedding_size)\n",
    "      self.word_embedding = nn.Embedding(vocab_size, embedding_size)\n",
    "      self.num_layers = num_layers\n",
    "      self.dropout = dropout\n",
    "      \n",
    "      # Convolution Layers\n",
    "      self.conv = nn.ModuleList([nn.Conv1d(num_channels_conv, num_channels_conv, kernel_size,\n",
    "                                      padding=kernel_size // 2) for _ in range(num_layers)])\n",
    "      \n",
    "    def forward(self, position_ids, sentence_as_wordids):\n",
    "      # position_ids refer to position of individual words in the sentence \n",
    "      # represented by sentence_as_wordids. \n",
    "      pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bNWV-oO3Pld0"
   },
   "source": [
    "The reason why we explicily use *nn.ModuleList* and not traditional Python Lists is to allow these modules to be visible to other Pytorch modules if GPU is used.\n",
    "\n",
    "We now define the computational graph of the encoder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "WzHRpgQjQNo2"
   },
   "outputs": [],
   "source": [
    "class ConvEncoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_size, dropout=0.2,\n",
    "                 num_channels_attn=512, num_channels_conv=512, max_len=MAX_LENGTH,\n",
    "                 kernel_size=3, num_layers=5):\n",
    "        super(ConvEncoder, self).__init__()\n",
    "        self.position_embedding = nn.Embedding(max_len, embedding_size)\n",
    "        self.word_embedding = nn.Embedding(vocab_size, embedding_size)\n",
    "        self.num_layers = num_layers\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.conv = nn.ModuleList([nn.Conv1d(num_channels_conv, num_channels_conv, kernel_size,\n",
    "                                      padding=kernel_size // 2) for _ in range(num_layers)])\n",
    "\n",
    "    def forward(self, position_ids, sentence_as_wordids):\n",
    "        # Retrieving position and word embeddings \n",
    "        position_embedding = self.position_embedding(position_ids)\n",
    "        word_embedding = self.word_embedding(sentence_as_wordids)\n",
    "        \n",
    "        # Applying dropout to the sum of position + word embeddings\n",
    "        embedded = F.dropout(position_embedding + word_embedding, self.dropout, self.training)\n",
    "        \n",
    "        # Transform the input to be compatible for Conv1d as follows\n",
    "        # Length * Channel ==> Num Batches * Channel * Length\n",
    "        embedded = torch.unsqueeze(embedded.transpose(0, 1), 0)\n",
    "        \n",
    "        # Successive application of convolution layers followed by residual connection\n",
    "        # and non-linearity\n",
    "        \n",
    "        cnn = embedded\n",
    "        for i, layer in enumerate(self.conv):\n",
    "          # layer(cnn) is the convolution operation on the input cnn after which\n",
    "          # we add the original input creating a residual connection\n",
    "          cnn = F.tanh(layer(cnn)+cnn)        \n",
    "\n",
    "        return cnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DTtX9uADTKwy"
   },
   "source": [
    "The only difference between Conv-a and Conv-c is that of embedding size and number of convolution layers which can be easily adjusted for each using the given constructor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tXrs1QPk1wpg"
   },
   "source": [
    "# Decoder\n",
    "We will now see how to build the decoder component of the translation model. To understand the decoder module, let's focus on the section 2 of the paper. The first paragraph indicates that we are getting a sequence of states, **z** defined as:\n",
    "$$\\mathbf{z} = (z_1, z_2 \\ldots z_m)$$\n",
    "\n",
    "The next paragraph describes how a typical recurrent neural network works. This part is not necessary in understanding the implementation, since PyTorch and other Deep Learning frameworks provide methods for easy construction of these recurrent networks. What is important to understand, however, is the notations they are using for describing the inputs and outputs, since they will be useful in understanding the equations later. The paper uses LSTM as the neural network, however, for this tutorial we are using a GRU instead. Since GRU and LSTM only differ in the internal mechanism of generating hidden states and outputs, this won't affect the implementation a lot.\n",
    "\n",
    "* $h_i$ represents the hidden state/output of the LSTM.\n",
    "* $c_i$ is the input context to the LSTM\n",
    "* $g_i$ is the embedding of the previous output of the LSTM. This gets concatenated with $c_i$ as input to the LSTM\n",
    "\n",
    "The outputs of the encoder module are $cnn_a$ (used in generating attention matrix) and $cnn_c$ (encoded sentence).\n",
    "\n",
    "The next word, $y_{i+1}$ is generated as:\n",
    "$$p(y_{i+1}|y_1, \\ldots, y_i, \\mathbf{x}) = \\text{softmax}(W_oh_{i+1} + b_o)$$\n",
    "\n",
    "For the `softmax` part, we can use PyTorch's `functional` module. For the linear transformation within the `softmax`, we can use `nn.Linear`. The input to this linear transformation is the GRU's hidden state, therefore of the same. The output will be a distribution over the entire output vocabulary, therefore equal to the output vocabulary size.\n",
    "\n",
    "So far, our decoder should like something like this:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "OueQ30y3eQPM"
   },
   "outputs": [],
   "source": [
    "%%script false\n",
    "\n",
    "class AttnDecoder(nn.Module):\n",
    "  def __init__(self, output_vocab_size, hidden_size_gru, embedding_size,\n",
    "               n_layers_gru):\n",
    "    \n",
    "    # This will generate the embedding g_i of previous output y_i\n",
    "    self.embedding = nn.Embedding(output_size, embedding_size)\n",
    "    \n",
    "    # A GRU \n",
    "    self.gru = nn.GRU(hidden_size_gru+embedding_size, hidden_size, n_layers_gru)\n",
    "    \n",
    "    # Dense layer for output transformation\n",
    "    self.dense_o = nn.Linear(hidden_size_gru, output_vocab_size)\n",
    "    \n",
    "  def forward(self, y_i, h_i, cnn_a, cnn_c):\n",
    "    \n",
    "    # generates the embedding of previous output\n",
    "    g_i = self.embedding(y_i)\n",
    "    \n",
    "    gru_output, gru_hidden = self.gru(torch.concat(g_i, input_context), h_i)\n",
    "    # gru_output: contains the output at each time step from the last layer of gru\n",
    "    # gru_hidden: contains hidden state of every layer of gru at the end\n",
    "    \n",
    "    # We want to compute a softmax over the last output of the last layer\n",
    "    output = F.log_softmax(self.dense_o(gru_hidden[-1]))\n",
    "    \n",
    "    # We return the softmax-ed output. We also need to collect the hidden state of the GRU\n",
    "    # to be used as h_i in the next forward pass\n",
    "    \n",
    "    return output, gru_hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1zL1wjgYer2u"
   },
   "source": [
    "In the code snippet above, we haven't included the generation of `input_context` $c_i$. The paper describes the generation of $c_i$ as follows (same section):\n",
    "\n",
    "We first transform the hidden state of GRU $h_i$ to match the size of $g_i$, the embedding of previous output and then add the embedding $g_i$.\n",
    "\n",
    "$$d_i = W_dh_i + b_d + g_i$$\n",
    "\n",
    "Corresponding code (rough - don't copy paste!) will be:\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "```python\n",
    "def __init__():\n",
    "  self.transform_gru_hidden = nn.Linear(gru_hidden_size, embedding_size)\n",
    "\n",
    "def forward():\n",
    "  d_i = self.transform_gru_hidden(gru_hidden[-1]) + g_i\n",
    "```\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Next, we generate the attention matrix $A$ as follows:\n",
    "$$a_{ij} = \\frac{exp \\left(d_i^Tz_j\\right)}{\\sum^m_{t=1}exp\\left(d_i^Tz_t\\right)}$$\n",
    "\n",
    "$z_j$ (described in the next page) is the $j^{th}$ column of $cnn\\_a$.\n",
    "\n",
    "Instead of generating $a_{ij}$ individually, we can generate the entire $\\mathbf{a_i}$ in one go, by modifying the equation slightly:\n",
    "\n",
    "$$\\mathbf{a_i} = \\text{softmax}(d_i^T\\mathbf{z})$$ where $\\mathbf{z}$ now corresponds to the entire $cnn\\_a$. This simplifies our implementation, since we can now quickly multiply matrices instead of iterating over individual vectors and computing the dot products.\n",
    "\n",
    "Pythonically, we can write this roughly as:\n",
    "\n",
    "---\n",
    "\n",
    "```python\n",
    "  a_i = F.softmax(torch.bmm(d_i, cnn_a).view(1, -1))\n",
    "```\n",
    "---\n",
    "\n",
    "Finally, we generate $c_i$ as\n",
    "$$c_i = \\sum_{j=1}^{m}a_{ij}\\left (cnn\\_c(\\mathbf{e})_j \\right )$$\n",
    "\n",
    "$cnn-c(\\mathbf{e})_j$ corresponds to `cnn_c` that we receive as the encoder input. As before, we can transform the equation a bit so that it becomes easier to implement it:\n",
    "$$c_i = \\mathbf{a}_i \\left (cnn\\_c \\right)$$ \n",
    "\n",
    "---\n",
    "\n",
    "```python\n",
    "  c_i = torch.bmm(a_i.view(1, 1, -1), cnn_c.transpose(1, 2))\n",
    "```\n",
    "---\n",
    "\n",
    "A bit of implementation trick here - `a_i` has dimension `(sequence_length,)`. `cnn_c` has dimension `embedding_size x sequence_length`. Now, for `a_i` and `cnn_c` to be multiplied together, we need to make them compatible for multiplication. Therefore, we transpose `cnn_c` to make it `sequence_length x embedding_size` and reshape `a_i` to `1 x sequence_length`\n",
    "\n",
    "We are almost done here with the decoder component. Few things we need to do to complete the implementation:\n",
    "* Make sure `__init__` and `forward` funcitons have all the arguments which are needed.\n",
    "* Add dropouts for embedding and decoder output $h_i$ (section 4.3, last line)\n",
    "* Add a function to initialize the hidden units of the GRU to zero after every sentence. (section 4.2, second line)\n",
    "\n",
    "Putting everything together, our decoder module now looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "VnSwjSdp8l40"
   },
   "outputs": [],
   "source": [
    "class AttnDecoder(nn.Module):\n",
    "  \n",
    "  def __init__(self, output_vocab_size, dropout = 0.2, hidden_size_gru = 128,\n",
    "               cnn_size = 128, attn_size = 128, n_layers_gru=1,\n",
    "               embedding_size = 128, max_sentece_len = MAX_LENGTH):\n",
    "\n",
    "    super(AttnDecoder, self).__init__()\n",
    "    \n",
    "    self.n_gru_layers = n_layers_gru\n",
    "    self.hidden_size_gru = hidden_size_gru\n",
    "    self.output_vocab_size = output_vocab_size\n",
    "    self.dropout = dropout\n",
    "    \n",
    "    self.embedding = nn.Embedding(output_vocab_size, hidden_size_gru)\n",
    "    self.gru = nn.GRU(hidden_size_gru + embedding_size, hidden_size_gru,\n",
    "                      n_layers_gru)\n",
    "    self.transform_gru_hidden = nn.Linear(hidden_size_gru, embedding_size)\n",
    "    self.dense_o = nn.Linear(hidden_size_gru, output_vocab_size)\n",
    "\n",
    "    self.n_layers_gru = n_layers_gru\n",
    "    \n",
    "  def forward(self, y_i, h_i, cnn_a, cnn_c):\n",
    "    \n",
    "    g_i = self.embedding(y_i)\n",
    "    g_i = F.dropout(g_i, self.dropout, self.training)\n",
    "    \n",
    "    d_i = self.transform_gru_hidden(h_i) + g_i\n",
    "    a_i = F.softmax(torch.bmm(d_i, cnn_a).view(1, -1))\n",
    "  \n",
    "    c_i = torch.bmm(a_i.view(1, 1, -1), cnn_c.transpose(1, 2))\n",
    "    gru_output, gru_hidden = self.gru(torch.cat((g_i, c_i), dim=-1), h_i)\n",
    "    \n",
    "    gru_hidden = F.dropout(gru_hidden, self.dropout, self.training)\n",
    "    softmax_output = F.log_softmax(self.dense_o(gru_hidden[-1]))\n",
    "    \n",
    "    return softmax_output, gru_hidden\n",
    "\n",
    "\n",
    "  # function to initialize the hidden layer of GRU. \n",
    "  def initHidden(self):\n",
    "    result = Variable(torch.zeros(self.n_layers_gru, 1, self.hidden_size_gru))\n",
    "    if use_cuda:\n",
    "        return result.cuda()\n",
    "    else:\n",
    "        return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pklVpEz082db"
   },
   "source": [
    "# Training the Model\n",
    "\n",
    "We now describe the process for training the network on the parallel dataset. In general, the steps involved in training a PyTorch model may be outlined as follows:\n",
    "1. Initialize the network weights\n",
    "2. Define and initialize the optimizers\n",
    "3. Define and initialize the loss criterion\n",
    "4. Repeat till convergence:\n",
    "   * Make a forward pass through the network\n",
    "   * Use the loss criterion to compute loss\n",
    "   * Use the optimizer to compute the gradients\n",
    "   * Backpropogate\n",
    "   \n",
    "We will describe (and implement) each of the steps described above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5IcyVANr1v9l"
   },
   "source": [
    "## Initialize the network weights\n",
    "\n",
    "This is easy. We create the objects corresponding to the `ConvEncoder` and `AttnDecoder` classes we have created above. Then we initialize the weights for different parts of the network as follows (section 4.2):\n",
    "* Convolution Layers: Samples from uniform distribution in range $(-kd^{-0.5}, kd^{0.5})$\n",
    "* Others: Samples from uniform distribution in range $(-0.05, 0.05)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "FxFB-YaC3P-x"
   },
   "outputs": [],
   "source": [
    "def init_weights(m):\n",
    "  \n",
    "    if not hasattr(m, 'weight'):\n",
    "        return\n",
    "    if type(m) == nn.Conv1d:\n",
    "        width = m.weight.data.shape[-1]/(m.weight.data.shape[0]**0.5)\n",
    "    else:\n",
    "        width = 0.05\n",
    "        \n",
    "    m.weight.data.uniform_(-width, width)\n",
    "\n",
    "\n",
    "encoder_a = ConvEncoder(len(french_vocab.word2id), embedding_size, dropout=dropout,\n",
    "                        num_channels_attn=attn_units, num_channels_conv=conv_units,\n",
    "                        num_layers=encoder_layers)\n",
    "encoder_c = ConvEncoder(len(french_vocab.word2id), embedding_size, dropout=dropout,\n",
    "                        num_channels_attn=attn_units, num_channels_conv=conv_units,\n",
    "                        num_layers=encoder_layers)\n",
    "decoder = AttnDecoder(len(english_vocab.word2id), dropout = dropout,\n",
    "                       hidden_size_gru = hidden_size_gru, embedding_size = embedding_size,\n",
    "                       attn_size = attn_units, cnn_size = conv_units)\n",
    "\n",
    "if use_cuda:\n",
    "    encoder_a = encoder_a.cuda()\n",
    "    encoder_c = encoder_c.cuda()\n",
    "    decoder = decoder.cuda()\n",
    "\n",
    "encoder_a.apply(init_weights)\n",
    "encoder_c.apply(init_weights)\n",
    "decoder.apply(init_weights)\n",
    "\n",
    "encoder_a.training = True\n",
    "encoder_c.training = True\n",
    "decoder.training = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "urCpdynx5aC8"
   },
   "source": [
    "## Define and Initialize the Optimizers\n",
    "We will use Adam optimzer `torch.optim.Adam` with a learning rate of $10^{-4}$. Here's how we can do it:\n",
    "\n",
    "---\n",
    "\n",
    "```python\n",
    "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
    "decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)\n",
    "\n",
    "encoder_optimizer.zero_grad()\n",
    "decoder_optimizer.zero_grad()\n",
    "```\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eF_iwfBB61CJ"
   },
   "source": [
    "## Define and Initialize Loss Criterion\n",
    "We will be using Negative Log Likelihood (also known as Cross-entropy loss) as the loss criterion for our network.\n",
    "\n",
    "---\n",
    "```python\n",
    "criterion = nn.NLLLoss()\n",
    "```\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qcbc4ybc82eh"
   },
   "source": [
    "## Training Steps\n",
    "\n",
    "We will define two functions:\n",
    "* `train`: This corresponds to one step of training. It will make a forward pass for one batch, compute loss, compute and backpropagate the gradients.\n",
    "* `trainIters`: This will sample a batch and call the train function, in a loop.\n",
    "\n",
    "Although the paper suggests using [beam search](https://en.wikipedia.org/wiki/Beam_search) while generating the output sentence, we will use greedy decoding instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "D8yhutcS-X7Y"
   },
   "outputs": [],
   "source": [
    "def trainIters(encoder_a, encoder_c, decoder, n_iters, batch_size=32, learning_rate=1e-4, print_every=100):\n",
    "  \n",
    "    encoder_a_optimizer = optim.Adam(encoder_a.parameters(), lr=learning_rate)\n",
    "    encoder_c_optimizer = optim.Adam(encoder_c.parameters(), lr=learning_rate)\n",
    "    decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)\n",
    "    \n",
    "    # Sample a training pair\n",
    "    training_pairs = list(zip(*(input_dataset, output_dataset)))\n",
    "    \n",
    "    criterion = nn.NLLLoss()\n",
    "    \n",
    "    \n",
    "    print_loss_total = 0\n",
    "    \n",
    "    # The important part of the code is the 3rd line, which performs one training\n",
    "    # step on the batch. We are using a variable `print_loss_total` to monitor\n",
    "    # the loss value as the training progresses\n",
    "    \n",
    "    for itr in range(1, n_iters + 1):\n",
    "        training_pair = random.sample(training_pairs, k=batch_size)\n",
    "        input_variable, target_variable = list(zip(*training_pair))\n",
    "        \n",
    "        loss = train(input_variable, target_variable, encoder_a, encoder_c,\n",
    "                     decoder, encoder_a_optimizer, encoder_c_optimizer, decoder_optimizer,\n",
    "                     criterion, batch_size=batch_size)\n",
    "        \n",
    "        print_loss_total += loss\n",
    "\n",
    "        if itr % print_every == 0:\n",
    "            print_loss_avg = print_loss_total / print_every\n",
    "            print(print_loss_avg)\n",
    "            print_loss_total=0\n",
    "    print(\"Training Completed\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "WThypCZkzS_S"
   },
   "outputs": [],
   "source": [
    "def train(input_variables, output_variables, encoder_a, encoder_c, decoder,\n",
    "          encoder_a_optimizer, encoder_c_optimizer, decoder_optimizer, criterion, \n",
    "          max_length=MAX_LENGTH, batch_size=32):\n",
    "    \n",
    "  # Initialize the gradients to zero\n",
    "  encoder_a_optimizer.zero_grad()\n",
    "  encoder_c_optimizer.zero_grad()\n",
    "  decoder_optimizer.zero_grad()\n",
    "\n",
    "  for count in range(batch_size):\n",
    "    # Length of input and output sentences\n",
    "    input_variable = input_variables[count]\n",
    "    output_variable = output_variables[count]\n",
    "\n",
    "    input_length = input_variable.size()[0]\n",
    "    output_length = output_variable.size()[0]\n",
    "\n",
    "    loss = 0\n",
    "\n",
    "    # Encoder outputs: We use this variable to collect the outputs\n",
    "    # from encoder after each time step. This will be sent to the decoder.\n",
    "    position_ids = Variable(torch.LongTensor(range(0, input_length)))\n",
    "    position_ids = position_ids.cuda() if use_cuda else position_ids\n",
    "    cnn_a = encoder_a(position_ids, input_variable)\n",
    "    cnn_c = encoder_c(position_ids, input_variable)\n",
    "    \n",
    "    cnn_a = cnn_a.cuda() if use_cuda else cnn_a\n",
    "    cnn_c = cnn_c.cuda() if use_cuda else cnn_c\n",
    "\n",
    "    prev_word = Variable(torch.LongTensor([[0]])) #SOS\n",
    "    prev_word = prev_word.cuda() if use_cuda else prev_word\n",
    "\n",
    "    decoder_hidden = decoder.initHidden()\n",
    "\n",
    "    for i in range(output_length):\n",
    "      decoder_output, decoder_hidden = \\\n",
    "          decoder(prev_word, decoder_hidden, cnn_a, cnn_c)\n",
    "      topv, topi = decoder_output.data.topk(1)\n",
    "      ni = topi[0][0]\n",
    "      prev_word = Variable(torch.LongTensor([[ni]]))\n",
    "      prev_word = prev_word.cuda() if use_cuda else prev_word\n",
    "      loss += criterion(decoder_output,output_variable[i])\n",
    "\n",
    "      if ni==1: #EOS\n",
    "        break\n",
    "\n",
    "  # Backpropagation\n",
    "  loss.backward()\n",
    "  encoder_a_optimizer.step()\n",
    "  decoder_optimizer.step()\n",
    "\n",
    "  return loss.data[0]/output_length\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zUY3SZ8hj_YA"
   },
   "source": [
    "To finally start the training, we simply call the trainIters method. (Be patient as the training will take time depending upon your machine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "pAPeFh3tkFfB"
   },
   "outputs": [],
   "source": [
    "trainIters(encoder_a,encoder_c, decoder, num_iterations, print_every=print_every, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gx7CNaqv9CPG"
   },
   "source": [
    "# Evaluation\n",
    "\n",
    "The evaluation function will be very similar to train function minus the backpropagation part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "QSz6ywQDEn1z"
   },
   "outputs": [],
   "source": [
    "def evaluate(sent_pair, encoder_a, encoder_c, decoder, source_vocab, target_vocab, max_length=MAX_LENGTH):\n",
    "    source_sent = sent_to_word_id(np.array([sent_pair[0]]), source_vocab)\n",
    "    if(len(source_sent) == 0):\n",
    "        return\n",
    "    source_sent = source_sent[0]\n",
    "    input_variable = Variable(torch.LongTensor(source_sent))\n",
    "    \n",
    "    if use_cuda:\n",
    "        input_variable = input_variable.cuda()\n",
    "        \n",
    "    input_length = input_variable.size()[0]\n",
    "    position_ids = Variable(torch.LongTensor(range(0, input_length)))\n",
    "    position_ids = position_ids.cuda() if use_cuda else position_ids\n",
    "    cnn_a = encoder_a(position_ids, input_variable)\n",
    "    cnn_c = encoder_c(position_ids, input_variable)\n",
    "    cnn_a = cnn_a.cuda() if use_cuda else cnn_a\n",
    "    cnn_c = cnn_c.cuda() if use_cuda else cnn_c\n",
    "    \n",
    "    prev_word = Variable(torch.LongTensor([[0]])) #SOS\n",
    "    prev_word = prev_word.cuda() if use_cuda else prev_word\n",
    "\n",
    "    decoder_hidden = decoder.initHidden()\n",
    "    target_sent = []\n",
    "    ni = 0\n",
    "    out_length = 0\n",
    "    while not ni==1 and out_length < 10:\n",
    "        decoder_output, decoder_hidden = \\\n",
    "            decoder(prev_word, decoder_hidden, cnn_a, cnn_c)\n",
    "\n",
    "        topv, topi = decoder_output.data.topk(1)\n",
    "        ni = topi[0][0]\n",
    "        target_sent.append(target_vocab.id2word[ni])\n",
    "        prev_word = Variable(torch.LongTensor([[ni]]))\n",
    "        prev_word = prev_word.cuda() if use_cuda else prev_word\n",
    "        out_length += 1\n",
    "        \n",
    "    print(\"Source: \" + sent_pair[0])\n",
    "    print(\"Translated: \"+' '.join(target_sent))\n",
    "    print(\"Expected: \"+sent_pair[1])\n",
    "    print(\"\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9APo5sY1VwEY"
   },
   "source": [
    "To evaluate the learned model, simply execute the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 236,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 240,
     "status": "error",
     "timestamp": 1510609482120,
     "user": {
      "displayName": "Dushyanta Dhyani",
      "photoUrl": "//lh3.googleusercontent.com/-hbB9NIM00XM/AAAAAAAAAAI/AAAAAAAAAmM/wTcTHmsn9M0/s50-c-k-no/photo.jpg",
      "userId": "106344207329436262036"
     },
     "user_tz": 300
    },
    "id": "TWL7YgVYV12M",
    "outputId": "65f94a27-321b-48dc-e179-383c427f281e"
   },
   "outputs": [],
   "source": [
    "encoder_a.training = False\n",
    "encoder_c.training = False\n",
    "decoder.training = False\n",
    "samples = data.sample(n=100)\n",
    "for (i, row) in samples.iterrows():\n",
    "    evaluate((row.french, row.english), encoder_a, encoder_c, decoder, french_vocab, english_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wY_JpN9gwJt1"
   },
   "source": [
    "## References\n",
    "\n",
    "1) Jonas  Gehring,  Michael  Auli,  David  Grangier,  and Yann  Dauphin.  2017. ** A  Convolutional  Encoder Model for Neural Machine Translation.**   In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: LongPapers). Association for Computational Linguistics,Vancouver,  Canada,  pages  123–135\n",
    "[http://www.aclweb.org/anthology/P17-1012](http://www.aclweb.org/anthology/P17-1012)\n",
    "\n",
    "2) [** Translation with a Sequence to Sequence Network and Attention **](http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html) - Official PyTorch Tutorial \n",
    "\n",
    "3) [**Tatoeba Project**](https://tatoeba.org/eng) (Downloaded from [http://www.manythings.org/anki/](http://www.manythings.org/anki/))\n",
    "\n",
    "4) Sutskever, I., Vinyals, O., and Le, Q. (2014). **Sequence to sequence learning with neural networks**. In Advances in Neural Information Processing Systems (NIPS 2014)\n",
    "\n",
    "5) Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. **Neural machine translation by jointly learning to align and translate** arXiv:1409.0473 [cs.CL], September 2014"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "NIPS_MLTrain.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}