Attention Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention Basics\n",
    "In this notebook, we look at how attention is implemented. We will focus on implementing attention in isolation from a larger model. That's because when implementing attention in a real-world model, a lot of the focus goes into piping the data and juggling the various vectors rather than the concepts of attention themselves.\n",
    "\n",
    "We will implement attention scoring as well as calculating an attention context vector.\n",
    "\n",
    "## Attention Scoring\n",
    "### Inputs to the scoring function\n",
    "Let's start by looking at the inputs we'll give to the scoring function. We will assume we're in the first step in the decoging phase. The first input to the scoring function is the hidden state of decoder (assuming a toy RNN with three hidden nodes -- not usable in real life, but easier to illustrate):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "dec_hidden_state = [5,1,20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize this vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbe415ff400>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEYCAYAAACZYo4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADRlJREFUeJzt3XuMXOV5x/Hvzwbqmo2NGyvgC9TpBVKVxNACSkqgtFDFKFZMI2wRKbFJUZdWDYHmZpwqIlalFtoImqZSywpbISoiFyCFRCqF0FKXRDYYc69tkpTUcY25JIbi1MExfvrHDLBe3t3xzs7M8+7s74MsvLM7My/m6+ecOTt7jiICs5GmZS/A6uQwrMhhWJHDsCKHYUUOw4ocRh+RdLykf5O0VdITki5v3v4Lku6W9N3mv+e0fCwfx+gfkuYB8yJii6Q3AQ8CFwAXAz+OiKslXQnMiYjVYz2WJ0YfiYinI2JL8/cvAVuBBcAy4Mbml91II5YxeWL0KUmLgA3AycCOiDhm2Of2RMSYm5Mjurq6Bpf3Og3/YK3WjvvP5rN89lJgcNhNQxExdMiTSAPArcAVEfG/0iFPe1h6EQa7/nlXL56mavPPn9+Rx2lGMDTa5yUdSSOKmyLitubNz0iaFxFPN/dDnm31PN7HyKQ2fo31cI3RsA7YGhHXDvvUHcCq5u9XAbe3WlpPJoaVtTPiWzgT+BDwmKSHm7d9Grga+KqkS4AdwPJWD+QwMnW4i4i4b4xHPXc8j+UwEmlaxydGxziMROr0yOggh5Gp3i4cRqYu7Hx2jMPIVG8XDiOTJ4aV1duFw8jkiWFl9XbhMDJ5YlhZvV04jEw+8mll9XbhMDJ5H8PK6u3CYWTyxLCyertwGJk8Mays3i4cRiZPDCurtwuHkckTw4ochpXV24XDyOSJYWX1duEwMnliWFm9XTiMTDVPDJ8fw4o8MRLVPDH6PoyL1l7EzBkzmaZpTJ8+nes/fn32kl5Xbxf9HwbAdX9yHbMHZmcv4w0m9cSQ9DYa54lcQOMMfLuAOyJia5fX1v/q7WLsnU9Jq4Ev0/hPuB94oPn7m5tnmK2eJD75D59k8HODfOM738heziEkjftXr7SaGJcAvx4RPxt+o6RrgSdonPSral+4/AvMnT2XPS/t4RN//wlOOPYEFv/y4uxlNUzWiQEcBEonqJzX/FyRpEFJmyVtHhoa9ZSUPTF39lwA5rxpDme9/Sy2/fe21PUMN5knxhXAPZK+C/ywedsJwK8AHxntTiNOUhpZJ4Dd9/I+IoKZM2ay7+V9bN6+mZXvWZmylqKKJ8aYYUTEnZJOBM6gsfMpYCfwQES80oP1Tciel/bwmfWfAeCVg69w3m+cxxm/dkbyql43qV+VRMRBYGMP1tJx8+fOZ92n1mUvY3T1djE1jmPUyj/UbGX1duEwMtV8ZmB/dzVRN16uSlov6VlJj4+4/TJJ25vXSvurVo/jiZGpOwPji8DfAV967Wmk36HxbY13RMTLkt7S6kEcRqJuvFyNiA3Ny14N98fA1RHxcvNrfCGbqnX4QjZjOBE4S9ImSf8u6fRWd/DESNTOxJA0SItrohUcAcwB3gmcTuOiNr8UY1wp0WFMMq2uiTaKncBtzRDul3QQmAs8N9odvClJ1MNvov0T8LvN5zwROAp4fqw7eGJk6sKrEkk3A+cAcyXtBK4C1gPrmy9h9wOrxtqMgMNI1aVXJR8Y5VMfHM/jOIxM9R74dBiZJvW33a2L6u3CYWTyxLCyertwGJlqnhg+wGVFnhiJap4YDiNTvV04jEyeGFZWbxcOI5MnhhXV/C5xh5Gp3i4cRiZvSqys3i4cRib/7KqV1duFw8jkfQwrq7cLh5HJE8PK6u3CYWTyxLCyervoTRjzzy+dKtQ8Mays3i56FEbO+V/rUhiaPvJpZfV24TAyeR/DyurtwmFk8sSwsnq7cBiZPDGsyG8GtrJ6u3AYmXyAy8rq7cJhZPLOp5XV24XDyOSJYWX1duEwMtU8MXwOrkxduF5J6dJXkv5a0jZJj0r6uqRjWj2Ow0jUpasPfBFYMuK2u4GTI+IdwJPAmlYP4jD6TERsAH484ra7IuJA88ONwMJWj+N9jERJ+xh/AHyl1Rd5YmRqYx9D0qCkzcN+DZYfvPB00p8BB4CbWn2tJ0aidiZGm5e+QtIqYClwbquL2IDDyNWjLYmkJcBq4Lcj4v8O5z4OI1E39jFGufTVGuDngLubz7kxIv5orMdxGJm6MDFGufTVuvE+jsNIVPORT4eRqd4uHEammieGj2NYkSdGIr9L3Ipq3pQ4jEz1duEwMnliWFm9XTiMTDVPjL5/ubrmmjW86/ffxdIPL81eyht14a19ndL3Ybx/yfu54ZobspdR1KW39nVE34dx+uLTmT1rdvYyyvpxYkj6cCcXMhWpjX96ZSITY+1onxj+9rOhoXG/2WjqqHhijPmqRNKjo30KOHa0+414+1n4PJ9lNb8qafVy9VjgPcCeEbcL+E5XVjSV1NtFyzC+CQxExMMjPyHp3q6sqMM+9ucf4/6H72fPi3s4e/nZXHbxZSx/7/LsZQF1TwwdxhuGJ8qbEnj1lNGHlLBh9YZx/+Gffc3ZPanJRz4T1TwxHEamertwGJk8Mays3i4cRiafztHK6u3CYWTym4GtyDufVlZvFw4jkyeGldXbhcPI5IlhZfV24TAy+QCXldXbhcPI5H0MK6u3C4eRyRPDyurtwmFk8sSwsnq7cBiZap4Yff9DzdYeT4xEnhhW1p1rov2ppCckPS7pZkkz2lmaw0jU6ROnSFoAfBQ4LSJOBqYDF7WzNm9KMnVnS3IE8POSfgbMhPZ+QNQTI1GnJ0ZE/A/wOWAH8DTwYkTc1c7aHEYiTdP4f41xTTRJc4BlwFtp/Bj10ZI+2M7avCnJ1MampMU10c4DnoqI5wAk3Qb8FvCP430eh5GoCy9XdwDvlDQT2AecC2xu54G8KekjEbEJuAXYAjxG4/9vWydB88RI1I0DXBFxFY0L5E2Iw8hU74FPh5Gp5kPivQljfk+eZfKptwtPjExTfmKs1agnEZ4yrorC/mC9XXhiZJryE8NGUW8XDiOTJ4aV1duFw8jkH2q2snq7cBiZvI9hZfV24TAyeWJYWb1dOIxMnhhWVPMpo/3WPivyxEjkTYmV1duFw8jkiWFl9XbhMDJ5YlhZvV04jEyeGFZWbxcOI5MnhpXV24XDyOS39llZvV04jEzex7CyertwGJk8Mays3i4cRiZPDCurtwuHkanmieH3fFqRJ0aimt8l3ndhzFo4iwu+dAEDxw0QB4MtQ1vY9LebmDFnBhd+5UKOWXQML/zgBW5ZcQs/feGnqWuteVPSd2EcPHCQuz5+F7sf2s1RA0cx+OAg37/7+5xy8Sk8dc9TfPuab3Pm6jN595Xv5ltXfit3sfV20XofQ9LbJJ0raWDE7Uu6t6z27d29l90P7QZg/979PLf1OWYtmMVJy07ikRsfAeCRGx/hpAtOylwm0PnLUnTSmGFI+ihwO3AZ8LikZcM+/RfdXFgnzP7F2cw7dR47N+1k4NgB9u7eCzTiOfotRyevjq5c+gpA0nRJD0n6ZrtLa7Up+UPgNyNir6RFwC2SFkXE5w9/mTmOPPpIVty6gjuvuJP9L+3PXk5RFyfA5cBWYFa7D9BqUzI9IvYCRMQPgHOA8yVdyxhhDL/YytBQWye/n5BpR0xjxa0reOymx9j29W0A7H1mLwPHNbaGA8cN8JNnf9Lzdb1Bdy6WtxB4L3DDRJbWKozdkk559YNmJEuBucDbR7tTRAxFxGkRcdrg4OBoX9Y171v3Pp7f+jwbr9v42m1P3vEki1ctBmDxqsVsv317z9c1Upf2Mf4G+BRwcCJra7UpWQkcGH5DRBwAVkq6fiJP3C3Hn3k8i1cu5plHn+HShy4F4J5P38N9V9/HhV+9kFMvOZUXd7zI15Z/LXmltLUxbl7qavjftqHmVY+QtBR4NiIelHTOhJYWERO5/+EInzL6tVNGH5LCj/7jR+P+w3/zWW8eaxP+l8CHaPxlnkFjH+O2iBj3ddF8SDxTh/cxImJNRCyMiEU0rrf6r+1EAX14gGsy8ZFPK+tiFxFxL3Bvu/d3GIk8Mays3i4cRiZPDCurtwuHkckTw8rq7cJhZPIPNVtZvV04jEx+M7AVeefTyurtwmFk8sSwsnq7cBiZap4YfqOOFXliJKp5YjiMTPV24TAyeWJYWb1dOIxMnhhWVm8XDiOTJ4aV1duFw8hU88TwkU8r8sRIVPPEcBiZ6u3CYWTyxLCyertwGJk8MazI7xK3snq7cBiZat6U9OTkbN1+gknk0BJ2tfFnM783c6YXYVRB0uCrpz201qbSIfHen4l2EptKYdg4OAwrmkpheP9iHKbMzqeNz1SaGDYOfR+GpCWStkv6nqQrs9czWfT1pkTSdOBJ4PeAncADwAci4j9TFzYJ9PvEOAP4XkT8V0TsB74MLGtxH6P/w1gA/HDYxzubt1kL/R5G6fsK/bvt7KB+D2MncPywjxcCu5LWMqn0exgPAL8q6a2SjqJx1Z87ktc0KfT1+zEi4oCkjwD/AkwH1kfEE8nLmhT6+uWqta/fNyXWJodhRQ7DihyGFTkMK3IYVuQwrMhhWNH/A5n5tJP6+5uJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe69001f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Let's visualize our decoder hidden state\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(dec_hidden_state)), annot=True, cmap=sns.light_palette(\"purple\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first scoring function will score a single annotation (encoder hidden state), which looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = [3,12,45] #e.g. Encoder hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbe69001e80>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEXCAYAAABoNDzDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADbRJREFUeJzt3X2QleV9xvHvxQJ1Aak24AvgC011kNFIq6BtrDWKCZNadWjaBtImpYk6TXxJ1YJpJ2NNZyxkbMwfnVpRSXR8q1UaLZOSJlGiCQnYGDQovjAJQSQEbNSENi7s8usf56DL5t49e5Zzzn3v2eszc2b2vD3nx+E69+95nrN734oIzPoalbsAK5ODYUkOhiU5GJbkYFiSg2FJDoYlORiW5GBY0ugWvIZPrb5NB1y7V/W/NwtDtR908FoRDPjF9pa8TNE6p+SuoC6tCYb1oyUf/iFxMHKSg2FJDoalqNyDwnIrs6w8YmTlVmIp3vm0NAfDkhwMS3IwLMX7GJbmYFiSg2EpBbcSn/m0JI8YWZU7YjgYORXcShyMrBwMS3IwLMWtxNIcDEtyMCzFrcTSyg2Gz3zmJNV/GdRm1SHpe5JWVa9Pl7RO0kuS/lXS2FrbcDCy0hAug3IVsKnX9WXAzRFxAvAa8NFaG3Awsmp8MCRNA34fuL16XcC5wIPVh9wJXFxrOw5GTkNoJZIulfTfvS6X9tnq54HFwL7q9XcAr0dEd/X6NmBqrdK885lV/TufEbEcWJ7cmnQBsDMivivpnAFepOZf2TsYWTX8qOTdwIWS3g8cAkykMoIcJml0ddSYBtScfsCtJKvG7mNExKciYlpEHA98EHg0Ij4EPAZ8oPqwjwAP16rMwcipSYerCUuAqyVtprLPcUetJ7iVtKmIWAOsqf78A2BOPc93MLIq98xnWwejq2sPH/qLq9izdw893T28b+7vceXHF+Uu623+riSPsWPHcOdtn2P8uE727u1m4aIrOPusM5j1rpm5S6tyMLKQxPhxnQB0d3fT3d1T2Ie0qGIOUDMYkmYAF1E5WxZUjoEfiYhNAz6xED09PcxfcBlbX36FhX9yMaeeUspoQdGtZMDDVUlLgPupRHs98GT15/skXdf88g5eR0cHDz9wO9/4yr/xzMbneXHzD3OX1EvTvkQ7aLXOY3wUmB0RSyPi7uplKZVDn36/oet9Pn/58uTZ25abOHECZ5w+iye+tT53Kb2UG4xarWQfMAX4UZ/bj+btL2l+SZ/z+ZFrAtif/vR1Ro8ezcSJE3jzzS7WrvsulyxakKWWpIJbSa1gfBL4uqSXgJertx0L/AZweTMLa4Sdr/4P1316KT379hH79jHvvefwnrN/O3dZvZQbDNVaRVHSKCqtYyqVf8k24MmI6Bnka2QbMYpSmTL6wCSsmln/XOIXPNeSNNU8KomIfcB3WlDLyFPugNHe5zHKV24yHIycCp4Z2MHIyiOGJTkYljKMz2NYUzkYllRuMMrdLbasPGLk5H0MS3MwLMnBsBS3EktzMCzJwbAUtxJLczAsycGwlHJz4WDkVW4yHIysHAxL8VGJpTkYluRgWIp/S9zSPGJYSsE7n+WOZSNCY6dBkHSIpPWSnpb0rKQbqrffI+kFSRslrZA0plZlDkZ76QLOjYhTgVnAPElnAvcAM4BTgE7gY7U25FaSVWNbSVSmLthdvTqmeomI+PJbryitpzJt9IA8YuTUhJmBq4vYbAB2Al+NiHW97hsD/BmwutZ2HIys6t/HqLUsRUT0RMQsKqPCHEkn97r7n4HHI+KJWpW5lWRVfysZaFmKPo97XdIaYB6wUdL1wGTgssG8jkeMnBrcSiRNlnRY9edOYC7wvKSPAe8DFlQnwqnJI0ZWDT+PcTRwp6QOKh/6ByJilaRuKhPsfbuyEhYrI+IzA23Iwciq4UclzwC/mbi97v/n1gSjMjGZ9VXwmU+PGFmN9GBs/3Ltx7S7Ke/PXUFdPGLk5FZiaQ6GJTkYluJWYmkOhiU5GJbiXwa2tHJHjHIja1l5xMjJRyWW5mBYkoNhKW4lluZgWJKDYSluJZZWbjB8gsuSPGLk5FZiaQ6GJTkYluJWYmkOhiU5GJbiVmJpDoYlORiWUm4uHIy8yv1GwsHIqtwhw8HIyUclluZgWJKDYSluJa3zqWX3seY7z/GOwyaw6gtLAFj2L4/w2NpnGTOmg2OnTOIflixg4oTOzJVCySNGucdLQzR/3hxuX3bALMq8+7QTWfWFxfzHHYs5ftpkbr3na5mq66vhy1IcI+kxSZuqy1Jc1ef+ayWFpEm1Kmu7YMw+9Z386sTxB9x21uwZjO7oAGDWzOPYsev1HKX9ssZPMt8NXBMRJwFnAp+QNLPyUjoGOB/YOpjShhwMSYuG+tycHvrPdZx9xkm5y2iKiPhxRDxV/fnnwCZgavXum4HFQAxmWwczYtzQ3x29Z8hfvrzmfOgtc8vdX6Wjo4ML556Wu5Sqxq8+8NaWpeOpzBK8TtKFwCsR8fRgKxtw51PSMwP8i47s73l9ZsiPEub5/PfV61nz7Wf54j9+HJVyNDCEOgaz+oCkCcBDwCeptJe/Bd5bz+vUOio5ksqs9a/1fW1gbT0vlNPj6zdx2/2PcvfnL6fzkLG5y+ml8QGtLlbzEHBPRKyUdAowHXi6+oGYBjwlaU5E7OhvO7WCsQqYEBEbEgWsGWrxzXT139/F+g2bee2N/+XsP/o7rvjzeSy/9+vs2dvNomtvAeDUmcfxmav/OHOl0OhgqPI/fwewKSI+BxAR3weO6PWYLcDpEfHqgNuqLKPVVEW0kuwqU0YfmIQNS+p/82ct6zdNks4CngC+D+xfl+Rv+qyJtoVBBKPtTnANLw1fluKbtTYaEccPZlsORlaF7AQnOBg5lXJ0lOBgZFVuMNrulLg1hkeMnNxKLK3cAdvByMojhqW4lViag2FJDoaluJVYWrnBKPd4ybLyiJGTW4mlORiW5GBYiluJpTkYluRgWIpbiaWVGwyf4LIkjxg5uZVYmoNhSQ6GpbiVWJqDYUnlHhQ6GDm5lVhaucEodyyzrDxi5ORWYmkOhiWN9GBU5p+yvtxKLG2kB+Pect+AllmYmqCv3PfFI0ZOBbcSn8fIqrHLUgBIWiFpp6SNfW6/QtIL1eUqPltrOx4xsmrKiPFF4J+Au956Fek9wEXAuyKiS9IR/Tz3LQ5GTk1oJRHxeHXlgd7+ElgaEV3Vx+ystR23kmFmsMtS9HEi8LuS1kn6hqTZtZ7gESOr5ixLkTAaOJzKqkezgQck/XoMMJG8g5FT645KtgErq0FYL2kfMAnY1d8T3EqyavxRST++BJwLIOlEYCzg1QfK1ZSFbO4DzgEmSdoGXA+sAFZUD2H3AB8ZqI2Ag5FZU45KFvRz15/Wsx0HI6eCz3w6GFmVu4vnYORU7oDhYORVbjIcjKwcDEvxzqelORiW5GBYiluJpTkYluRgWIpbiaWVG4xyT9ZbVh4xcnIrsTQHw5IcDEtxK7E0B8OSHAxLcjAspeB9DJ/gsiSPGDmp3M+lg5FVua2kbYPRsw/+8LZjOfLQbm5duJ3rvnQk6380jkN/pQeApRf/hJOO6spcpYPRcnetO4x3TtrD7q63h+vF5+9i3szdGavqYzjvfEqaIek8SRP63D6veWUdnB0/G82alybwgd96I3cpNbTsr93rNmAwJF0JPAxcAWyUdFGvu29sZmEH48bVk/nrubsY1ed9vPnRSfzBLcdx4+rJ7Oku4dM6TIMBXAKcFhEXU/nT+k9Luqp6X79V9p4OaPnyeid/OTiPvTieXxvfw8lTDtx/uPq8V1n9iS08dMlW3nhzFMu/dXhL60qS6r+0SK19jI6I2A0QEVsknQM8KOk4BghGn+mAgnsva0Stg/LU1k4efWE8j780na5usbtrFNeuPIqb5u8AYOzoYP6sn7FibQHBGMY7nzskzYqIDQARsVvSBVQm4jil6dUNwTVzX+WauZXJYtZt6WTF2sO5af4Odv68gyMO7SECvvb8BE44Yk/mSmE4B+PDQHfvGyKiG/iwpFubVlUTXLvyaF77vw4iYMZRXdxwwU9yl1RyLlCNGXcaITyXOPvnEj/wjdj1zfrf/MlnteTNbNvzGMNDuR8YByOngk9wORhZlRuMcr/eGxGasvrAX1VXGNgo6T5JhwylMgcjpwaf4JI0FbgSOD0iTgY6gA8OpTS3kqya0kpGA52S9gLjgO1D2YhHjKwa20oi4hXgJmAr8GPgjYj4r6FU5mBkVX8wBlqWQtLhVBasmQ5MAcZLqmtG4P3cSnIaQiepsSzFXOCHEbELQNJK4HeAu+t9HY8YWTX8qGQrcKakcZIEnAdsGkplHjFyavAvA0fEOkkPAk9R+Y7re9S/6A3gYGTW+KOSiLieylIUB8XByKrcM58ORk7+rsTSHAxLcjAspdxcOBh5lZsMByMrB8NSfFRiaQ6GJTkYluJWYmkOhiU5GJZScCvxL+pYkkeMrModMRyMrBwMSyl4H8PByMrBsCQHw1I8ZbSlecSwlIJ3PlszB5ftd2ASfrG9/vemc0pL0tSKYBRB0qXVv/u0QSh376fxLq39ENtvJAXD6uBgWNJICob3L+owYnY+rT4jacSwOrR9MCTNk/SCpM2Srstdz3DR1q1EUgfwInA+sA14ElgQEc9lLWwYaPcRYw6wOSJ+EBF7gPupzGpnNbR7MKYCL/e6vq16m9XQ7sFIfa/Qvr2zgdo9GNuAY3pdn8YQp1Aeado9GE8CJ0iaLmkslQnXH8lc07DQ1r+PERHdki4HvkJlJv4VEfFs5rKGhbY+XLWha/dWYkPkYFiSg2FJDoYlORiW5GBYkoNhSQ6GJf0/HvkXEOA/9BEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe3f57c588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the single annotation\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(annotation)), annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring a Single Annotation\n",
    "Let's calculate the dot product of a single annotation. Numpy's [dot()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) is a good candidate for this operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "927"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def single_dot_attention_score(dec_hidden_state, enc_hidden_state):\n",
    "    # TODO: return the dot product of the two vectors\n",
    "    return np.dot(dec_hidden_state, enc_hidden_state)\n",
    "    \n",
    "single_dot_attention_score(dec_hidden_state, annotation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Annotations Matrix\n",
    "Let's now look at scoring all the annotations at once. To do that, here's our annotation matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations = np.transpose([[3,12,45], [59,2,5], [1,43,5], [4,3,45.3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it can be visualized like this (each column is a hidden state of an encoder time step):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD8CAYAAAA2Y2wxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFsJJREFUeJzt3X2UVXW9x/H3d2YgBmZGHAJE0EDFpDTRVEyLDEWxBzHFfIrQuE6lpF5AQb0uo24GK3u8rYq5hYEK4lW8mJbPcPEhUFRClBRSUxJEEYRhgJk5fO8f5xiTwZyBOb/z2+z5vNbaa5+zz5zf/s5m9ofv/PY+YO6OiIiEUxK7ABGRtFPQiogEpqAVEQlMQSsiEpiCVkQkMAWtiEhgCloRkcAUtCIigSloRUQCKyvCPvTRMxFpLWvzCDOt9Zlzgbd9f61QjKCFuleLsptEq+iXXc8syp9rsl2QOw+2vBm3jiQo3z+73rQibh1JUNk/dgXBFCdoRUSKJnnNjIJWRNLFknfpSUErIimjjlZEJCxLXtAmr8cWEUkZdbQikjLJ62gVtCKSLgmcOlDQikjKJG9GVEErIimjjlZEJCxNHYiIhKagFREJTEErIhKWpg5EREJT0IqIhKWOVkQkNAWtiEhgCloRkbA0dSAiEpo+gisiEpY6WhGR0JIXtMnrsUVEUkYdrYiki6YORERCS94v6gpaEUkXdbQiIqEpaEVEAlPQiogEpqAVEQmrgHO0ZvYasAnIAE3ufoyZVQOzgb7Aa8BX3H19S+Mk7/KciEib2G4srfI5dx/o7sfknk8EHnH3/sAjuectSnVHu21bAxdeMp6GhkYymQynnfwZLv/myNhlFdWQn/ajy4e2U2JOaQnMqXmdv6zpyA339aS+oYTeXRu56aw1VHxoe+xSi+qaG6Ywf8FCulV35d67bo5dTnSZTIazR/47PXt0Y+pPb4hdTtuEv+tgOHBS7vF0YD4woaU3pDpoO3bswPRfT6FL53IaG5u4YPQ4Bp94DAOPGBC7tKKaPuoNqjvvCNLrfr8fE4a+zXF9t3Dnc1X85ol9uXLIuogVFt9ZZwzjq+d9mQn/8YPYpSTCjFn3cHC/A6jbXB+7lAIoaNA68KCZOTDV3WuBnu6+GsDdV5tZj3yD5J06MLPDzGyCmf3czH6We7xXJJWZ0aVzOQBNTU00NTVhCZwoL7ZX3+nAsR/ZAsCJB9Xz4PKKyBUV37GfPJJ9qqpil5EIa956h/lPPM2IM0+NXUqBtH7qwMxqzGxxs6XmA4Od6O5HA6cDl5nZ4D2pqMWgNbMJwO25qp4Cns49nmVmeeclkiCTyTD8/Es5Yeh5nHD80Rx5xGGxSyoug9G39OGs2gOZ/cw+ABzao4FHXuoCwP0vVrB6Y4eYFUpkN/6olqsu/zolCbzRf4+YtXpx91p3P6bZUtt8KHd/M7deC9wNHAe8ZWa9sruyXsDafCXlmzoYDXzc3Rv/+fuwHwMvAJNb+73HUlpaytxZv2TjpjouG/ddXl75Goce0jd2WUUz6+uv07Myw7rNpVx8Sx8O+nAD3x++hu//sQe/XNCNIYfW0bHUY5cpkcx77Cmqq7ty+IBDWLR4aexyCqQwf2GYWRegxN035R6fCnwXuAcYRTb/RgFz842VL2i3A/sDf/vA9l6513ZVYA1QAzB16lRqLhiar47gqiorGHTMJ3jsycXtKmh7VmYA6NYlw9DD6lj6906MPmE900b+HYBX13Vg/or2N3UgWc/++UUeXbCIBU8sZltDA3V1Wxh//U3c9L3xsUvbc1awm6l6AndbttMvA2a6+/1m9jRwh5mNBl4Hzsk3UL6gvRJ4xMxWAG/kth0IHAKM2dWbcu33+y24U/dqvjqCeHf9BsrKyqiqrGDr1m08ueg5Lhn1lSi1xFDfYGx3qPiQU99gPPHXzlz62XWs21xKty4Ztjv8akE3zjtmQ+xSJZJxYy5i3JiLAFi0eCnTbr177w5ZoFAdrbu/Ahy5k+3rgJN3Z6wWgzaX3oeSnZfoTfY7WAU87e6Z3dlRDGvfeZeJN/yITCaDuzPslMF8bvCg2GUVzbrNZVw2e38AMtvhi4dvYvAh9Uxf2JWZT3cFYOiAOs4euDFmmVGMnfg9nlq8hPUb3mPwqefw7W9dxDlf/kLssqQgkjfXbO7B5+eidbSJUtEvu56ZvB+Corsg9zO35c24dSRBefYvQjatiFtHElT2h0Kk5H0fa32ofeHFopyQqb6PVkTao+Q1MwpaEUmXwl0MKxgFrYikjDpaEZHAFLQiImEl8BNuCloRSRkFrYhIYApaEZGwNHUgIhKaglZEJDAFrYhIWJo6EBEJTUErIhKYglZEJCz9WwciIqGpoxURCSt5OaugFZG0SV7SKmhFJGUUtCIiYelimIhIaOpoRUTC0ifDRERCU9CKiASmoBURCUtTByIioSloRUQCU9CKiISVwKmD5N3ZKyLSJrYbSytGMys1s+fM7N7c835mtsjMVpjZbDPrmG8MBa2IpExhgxa4Alje7PkU4Cfu3h9YD4zON4CCVkTSxaz1S96hrA/wBeA3uecGDAHuzH3JdODMfONojlZEUqag/eNPgauBytzzbsAGd2/KPV8F9M43SHGCtqJfUXazV7jAY1eQHOX7x64gOSr7x64gPXbjYpiZ1QA1zTbVuntt7rUvAmvd/RkzO+n9t+xkmLwntTpaEUmZ1gdtLlRrd/HyicAZZvZ5oBNQRbbD7WpmZbmutg/wZr79FCdo1zxUlN0k2n5Ds+v6N+LWkQSdD8iuHz4pahmJcMr87Lru1ahlJELCfvN192uAawByHe14d7/QzP4HGAHcDowC5uYbSxfDRCRdCngxbBcmAGPNbCXZOdvf5nuDpg5EJGUK3z+6+3xgfu7xK8Bxu/N+Ba2IpEsCPxmmoBWRlFHQiogEpqAVEQlLUwciIqEpaEVEAlPQioiEpakDEZHQFLQiIoEpaEVEwtLUgYhIaMn7J1wUtCKSLupoRURCS17QJq/HFhFJGXW0IpIumjoQEQlNQSsiEpYlb0ZUQSsiKaOOVkQkMAWtiEhYuhgmIhKaglZEJDAFrYhIWJo6EBEJTUErIhKYglZEJCxNHYiIhKagDe6aybcy/0/L6LZvJff+7joApvzqbuY9uYwOZaUcuP+H+cHEr1JV2TlypcW1es1arr5+Cu+sW0+JGV85+wuMuuCs2GUVXWa7c/aUd+jZtZSp36rm2ls3sOz1RtyhX48yfjByH7p0St5HOEPZtq2BCy8ZT0NDI5lMhtNO/gyXf3Nk7LLaJoEfwU1eRW101unH85sfXvZP20485jDuvflafn/ztfQ9oAdTb3swUnXxlJaWMnHsN/njnGnMnvFfzJw9l5V//VvssopuxrzNHLzfjv7i2rOruOfa7vz+uu70qi7ltgX1Easrvo4dOzD911O45/Zf8b8zf8ljTy5myfPLY5fVRrYbSwujmHUys6fM7M9m9oKZTcpt72dmi8xshZnNNrOO+SpKXdAee+Qh7POBbvXTxw6grKwUgIEf68eatzfEKC2qHt278fEB/QGo6NKZg/odyFtvvxO5quJasz7D/GXbGHHCjp+PivLsKeDubG3wWKVFY2Z06VwOQFNTE01NTVgCf/XePYUJWmAbMMTdjwQGAsPM7HhgCvATd+8PrAdG5xsodUGbz11/+BODB30sdhlRrXpzDctfWsmRhx8Wu5SiuvHOjVz15SpKPnB+XXPLBk68Zi2vvNXEyJO6xCkuokwmw/DzL+WEoedxwvFHc+QRe/nPhVnrlxZ4Vl3uaYfc4sAQ4M7c9unAmflK2uOgNbOL9/S9sfzqlvspLS3hjKHHxi4lms31W7h8/CSuHX8pFRXtJ1TmPb+V6soSDj+ww7+89oORXXnsxh4cvF8Zf3hmS4Tq4iotLWXurF/yf3+8laXLXuLlla/FLikxzKzUzJYAa4GHgL8CG9y9Kfclq4De+cZpS0c7qYXiasxssZktrq2tbcMuCufu+xcy/8ll3HT9RVgCb/8ohsbGJi4f/x2+dPrJnHryZ2KXU1TPvtLAo89vZcj1axk7bQMLX9rG+N+t/8frpSXG5z/ZiQeXbI1YZVxVlRUMOuYTPPbk4tiltFHrpw6aZ1VuqWk+krtn3H0g0Ac4Dhiwkx3mnXNq8a4DM1vawnfSc1fvc/da4P2EddY8lK+OoBYsepH/nvkwt/78Cso75Z23TiV357pJN3FQv49w8cgRscspunHDqxg3vAqARS9vY9ojm/nhqK78bW0TH+lRhrsz7/ltHNQzdTfitOjd9RsoKyujqrKCrVu38eSi57hk1Fdil9U2u3HXwQeyqqWv22Bm84Hjga5mVpbravsAb+Z7f76fqp7AaWQnfJsz4Ml8g8cwdtLNPLVkBevfq2PwiP/g2xd/ntrbHqShoYmLx/0CgCM/1pfvjjs/cqXF9cySZcy972EO7d+P4ed+A4CxY77OZz8zKHJl8bjDhFs2sHmr4w4f7V3GpPP2iV1WUa19510m3vAjMpkM7s6wUwbzucF7+89EYX5jNbPuQGMuZMuBU8heCJsHjABuB0YBc/OO5b7rrtfMfgvc7O6P7+S1me5+QSvqjd7RJsJ+Q7Pr+jfi1pEEnQ/Irh8+KWoZiXDK/Oy67tWoZSRCRT8oREoumdD620cGTtnl/szsE2QvdpWSnWa9w92/a2YHkQ3ZauA54Kvuvq2l3bTY0br7Lm9baGXIiogUWWE6WndfChy1k+2vkJ2vbbX2NSElIu1A8i52K2hFJF0SeFeRglZEUkZBKyISljpaEZHQFLQiIoEpaEVEAlPQioiElcB/+FtBKyIpo45WRCQs3XUgIhJa8oI2eZMZIiIpo45WRNJFUwciIqEl7xd1Ba2IpIs6WhGR0BS0IiKBKWhFRMLS1IGISGgKWhGRsNTRioiEpqAVEQlMQSsiEpamDkREQlPQiogEpo/gioiEpakDEZHQkhe0yeuxRURSRh2tiKRLAqcO1NGKSMrYbiwtjGJ2gJnNM7PlZvaCmV2R215tZg+Z2Yrcet+8Fbl7m76lVgi+AxFJjba3oyuntj5zDvnGLvdnZr2AXu7+rJlVAs8AZwIXAe+6+2Qzmwjs6+4TWtqNOloRSZnCdLTuvtrdn8093gQsB3oDw4HpuS+bTjZ8W1ScOdoHji/KbhLttIXZ9cblcetIgqoB2bWOxY5j8cCguHUkwWmLCjRQ4edozawvcBSwCOjp7qshG8Zm1iPf+9XRiki6mLV6MbMaM1vcbKn51+GsArgLuNLdN+5JSbrrQERSpvUdrbvXArW7HMmsA9mQvc3d5+Q2v2VmvXLdbC9gbb79qKMVkXSxktYvLQ1jZsBvgeXu/uNmL90DjMo9HgXMzVeSOloRkZ07ERgJPG9mS3LbrgUmA3eY2WjgdeCcfAMpaEUkZQpzMczdH29hsJN3ZywFrYikSwI/GaagFZGUUdCKiASmoBURCSvP3QQxKGhFJGXU0YqIBKagFREJK3k5q6AVkbRJXtIqaEUkZRS0IiJh6a4DEZHQ1NGKiISlj+CKiISmoBURCUxBKyISVgIvhiWvIhGRlFFHKyLpoothIiKhKWhFRAJT0IqIhKWpAxGR0JJ3jV9BKyIpo45WRCSsBE4dJK/HFhFJGXW0IpIyyetoUxu0me3O2T/cRM+uJUz9RgUTb93MUyubqCzP/iFMvrAzA/qk9tvfqSFnXEKXzuWUlJRQWlbKnBk/il1SNDoW758jdfTsas3OkQyV5dnX99pzJIFTB3vhUWydGfO3cfB+JdRt3bHt6uHlDDuqY7yiEmD6r/+T6q5VsctIhPZ+LHacI/6PbVcP75SCcyR5M6J5KzKzw8zsZDOr+MD2YeHKaps167cz/8VGRnzqQ7FLEUmk7DnSxIhP7e2huhNmrV+KpMWgNbPLgbnAt4FlZja82cs3hiysLW6cU89VZ5RT8oHj+JP7tvClyRu5cU49DY2+8zenmRmjx3yHs0aOZfacB2JXE1c7PxY3ztnCVWd02sk5sjV3jmzZi88R240lz0hm08xsrZkta7at2sweMrMVufW++cbJN3VwCfBJd68zs77AnWbW191/1qoqI5i3rIHqyhIOP7CMRSsa/7F97JfK6V5lNDbB9bPrqX14K2NOL49YafHN+s1kenavZt27G7h4zHc4qG8fjj3647HLiqI9H4t5yxqprrRWnCPbGHN6p4iV7qmCRtPvgF8AM5ptmwg84u6TzWxi7vmElgbJN3VQ6u51AO7+GnAScLqZ/ZgWvhszqzGzxWa2uLa2Ns8uCuvZVzI8+nwDQ77zHmN/t5mFLzcyfsZmeuxTgpnRsYNx1qCOPP96pqh1JUHP7tUAdKvuytCTBrH0hRWRK4qnPR+LZ19p4tHnG3PnSD0LX27axTnSFLvUPVPAqQN3XwC8+4HNw4HpucfTgTPzjZOvo11jZgPdfUlup3Vm9kVgGnBEC8XVAu8nrPPAtHx1FMy4M8oZd0a2U120opFpj27jpq91Ye172+mxTwnuzsNLG+nfK3kT5iHVb9nK9u1ORZdy6rds5YmFS7j0386NXVYU7f1YtP4cKY1c6Z4Kfm73dPfVAO6+2sx65HtDvqD9GvBPf625exPwNTObusdlRjB+xmbW123HgcN6lzHp3M6xSyqqdes2cNnVkwHINGX44rDBDD7h6MhVxaFjsXPjZ9Q3O0dKmXTuXjq1thszB2ZWA9Q021SbaxQLW5J78Alv54HjQ+8j+U5bmF1vXB63jiSoGpBd61jsOBYPDIpbRxKctggKMcH69uOtD7Xun867v9z1qXvd/fDc85eAk3LdbC9gvrt/tKUx2tfvzyLSDhTuroNduAcYlXs8iuydWS1K7QcWRKSdKuD9sWY2i+xNAB82s1XADcBk4A4zGw28DpyTbxwFrYikTOGC1t3P38VLJ+/OOApaEUmXBP534wpaEUmZ5H2WSkErIimjoBURCUxBKyISVvJyVkErImmTvKRV0IpIuuiuAxGR0NTRiogEpqAVEQlL/zmjiEhoCloRkbDU0YqIhKagFREJTEErIhKWpg5EREJT0IqIBKagFREJSx/BFREJTR2tiEhYCbwYlrweW0QkZdTRikjKJK+jVdCKSMooaEVEwtJdByIioamjFREJK4F3HShoRSRlFLQiIoElL2jN3UPvI/gORCQ12p6SW95sfeaU71+UVC5G0CaCmdW4e23sOpJAx2IHHYsddCzCSd59EOHUxC4gQXQsdtCx2EHHIpD2FLQiIlEoaEVEAmtPQau5px10LHbQsdhBxyKQdnMxTEQklvbU0YqIRJH6oDWzYWb2kpmtNLOJseuJycymmdlaM1sWu5aYzOwAM5tnZsvN7AUzuyJ2TbGYWScze8rM/pw7FpNi15RGqZ46MLNS4GVgKLAKeBo4391fjFpYJGY2GKgDZrj74bHricXMegG93P1ZM6sEngHObI8/F2ZmQBd3rzOzDsDjwBXuvjByaamS9o72OGClu7/i7g3A7cDwyDVF4+4LgHdj1xGbu69292dzjzcBy4HecauKw7Pqck875Jb0dl+RpD1oewNvNHu+inZ6QsnOmVlf4ChgUdxK4jGzUjNbAqwFHnL3dnssQkl70O7sc8z621oAMLMK4C7gSnffGLueWNw94+4DgT7AcWbWbqeVQkl70K4CDmj2vA/wZqRaJEFy85F3Abe5+5zY9SSBu28A5gPDIpeSOmkP2qeB/mbWz8w6AucB90SuSSLLXQD6LbDc3X8cu56YzKy7mXXNPS4HTgH+Ereq9El10Lp7EzAGeIDsBY873P2FuFXFY2azgD8BHzWzVWY2OnZNkZwIjASGmNmS3PL52EVF0guYZ2ZLyTYmD7n7vZFrSp1U394lIpIEqe5oRUSSQEErIhKYglZEJDAFrYhIYApaEZHAFLQiIoEpaEVEAlPQiogE9v+Buxus/WCn0wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe3f4cfb38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotation (each column is an annotation)\n",
    "ax = sns.heatmap(annotations, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring All Annotations at Once\n",
    "Let's calculate the scores of all the annotations in one step using matrix multiplication. Let's continue to us the dot scoring method\n",
    "\n",
    "<img src=\"images/scoring_functions.png\" />\n",
    "\n",
    "To do that, we'll have to transpose `dec_hidden_state` and [matrix multiply](https://docs.scipy.org/doc/numpy/reference/generated/numpy.matmul.html) it with `annotations`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 927.,  397.,  148.,  929.])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def dot_attention_score(dec_hidden_state, annotations):\n",
    "    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states\n",
    "    return np.matmul(np.transpose(dec_hidden_state), annotations)\n",
    "    \n",
    "attention_weights_raw = dot_attention_score(dec_hidden_state, annotations)\n",
    "attention_weights_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these scores, can you guess which of the four vectors will get the most attention from the decoder at this time step?\n",
    "\n",
    "## Softmax\n",
    "Now that we have our scores, let's apply softmax:\n",
    "<img src=\"images/softmax.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.11920292,  7.9471515e-232,  5.7661442e-340,  0.88079708], dtype=float128)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    x = np.array(x, dtype=np.float128)\n",
    "    e_x = np.exp(x)\n",
    "    return e_x / e_x.sum(axis=0) \n",
    "\n",
    "attention_weights = softmax(attention_weights_raw)\n",
    "attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even when knowing which annotation will get the most focus, it's interesting to see how drastic softmax makes the end score become. The first and last annotation had the respective scores of 927 and 929. But after softmax, the attention they'll get is 0.12 and 0.88 respectively.\n",
    "\n",
    "# Applying the scores back on the annotations\n",
    "Now that we have our scores, let's multiply each annotation by its score to proceed closer to the attention context vector. This is the multiplication part of this formula (we'll tackle the summation part in the latter cells)\n",
    "\n",
    "<img src=\"images/Context_vector.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.35760877,  4.6888194e-230,  5.7661442e-340,  3.5231883],\n",
       "       [ 1.4304351,  1.5894303e-231,  2.479442e-338,  2.6423912],\n",
       "       [ 5.3641315,  3.9735758e-231,  2.8830721e-339,  39.900108]], dtype=float128)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def apply_attention_scores(attention_weights, annotations):\n",
    "    # TODO: Multiple the annotations by their weights\n",
    "    return attention_weights * annotations\n",
    "\n",
    "applied_attention = apply_attention_scores(attention_weights, annotations)\n",
    "applied_attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize how the context vector looks now that we've applied the attention scores back on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD8CAYAAAA2Y2wxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcjXX/x/HXZ2a4G/s2BiX7GklRqKihKEnWEKVoSt0kJUo3P61KdadN5i6ttGlxt7oJKdlKEpE2CRlkGYOYOfP9/THTMJgxZs51znF5Px+P62HOua5zXZ9zzfE+n/le13WOOecQERHvRIW7ABERv1PQioh4TEErIuIxBa2IiMcUtCIiHlPQioh4TEErIuIxBa2IiMcUtCIiHosJwTZ06ZmI5JcVeg1TLf+Z08cVfnv5EIqghT0bQrKZiFbs5Mx/924Mbx2RILZK5r/aFwf2xY7vwltHJCjTONwVeCY0QSsiEjIhaVKPiYJWRPzFIu/Qk4JWRHxGHa2IiLcs8oI28npsERGfUUcrIj4TeR2tglZE/CUChw4UtCLiM5E3IqqgFRGfUUcrIuItDR2IiHhNQSsi4jEFrYiItzR0ICLiNQWtiIi31NGKiHhNQSsi4jEFrYiItzR0ICLiNV2CKyLiLXW0IiJei7ygjbweW0TEZ9TRioi/aOhARMRrkfeHuoJWRPxFHa2IiNcUtCIiHlPQioh4TEErIuItjdGKiHhNQRt08+Yv5v7xT5GRkUGPKy4l8bo+Oea/9tZ/mfrmdKKioihWLJZ77x5G7VrVAVi95mfG3PdvUnfvJioqimmvTuQf/ygahmeRt0AgQLc+NxJfsQKTnnwwx7wHxj/NoiXfAPDXX/v4c9t2vvrig3yv+78fzuQ/L74OQPHYWP5v1FDq16vNvn37ueq6W9iftp9AeoD27dow5KZrAfh9wx8MG3EPO3fuomGDOjx8/10ULVIkSM82NObNX8z9Dz9FRkaAHl06Hva68aN9+/Zz1Y2j2b8/jUAgQPuElgxJvDLHMu98MIeHn3yF+LhyAPTt0YEenduFo9yCU0cbXIFAgHvGTeCFieOJj4+j+1WDSGjTKjtIATpd0pbePS4H4NO583nwsYk8//RDpKcHGH73g4y/907q16vF9h07iYmJDtMzydvLU9+mVo1TSd2957B5dw2/OfvnV157h+9X/3hM6z7l5Mq8+vzjlC5Vks++WMS/7n2Ut16dSNGiRXjpP49RvFgsaWnp9Ll2MK3PO4czTm/II49Pon/fHnTskMDo+x5j2rsf0adn50I/z1AJBALc8+AEXnj279fNjYe9bvyoaNEivPT0mMzfaXo6fRLvpnXLppzRuG6O5S5t14rRwweGqcpgiLygPeqZvWZW38xGmNkTZjYh6+cGoSjuaJavWE21qidT9ZQqFC1ShI7tE/h07pc5lilRonj2z3v3/oVl/RLmL1hCvTo1qV+vFgBly5QmOjrygnZT8hbmfr6Q7l07HnXZDz+ezWUd2mbffu7F1+nW50Y69RjAE8+8cMTHnHlGI0qXKgnAGac3ZFPyVgDMjOLFYgFIT08nPT2AGTjnWLjkG9q3awNAl07t+XTOF4V6jqGW+bqpcsjrZn64y/Jczt9pIPt36j92DFNo5NnRmtkIoDfwOrA46+5TgNfM7HXn3DiP68tT8uatVIqvmH07Pr4Cy1esOmy5KW+8xwuvvkVaWjovTXoUgF/XrccMBtx0B9u27+DS9glc379XyGrPrwfGP8XwoTewe/fePJfbsHET6zf+QYuzmwLwxZdL+G3deqZNmYhzjkG3jGLJ19/S/Kwmua5j2rsf0fq8s7NvBwIBuva+gXW/b6DPlVfQpHFDtm3fSamSJbK7/0rxcSRv3hqEZxo6yZu3UqnSwa+bOJZ/d/jrxo8CgQBdrxnBuvWb6NO9PU0a1T1smf/NWciSZd9To2oV7ry1P5XjK4Sh0kKIwHePo3W0A4DmzrlxzrlXs6ZxwNlZ88LK4Q67z47wLnXVlVcw6/0p3H5LIhOfexXIfMF9/c0Kxt8/iqmTn2DW7C9YsGip5zUfiznzFlCubBkaNax31GU/nDGH9u3aZHfl8xd+xfwFX3HFldfTpVciv6xdx9p163N9/MIl3zDtvY+4/ZbE7Puio6OZ/uZzfDbjLZavWM2an34Fd4R9HoEv7Lw4HzyHgoqOjmb6q4/w2fuTWL7yJ9b8vC7H/AvPb8bs9yby/pTHaHl2Y0aMfSpMlRZG5HW0RwvaDKDKEe6vnDXviMws0cy+MrOvkpKSClNfnipVjGNT8ubs28nJW6kYl/u7b8f2FzIr60/EShXjOPusJpQrW5rY2JNofd45rFy9xrNaC2LpshXM/uxLEi7pxbCR97BwyTfcftf9R1z2o09m07FDQvZt5xyJA/ow/c3nmP7mc8x8fwo9unRkyuvv0rnnQDr3HJjdia5e8zN3j32EZx6/j7JlSh+27lKlSnBOszP4fP5iypYtTcquVNLTA0Dm0EbFuPIePHvvVIqPY9Omg183x99zKKxSJYtzzlmn8fmCb3LcX7Z0SYoWzTyw2bNzO1au/iUc5RWOReV/CpGjbWko8KmZfWxmSVnTJ8CnwC25Pcg5l+Sca+aca5aYmJjbYoXW+LT6rF23gd83/MH+tDQ+nDGbhAta5lhm7W8Huri5ny+kWtWTATivVXN++PFn9u79i/T0AEu+/pbaNat7VmtB3Dbkeub97y1mf/w6j40bTYvmTXnkgVGHLffL2nWkpOyiaZPTsu87r2Vz3n7vY3bvyRxySE7ewp/btnNVry7Z4RtfsQIb/0hm8G2jefi+O6lRrWr247dt20FKSiqQeTbDl4u+pmaNUzEzzmnWlBmzPgPg3fdnkHDBuV7uhqA74uumTatwl+W5bdt3krJrN5D1O128nJrVT86xzOat27N/nv35V9Q6ZP7xIfI62jzHaJ1zn5hZXTKHCk4ms7L1wBLnXCAE9eUpJiaa0SMGM/CmEQQyAnTrfAl1atVgwjMv0KhhXdpecC6vvvEeCxZ9TUxMDKVKleShe0cAULpUSfr37UH3voMwM1qfdw4XnN8izM8ofyY8M5lGDevRNivgPvx4Npd2SMjx5+95rZrz86+/0evqzLMSihWLZfz9d1G+XNkc63o66WV27Ehh7AOPAxAdE807UyexeeufjPzXOAIZGbiMDDpcfAEXts58Exs+NJFbR9zL408/T4N6dejR5dJQPO2giYmJZvTIIQwcdAeBjIzM103tGuEuy3Obt25n5D1PZf1OHR3atuLC85oxYdLrNGpQi7atm/PKGx8x+/MlREdHU7pUCR4c/c9wl10AkTcMZEcarwoyx54NXm8j8hXL6gz2bgxvHZEgNms0SvviwL7Y8V1464gEZRpDMFLyw4b5D7WO3+e6PTM7CZgH/IPMpnSac26MmU0BmgFpZJ4kcINzLi2vzUTeBzeKiBRK0IYO9gEJzrkmwBlABzNrAUwB6gONgVjgqCcdH9cXLIiIHCZIB7lc5p/7qVk3i2RNzjn3UfamzBaTecprntTRiojP5L+jPfgMqawpx9F7M4s2s2XAZmCmc27RQfOKAP2AT45WkTpaEfGZ/A/zOueSgFzPQc066H+GmZUB3jWzRs65FVmznwHmOec+P9p21NGKiL+Y5X/KJ+fcDmAu0CFzEzYGiAOG5efxCloR8ZngHAwzs7isThYziwXaAavNbCDQHujtnMv1wq2DaehARHwmaOfRVgZeMrNoMpvSN51zH5hZOvAbsCDr3PV3nHP35LUiBa2I+EuQPrfCObccaHqE+485NxW0IuIzkXdlmIJWRHxGQSsi4q0I/MhLBa2I+IyCVkTEYwpaERFvhfADvfNLQSsiPqOOVkTEW5GXswpaEfGbyEtaBa2I+IyCVkTEWzoYJiLiNXW0IiLe0pVhIiJeU9CKiHhMQSsi4i0NHYiIeE1BKyLiMQWtiIi3NHQgIuI1Ba2IiMcUtCIi3tLQgYiI107UzzoodnJINnNciK0S7goih/bFAWUah7sC/1BHKyLitRM1aFN/CclmIlqJmpn/7t0Y3joiwd+drPbFgX2Rsjq8dUSCUvXDXYFn1NGKiL9o6EBExGsn6sEwEZFQUUcrIuI1Ba2IiMcUtCIi3tLQgYiI1xS0IiIei7ygjbzzIERECsMs/1Oeq7GqZjbHzFaZ2Uozu+WQ+bebmTOzCkcrSR2tiPhM0DradOA259xSMysJfG1mM51z35tZVeAiYF1+VqSOVkR8xo5hyp1z7g/n3NKsn3cBq4C/PyHr38AdgMtPRepoRcRfPDjrwMyqA02BRWZ2ObDBOfet5XNbCloR8Zn8/6FuZolA4kF3JTnnkg5ZpgTwNjCUzOGEUcDFx1KRglZE/OUYOtqsUE3Kbb6ZFSEzZKc4594xs8ZADeDvbvYUYKmZne2c25TbehS0IuIzwRk6sMwkfR5Y5Zx7DMA59x1Q8aBl1gLNnHNb81qXDoaJiBzZuUA/IMHMlmVNlxZkRepoRcRfgnQwzDn3BUdpj51z1fOzLgWtiPhM5F0ZpqAVEX+xyBsRVdCKiM+ooxUR8ZiCVkTEW/o8WhERryloRUQ8pqAVEfGWhg5ERLymoBUR8ZiCVkTEWxo6EBHxWuQFbeRdq1YId459jJbtenFZzxvzXG75yh9o0Lwjn8z6PESVFcydYx6i5YVduKzbtbkus2jJMjr3HEjHrv3pO+CWXJc7kvkLvqJr70Q6db+Orr0TWbB4afa8ATfdweU9B9Cxa39G3/cYgUAAgI//N5eOXftTv2kC3638oWBPLALMm7+Y9p2v5qJOV5E0eWq4ywmJPzZtod+No7ikx8107PlPXnrt/SMut+jr7+jcZygde/6Tvol3hbjKILCo/E8h4quOtmuni+jb83JGjHkk12UCgQCPPPEC57U8M4SVFUzXyzvQt1cXRtz94BHnp6SkMvbBx3nu6YeoUjmeP7dtP6b1ly1bmokTHiC+YgXW/PQrAwbdwecz3wJgwsNjKFGiOM45htw+hk9mfkbHDgnUrV2DJx+7hzH3Plbo5xcugUCAex6cwAvPjic+Po7uV91IQptW1K5VPdyleSo6JpqRQ6/jtPq1SN29h25X38a55zShds1Ts5dJ2ZXK2Iee5bkn/o8qleL4c9uOMFZcUJHX0foqaJuf2Zj1G5PzXOaVN/5L+7bn8t3KNSGqquCan9WE9Rty/dB23v94FhclnE+VyvEAlC9XNnve9A9n8srUd0hLS6NJ4waMuWso0dHROR7fsH6d7J/r1KrO/v372b9/P0WLFqVEieIApKcHSEtLzx72qlWzWrCeXtgsX7GaalWrUPWUKgB0bJ/Ap3Pn+z5oK1YoR8UK5QAoUbwYNaufQvKWbTmC9v1P5nHRhS2pUikOgPLlyoSl1sKJvKD11dDB0SRv3sqsOV/Sq1uBPrs34qz9bT0pKbvoN2AoXXsn8t77MwD4+Zff+HjGHF578Ummv/kcUVFRvP/RrDzXNWPWPBrUr03RokWz7xswaDitErpQvFgs7du18fS5hFLy5q1UqpT9IfnEx8eRvDnPD8j3nfUbk1n1wy80Oa1ujvvXrttISkoq/W4YRdd+w3jvw9lhqrAQzPI/hUiBO1ozu9Y590Iwi/Ha/Y9M4vYh1x3W2R2vAoEAK1et4cWkR/nrr/30uvpmmpzekAWLl7Ji1Rq6X5U5Vv3Xvv05ut1D/fjTrzwyIYnJEx/Ocf/zE8ezb99+br/rPhYu/oZzWzbz9PmEinOHf0N0fr/N1A9279nLkBEPcdewgZQoUSzHvEAgwMrVP/PiM/fy17799LruDpo0qkeNaifnsjbJj8IMHYwFjhi0B3+z5KRJk0js064QmwmeFat+ZNid4wDYviOFz+YvISY6mnYXtgpzZQVTKT6OsmVKUyw2lmKxsTQ763RW//Azzjm6dGrPbUOuz7H8zNmf89SzLwFw35jhND6tHpuSt/DPYaN56N6RnFr18P9M//hHURLatOLTufN9E7SV4uPYtGlz9u3k5C1UjCsfxopCJy09nSEjxtGpQxsuTmh52PxKFctTtkwpisWeRLHYk2jW9DRW/7j2OAvayHvTzDNozWx5brOA+Nwed8g3SzpSfylYdUE2+/0Xs38eOeZRLjj/7OM2ZAHaXnAu94ybkDWOmsby71bRv28P6tSqzk1D76Z/3+6UL1eWHTtT2L17DxclnM9FCednPz4lJZXEwSMZNmQgZzVtnH3/7j172b17DxXjypOeHuCzLxbR7MzTw/EUPdH4tPqsXbeB3zf8QXzFCnw4YzaPPnB3uMvynHOOUfc+Sc3qVbn2qs5HXKZtm3O45+GkzNdUejrLV6yhf+/LQ1xpIR2HH/wdD7QHDj2cbcCXnlRUCMPuGsfir5azfUcKrS/py+Ab+pGeng5A7+4dw1zdsRs28l4Wf7WM7Tt20vriHgwe1J/09MzTrHr3uJxaNatxfquzubznAKLM6N6lI3Vr1wBg6D+v47obh5PhHEViohl951BOrlIpx/pffeNd1q3byDNJr/BM0isATH52PM45Bt0yiv1paWQEArQ4+0x6dc/8zzZz9ufcO+4Jtm3fyQ2D76RBvVo8P3F8CPdK4cXERDN65BAGDrqDQEYG3TpfQp2s/eZnX3+7iukfzaVu7Wp07jMUgGE392Xjpi0A9O52CbVqVOX8Vk25vM8QoiyK7p0vom7t4+0AaOR1tHak8arsmWbPAy9kfUnZofOmOuf65GMbEdPRhlWJmpn/7t0Y3joiQWzm0X7tCw7si5TV4a0jEpSqD8FIyWUjcg+1Q53xUEhSOc+O1jk3II95+QlZEZEQi7yO1lfn0YqIKGhFRLwWgafqKWhFxGcUtCIi3lJHKyLiNQWtiIjHFLQiIh5T0IqIeOs4vARXROQ4o45WRMRbOutARMRrkRe0kTeYISLiM+poRcRfInDoQB2tiPhM1DFMeTOzyWa22cxWHHL/YDP7wcxWmtnDuT3+b+poRcRfgtvRvgg8Bbx8YPV2IdAZON05t8/MKuby2GzqaEXEZ+wYprw55+YB2w65exAwzjm3L2uZzYc98BAKWhHxmeAFbS7qAueb2SIz+8zMmh/tARo6EBF/OYahg4O/sTtLUtaXy+YlBigLtACaA2+aWU2Xx/eCKWhFxGfyH7SHfGN3fq0H3skK1sVmlgFUALbk9gANHYiIv5jlfyqY94CEzE1ZXaAosDWvB6ijFRGfCd5ZB2b2GnABUMHM1gNjgMnA5KxTvvYD1+Q1bAAKWhHxneAFrXOudy6z+h7LehS0IuIvEXhlmIJWRHxGQSsi4rHIO8avoBURf9HQgYiI1yIvaCOvxxYR8Rl1tCLiLxo6EBHx2okatCVqhmQzx4XYKuGuIHJoXxxQqn64K/APfd24iIjXTtSOduvCkGwmolVokfnv3o3hrSMS/N3Jal8c2BdTIy8cQq5Pnh8XcAwib1+qoxURf9HBMBERryloRUS8FYEHwyKvIhERn1FHKyI+o6EDERFv6WCYiIjXFLQiIh5T0IqIeCsCzzpQ0IqIz6ijFRHxmIJWRMRbkZezCloR8ZvIS1oFrYj4jIJWRMRbOutARMRr6mhFRLylS3BFRLymoBUR8ZiCVkTEWxF4MCzyKhIR8Rl1tCLiLzoYJiLiNQWtiIjHIi9oNUYrIv5ilv/pqKuyW81spZmtMLPXzOykgpSkoBURn4k6hil3ZnYyMARo5pxrBEQDvQpSkYYORMRngjp0EAPEmlkaUAzYWJCVqKMVEX85hqEDM0s0s68OmhL/Xo1zbgPwCLAO+APY6Zz7X0FKUkcrIics51wSkHSkeWZWFugM1AB2AG+ZWV/n3KvHuh11tCLiM3YMU57aAb8657Y459KAd4BWBanId0Gb0O02OvUbRedr/kXX68bkutzyVb/Q4Pz+fDJnSQirC55AIMAVV17PDYPvzPdj/ti0mX4Db+WSLtfQsWt/XpoyLXve409PplOPAXTuOZDrbhxO8uatAPz86zquvPpmGjW/mOdfeiPozyNc5s1fTPvOV3NRp6tImjw13OWERSADrph0KjdMrQLA79tj6PFcVS5+sjpDp1VmfyDMBRZU8M46WAe0MLNiZmZAW2BVQUry5dDBS0+OpFyZkrnODwQyeOSZNznv7MYhrCq4Xp76NrVqnErq7j35fkx0dDQjbxvEaQ3qkrp7D91638C5LZpRu1Z1Bl5zJUNvvi573U8nvcw9dw+jTOmSjLpjMJ/O+cKrpxJygUCAex6cwAvPjic+Po7uV91IQptW1K5VPdylhdTLi8pQq8J+Uvdl9luPzIqjf4sddGy0i9EfVGTa0tL0ab4zzFUWRHD6R+fcIjObBiwF0oFvyGWYodAVmVl9M2trZiUOub9DQTYYCV6ZNpP2FzSjfNlS4S6lQDYlb2Hu5wvp3rVj9n0rvv+BvgNuoWvvRAYMGs7mLX8e9riKceU5rUFdAEoUL0bNmqdmd64lShTPXm7v3r+wrHf78uXKcnqj+sTE+Oc9efmK1VSrWoWqp1ShaJEidGyfwKdz54e7rJDalBLD3B9L0P3MzCB1Dhb+Woz2DXcB0KVJCp/+UCKvVUSuIJ5H65wb45yr75xr5Jzr55zbV5CS8gxaMxsCTAcGAyvMrPNBsx8oyAY9ZzDg1vF0vW40b0yfc9js5C3bmDXva3pdkRCG4oLjgfFPMXzoDURlfUpRWlo69417kifGj+Wd15LodsUl/Pup5/Jcx/oNm1i1+ieaNG6Qfd+/n3yONu178v5Hs7hl0LWePodwSt68lUqVKmbfjo+Py37DOVE88Ekcw9ttISora7bvjaLUSQFishKhUql0klOO1zfXoI3RBs3R9uT1wFnOuVQzqw5MM7PqzrkJROJ1bsBrE+8mPq4sf25P4dqhD1OzWmWan1E/e/79E6Zy+6CeREcfn8PTc+YtoFzZMjRqWI9FS5YB8Otvv7Pm51+59sbbAcjIyCCuQvlc17F7z16G3D6au4bfnKOTvXXwQG4dPJBJz0/h1dffZchN/gxb59xh91k+uhu/mLOmOOWKB2hUZR+L1sZm3ukOf/7H7y6JvMKPFrTRzrlUAOfcWjO7gMywrUYezybrXLREgEmTJpHY9fQglXt08XFlAShfthQXtT6L5d//kiNoV6z+lWFjJgKwfecuPlvwLTHRUbRrfVbIaiyMpctWMPuzL5n3xSL27d9P6u49PDnxBerUqs4bLz+dY9k/Nm3mxiF3AdCrx+X07nE5aWnpDLltNJ0ubcfFbVsfcRuXXdKWGwbf6dugrRQfx6ZNm7NvJydvoWJc7m9MfrN0XSyzfyjOvB9rsC/dSN0Xxf0z4kj5K5r0DIiJyhxaqFgyPdylFkwEvkMcLWg3mdkZzrllAFmd7WXAZCDXI0mHnJvm2LowKMUezZ69+8jIyKBE8Vj27N3H/MUruOnazjmWmT3t0eyfR973Hy4494zjJmQBbhtyPbcNuR6ARUuWMfnlN3h03L/o2LU/33y7kqZNTiMtLZ21v/1Ondo1mP7mgSEE5xyjxj5MzRrVuLZfzxzrXfvbeqpXOwWA2Z99Sc0ap4buSYVY49Pqs3bdBn7f8AfxFSvw4YzZPPrA3eEuK2Rua7eV29plDpUsWhvL5C/L8mjXTQx5qzIzvi9Jx0a7ePfbUiTUSw1zpQUVeX+tHi1orybzaFs251w6cLWZTfKsqgL6c9tObr7rCQAC6QEuu7glrVuczmvvzgagd5fjd1w2L0WLFOGJ8WO57+En2ZWaSiA9wDVXdadO7Ro5lvt62QqmfzCTunVq0rnnQACGDR5Im/Nb8OgTSfy69ncsKoqTK8czdtStAGzZuo1ufW4gdfceosx4aco0PnrnxRxDDsebmJhoRo8cwsBBdxDIyKBb50sO21cnouHttnLrtMo8Prs8DSrvo0fTlHCXVDCR19BiRxqvCrKQdbQRrUKLzH/3FuhSaX+JzTxvU/uCA/tiagSmQ6j1cRCMmNzyRf5DLe68kOz44/WwoohILiLvTUtBKyL+chweDBMROc4oaEVEvBWBXzeuoBURn1FHKyLiMQWtiIjHFLQiIt6KvJxV0IqI30Re0ipoRcRfdNaBiIjX1NGKiHhMQSsi4i1dgisi4jUFrYiIt9TRioh4TUErIuIxBa2IiLc0dCAi4jUFrYiIxxS0IiLe0iW4IiJeU0crIuKtCDwYFnk9toiIz6ijFRGfibyOVkErIj6joBUR8ZbOOhAR8VrkdbSRF/0iIoVhlv/pqKuyDmb2g5n9ZGYjC1qSglZEfMaOYcpjLWbRwNPAJUBDoLeZNSxIRQpaEfGZ4AQtcDbwk3PuF+fcfuB1oHNBKgrNGG2FFiHZzHEhtkq4K4gc2hcH9HHhrsA/gnfBwsnA7wfdXg+cU5AVhSJoI2Jk2swSnXNJ4a4jEmhfHKB9cYBv9kVslXxnjpklAokH3ZV00D440noK9I54Ig0dJB59kROG9sUB2hcHnHD7wjmX5JxrdtB08BvNeqDqQbdPATYWZDsnUtCKiByLJUAdM6thZkWBXsB/C7IinUcrInIEzrl0M/snMAOIBiY751YWZF0nUtAe/2NPwaN9cYD2xQHaF4dwzn0EfFTY9ZhzOtopIuIljdGKiHjM90EbrEvo/MDMJpvZZjNbEe5awsnMqprZHDNbZWYrzeyWcNcULmZ2kpktNrNvs/bF2HDX5Ee+HjrIuoRuDXARmadqLAF6O+e+D2thYWJmrYFU4GXnXKNw1xMuZlYZqOycW2pmJYGvgStOxNeFmRlQ3DmXamZFgC+AW5xzC8Ncmq/4vaMN2iV0fuCcmwdsC3cd4eac+8M5tzTr513AKjKvAjrhuEypWTeLZE3+7b7CxO9Be6RL6E7I/1ByZGZWHWgKLApvJeFjZtFmtgzYDMx0zp2w+8Irfg/aoF1CJ/5jZiWAt4GhzrmUcNcTLs65gHPuDDKvfDrbzE7YYSWv+D1og3YJnfhL1njk28AU59w74a4nEjjndgBzgQ5hLsV3/B60QbuETvwj6wDQ88Aq59xj4a4nnMwszszKZP0cC7QDVoe3Kv/xddA659KBvy+hWwW8WdBL6PzAzF4DFgD1zGy9mQ0Id03HbegzAAAAWklEQVRhci7QD0gws2VZ06XhLipMKgNzzGw5mY3JTOfcB2GuyXd8fXqXiEgk8HVHKyISCRS0IiIeU9CKiHhMQSsi4jEFrYiIxxS0IiIeU9CKiHhMQSsi4rH/B+msXAI+T0uPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe3f3cbf28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotations after applying attention to them\n",
    "ax = sns.heatmap(applied_attention, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrast this with the raw annotations visualized earlier in the notebook, and we can see that the second and third annotations (columns) have been nearly wiped out. The first annotation maintains some of its value, and the fourth annotation is the most pronounced.\n",
    "\n",
    "# Calculating the Attention Context Vector\n",
    "All that remains to produce our attention context vector now is to sum up the four columns to produce a single attention context vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.8807971,  4.0728263,  45.264239], dtype=float128)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def calculate_attention_vector(applied_attention):\n",
    "    return np.sum(applied_attention, axis=1)\n",
    "\n",
    "attention_vector = calculate_attention_vector(applied_attention)\n",
    "attention_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbe3f2f0a58>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEXCAYAAABoNDzDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADfVJREFUeJzt3X+MVeWdx/H3dwYItGjVhYICrizbQlj8FX9Ul25LpbUoROmuTbWtWrMdqEmV3VVBMBvTTegqbalNzKYinS7tEmsRs7Zm065Rwd9ARWBHEOsqtaAEtxWyVpdhZr77xz3KzO135s4d77nPmTufV3IT7jnnnvvkzofvc8655z6PuTsi5ZpSN0CKScGQkIIhIQVDQgqGhBQMCSkYElIwJKRgSGhYHd5Dl1aPsh5PrPrPxr3nPvJSj2Dw2mv1eJdiO+mk1C2oTl2CITGry//9gVEwElIwJKRgSKipwOeEBW6apKSKkZC6EgkpGBJSMCSkYEhIwZCQgiEhBUNCCoaEihwMXfmUkCpGQkWuGApGQgqGhBQMCSkYElIwJKRgSEjBkJCCISEFQ0IKhoSKHAx9V5KQWfWP/u3Xms3sOTN7MHs+2cw2mdmvzexeMxtRaR8KRkJ5BQNYBOzq9vx24Lvu/hHgTeBvK+1AwUgoj2CY2URgLrA6e27ABcB92SZrgPmV9qNgJDSQYJjZAjP7VbfHgrLd3gEsBrqy538CHHT3juz5XmBCpbbp4DOhgRx8uvsqYFW8P5sHHHD3Z81s1ruLo91Ueh8FI6EczkpmApeY2cXASOBYShXkODMbllWNiUDFEUvUlTQQd1/q7hPd/RTgcuARd/8S8ChwWbbZ1cADlfalYCSU41lJuSXAP5jZS5SOOX5Q6QUN05W0tx9m0aIv0d7eTmdnJ5/85Ge55prre2yzf/8+VqxYxqFDv+eYY47jllu+xdix4xO1ON8LXO6+AdiQ/ftl4NxqXt8wwRg+fAQrV65h1KgP0tFxhOuu+yIf+9gnmD79jPe2+f73b+fCC+czZ87n2Lr1ae6++zssW/atZG3Wlc86MDNGjfogAB0dHXR2dlB+QL5nz39z1lnnA3Dmmefx5JMP17uZPdSxK6laxYphZtOASymd+zqlI9qfufuuPl+YQGdnJwsX/jX79r3K/PlfZPr003usnzJlGhs3/pLLLruaxx9/iLff/gOHDr3Jhz50fJL2DtqKYWZLgJ9Q+q+3GdiS/fseM7s5/+ZVp7m5mdWrH2Dduo288MIOXnnlxR7rr712MTt2bKGlZT7bt29mzJhxNDen602LXDGsr6mvzOxF4C/c/UjZ8hHA89m19+h1C4AFAHfddddZ8+aVX5zL35o1dzJy5Ci+8IX4a4F33vkDV111EevWPVaX9mTjfPb4006dWv0AsLt3F2MA2C7gJOA3ZctP5Ogl1z9SdnXO6zEA7MGDv2fYsGGMHn0shw//H88++xRXXNHSY5t3z0aamppYu3YVF130N/k3rA9F7koqBePvgIfN7NfAb7NlJwN/Dnw9z4ZV63e/O8Btt91MV1cnXV3OrFlzOP/8T9Ha+j2mTp3BzJmz2bZtM3ffvRIz47TTzmbRoluTtrnIweizKwEwsyZK58ATKJXCvcAWd+/s53vUpWIUXdSVTJ9efVeyc2cxuhLcvQt4pg5tkQJpmAtcg1GRuxIFI6EijwysYCSkiiEhBUNCCoaEFAwJFTkYBT4ulpRUMRIqcsVQMBJSMCSkYEhIwZCQgiEhBUNCCoaEFAwJFTkYuvIpIVWMhIpcMRSMhBQMCSkYElIwJKRgSEh3iUtIFUNCCoaEihyMAvdykpIqRkKqGBKq9VBLZjbSzDab2XYze97MvpEtX2tmu82szcxazWx4pbYpGAnlMAbXYeACdz8dOAOYY2bnAWuBacCpwCjgq5V2pK4koVp3JV4aBeet7Onw7OHu/h9H39M2UxpPvE+qGAnlNF9Js5ltAw4AD7n7pm7rhgNXAr+otB8FI6E85itx9053P4NSVTjXzGZ0W/0vwGPu/niltqkrSajW85WUbXfQzDYAc4A2M7sVGAss7M/71CUY2cBkUqbWxxhmNhY4koViFPBp4HYz+yrwWWB2NqZaRaoYCeVwHeNEYI2ZNVM6TPipuz9oZh2Uxmp9ujRFGve7+z/1taO6BEPDOdanarr7DuDMYHnVf2dVjISKfOVTwUhIwZCQgiEhBUNCCoaEFAwJ6WZgCRW5YhQ4s5KSKkZCRa4YCkZCCoaEFAwJKRgSUjAkpGBISMGQUJGDoQtcElLFSKjIFUPBSEjBkJCCISEFQ0IKhoQUDAkpGBJSMCRU5GDoyqeEVDES0l3iEipyV6JgJKRgSEjBkJCCIaEiB6PAx8UD09nZSUvLfJYu/ePB6bZv38KCBZ9j9uzpbNxYcajL3OUxzmetNFww1q//ESefPCVcN27ciSxZ8s/Mnj2vzq2KKRh18sYb+3nmmQ3MnXtZuH78+IlMmTKNpoJcQGjIYJjZNbVsSC3ceec3WbjwpsL84Qez9/MJfqO3Fd2HNV61quIgtjXx9NOPctxxJzB16ozKGxdEDtNSTDKzR81sVzYtxaKy9TeamZvZmEpt6/OsxMx29LYKGNfb68qGNfZ6jPPZ1raVp556hE2bHqO9/TBvv/0Wy5ffyC23fDv/Nx+gHLqGDuAGd99qZscAz5rZQ+6+08wmAZ8BXu3Pjiqdro6jNNTwm2XLDXiqykbnqqXlBlpabgBg27ZN3Htva6FDAblMS/E68Hr27/81s13ABGAn8F1gMfBAf/ZVqSt5EBjt7r8pe+wBNgyw/XXV2vo9nnzyYQBeeGEHn//8J9i48ResXHkrX/nK3KRty/Pg08xOoTRK8CYzuwTY5+7b+/360twnuapLV1J02ZDRPf60S5ZQ9Ye/YoUtBLpPRbEq67rfY2ajgY3AckpzkzwKXOjuh8xsD3C2u/9PX++jK58J5TEtRTZZzXpgrbvfb2anApOB7dkA8xOBrWZ2rrvv720/CkZCOUxLYcAPgF3uvhLA3f8L+HC3bfbQj4qhE/6EcjjGmElpaqsLzGxb9rh4IG1TxUgoh7OSJyg7jgm2OaU/+1LFkJAqRkJF/tpdwUioyF/pKBgJqWJISMGQkIIhIQVDQgqGhIocjAKfMElKqhgJFbliKBgJKRgSUjAkpGBISMGQkIIhIQVDQkUOhi5wSUgVI6EiVwwFIyEFQ0IKhoQUDAkpGBLSXeISUsWQUJGDUeBiJimpYiRU5IqhYCSkYEhoyAcjG39Kygz5YEhsyAejyB9AvUSDIxb5c1HFSEjBkJCCISEFQ0JFDoYuiUtIwUgoj0HmzazVzA6YWVvZ8uvMbHc2j8mKSvtRV5JQTl3JvwJ3Aj86+j72KeBS4DR3P2xmH+7lte9RMBLKIxju/lg2JUV31wK3ufvhbJsDlfajriShgXQl3acVyx4LKr8THwX+ysw2mdlGMzun0gtUMRLKY1qKXgwDjgfOA84Bfmpmf+Z9TFajYCRUx9PVvcD9WRA2m1kXMAZ4o7cXqCtJqKmp+scA/TtwAYCZfRQYAWiGo6HEzO4BZgFjzGwvcCvQCrRmp7DtwNV9dSOgYCSV01nJFb2s+nI1+1EwEiryJXEFIyEFQ0IKhoQUDAkpGBJSMCSkYEhIwZBQkYOh70okpIqRUJErhoKRkIIhIQVDQgqGhBQMCSkYElIwJFTkYOgCl4RUMRLSkNESUleSQFMTbN0KP/956fkPfwgvvwzPPVd6nH562vZBPr92r5WGrRiLFsGuXXDssUeX3XQTrF+frk3lBnXFMLNpZjbbzEaXLZ+TX7PenwkTYO5cWL06dUv6VuSK0WcwzOx64AHgOqDNzC7ttvqbeTbs/bjjDli8GLq6ei5fvhy2b4eVK2HEiDRt627QBgNoAc5y9/mUfvb2j2a2KFvXazO7/1R/1apqf5j9/sydCwcOlI4vulu6FKZNg3POgRNOgCVL6tqsUJGDUekYo9nd3wJw9z1mNgu4z8z+lD6CUfZTfV+4sBZN7Z+ZM+GSS+Dii2HkyNIxxo9/DFdeWVrf3l46EL3xxvq1qTeD+Rhjv5md8e6TLCTzKP2E/tQ8GzZQy5bBpEkweTJcfjk88kgpFOPHH91m/nxoa+t9H/UymCvGVUBH9wXu3gFcZWZ35daqHKxdC2PHlj7cbdvga19L3aJiswq/hq8FL3LJrJfsY+7xSTzxBFV/+B//eO9deC017HWMwaDI/2EUjIQUDAkpGBJSMCSkYEhIwZCQgiGhIgejYW/UGarM7O+zqSfazOweMxs5kP0oGAnV+rsSM5sAXA+c7e4zgGbg8oG0TV1JQjndDDwMGGVmR4APAK8NZCeqGAnVeloKd98HfBt4FXgdOOTu/zmQtqliJFTraSnM7HhKMxlNBg4C68zsy+7+b9W+jypGQjncj/Fp4BV3f8PdjwD3A385kLapYiSUw+nqq8B5ZvYB4B1gNvCrgexIwUio1sFw901mdh+wldINVs9R/WxIgG7UqZvoRp22tupv1JkxQzfqNLwi/4dRMBJSMCSkYEhIwZCQgiEhBUNCCoaEFAwJFTkY+hJNQqoYCRW5YigYCSkYElIwJKRgSEjBkJCGjJaQKoaEhnww8r97UGqtHvd8FoKZLch+kyH9UODDn5pbUHkTeddQCoZUQcGQ0FAKho4vqjBkDj6lOkOpYkgVGj4YZjbHzHab2UtmdnPq9gwWDd2VmFkz8CLwGWAvsAW4wt13Jm3YINDoFeNc4CV3f9nd24GfUBpYRCpo9GBMAH7b7fnebJlU0OjBiL6maty+s4YaPRh7gUndnk9kgKPYDTWNHowtwEfMbLKZjaA05uXPErdpUGjo+zHcvcPMvg78ktJgqK3u/nziZg0KDX26KgPX6F2JDJCCISEFQ0IKhoQUDAkpGBJSMCSkYEjo/wGgulYn2LmcNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe3f364b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the attention context vector\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(attention_vector)), annot=True, cmap=sns.light_palette(\"Blue\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the context vector, we can concatinate it with the hidden state and pass it through a hidden layer to produce the the result of this decoding time step."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}