backprop_workbook_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backprop Workbook 01: Backprop to Output Layer\n",
    "\n",
    "**For these questions, assume that an $x$ input has 1024 dimensions, that the first hidden layer should have $512$ units, a second layer has $256$ units, and that there are $10$ classes to choose from at the end.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cell to run for Latex commands**\n",
    "\n",
    "\\\\[\n",
    "\\newcommand{\\fpartial}[2]{\\frac{\\partial #1}{\\partial #2}}\n",
    "\\newcommand{\\grad}[1]{\\nabla #1}\n",
    "\\newcommand{\\softmax}[0]{\\text{SOFTMAX}}\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. What is $\\fpartial{}{x} \\log x$?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\frac{1}{x}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. What is $\\fpartial{}{x} e^x$?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$e^x$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. What is $\\fpartial{}{x} e^{(x^2)}$?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$2x e^{(x^2)}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. What is $\\fpartial{}{x} \\log(x^2)$? Use the chain rule.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the chain rule, we have:\n",
    "\n",
    "\\\\[\n",
    "\\fpartial{}{x} \\log(x^2)\n",
    "=\n",
    "\\frac{1}{x^2} \\fpartial{}{x} x^2\n",
    "=\n",
    "\\frac{2x}{x^2}\n",
    "=\n",
    "\\frac{2}{x}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Same question, but use the property that $\\log$ pulls down exponents.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have $\\log x^2 = 2 \\log x$. Since the partial of $\\log x$ is $\\frac{1}{x}$ we are done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Give the rules for $\\log a^b$, $\\log ab$, and $\\log \\frac{a}{b}$.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "\\log ab\n",
    "&=\n",
    "\\log(b) + \\log(a)\n",
    "\\\\\n",
    "\\log a^b\n",
    "&=\n",
    "b\\log(a)\n",
    "\\\\\n",
    "\\log \\frac{a}{b}\n",
    "&=\n",
    "\\log(a b^{-1})\n",
    "=\n",
    "\\log(a) + \\log(b^{-1})\n",
    "=\n",
    "\\log(a) - \\log(b)\n",
    "\\end{align}\n",
    "\\\\]\n",
    "\n",
    "Note that the law about exponents follows from the law about products, because $a^b$ is just $a$ multiplied $b$ times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Explain the chain rule. Given functions $f$ and $g$, write a function for the derivative of the composition function $f \\circ g$ where $(f \\circ g)(x) := f(g(x))$.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "(f \\circ g)'(x)\n",
    "=\n",
    "f'(g(x))g'(x)\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation: The Concept"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent is about changing the weight matrices $W^{(i)}$ and the bias vectors $b^{(i)}$ so that the loss goes down. To do this, we need to know how the loss changes as we change the weight matrices and bias vectors.\n",
    "\n",
    "That means we need to calculate gradients like $\\grad_{W^{(i)}} CE(h^{(3)}, y)$ and $\\grad_{b^{(i)}} CE(h^{(3)}, y)$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**0. What is a partial derivative? What is a gradient?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say you have a function $f(x_1, x_2) = x_1 x_2$ which is a function of two scalar variables with a single scalar valued output.\n",
    "\n",
    "There are two derivatives you can take: one which asks how changes in $x_1$ change the output, and another which asks how changes in $x_2$ change the output.\n",
    "\n",
    "These are denoted:\n",
    "\n",
    "\\\\[\n",
    "\\fpartial{}{x_1} f(x_1, x_2)\n",
    "\\\\\n",
    "\\fpartial{}{x_2} f(x_1, x_2)\n",
    "\\\\]\n",
    "\n",
    "They are called *partial derivatives*. You calculate them just like normal derivatives, except if you're doing a partial with respect to (written *wrt*) $x_1$, you just treat $x_2$ like a constant (and vice versa).\n",
    "\n",
    "A gradient is just the vector of partial derivatives.\n",
    "\n",
    "\\\\[\n",
    "\\grad f\\left((x_1, x_2)\\right)\n",
    "=\n",
    "\\left(\n",
    "    \\fpartial{}{x_1} f(x_1, x_2), \\fpartial{}{x_2} f(x_1, x_2)\n",
    "\\right)\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. What is the shape of $\\grad_{W^{(3)}} CE(h^{(3)}, y)$? What does the entry at position $(i, j)$ of this matrix represent?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape is $(256, 10)$. The entry at position $(i, j)$ represents how changing $W^{(3)}_{i, j}$ would change the loss $CE(h^{(3)}, y)$. That is:\n",
    "\n",
    "\\\\[\n",
    "\\left(\n",
    "    \\grad_{W^{(3)}} CE(h^{(3)}, y)\n",
    "\\right)_{i, j}\n",
    "=\n",
    "\\fpartial{}{W^{(3)}_{i, j}}\n",
    "CE(h^{(3)}, y)\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. How is it that changing a weight in $W^{(3)}$ might change the loss?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A change to $W^{(3)}_{i, j}$ changes $z^{(3)}_j$, which changes all the $h^{(3)}$ values. Since that changes the probability of the correct class $h^{(3)}_{y^*}$, this will change the cross entropy loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation to $z^{(3)}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to calculate $\\grad_{W^{(3)}} CE(h^{(3)}, y)$, we will first calculate $\\grad_{z^{(3)}} CE(h^{(3)}, y)$. Once we know how changing $z^{(3)}$ changes the cross entropy, we can then think about how changing $W^{(3)}$ changes $z^{(3)}$.\n",
    "\n",
    "Sometimes it makes sense to break the process up further:\n",
    "\n",
    "1. See how changes to $h^{(3)}$ change the loss $CE(h^{(3)}, y)$.\n",
    "2. See how changes to $z^{(3)}$ change $h^{(3)}$.\n",
    "3. See how changes to $W^{(3)}$ change $z^{(3)}$.\n",
    "\n",
    "In the case of the cross entropy loss plus a softmax, it turns out to be nicer to combine steps one and two."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. What is the shape of $\\grad_{z^{(3)}} CE(h^{(3)}, y)$? What does each entry represent?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape is $(10,)$. Each entry represents how a change in each of ten $z^{(3)}_i$ values would change the loss. Formula-wise this is:\n",
    "\n",
    "\\\\[\n",
    "\\left(\n",
    "    \\grad_{z^{(3)}} CE(h^{(3)}, y)\n",
    "\\right)_i\n",
    "=\n",
    "\\fpartial{}{z^{(3)}_i}\n",
    "CE(h^{(3)}, y)\n",
    "\\\\]\n",
    "\n",
    "Note how I wrapped the gradient formula in parentheses and subscripted by $i$. This is how I write \"get the $i$-th component of the value of this formula wrapped in parentheses.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Recall that $h^{(3)} = \\text{SOFTMAX}\\left(z^{(3)}\\right)$. So before anything, let's write $CE_\\text{vector}(h^{(3)}, y)$ in terms of the $z^{(3)}_i$ values by expanding the formulas for cross-entropy and for $h^{(3)}$.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    CE\\left(h^{(3)}, y\\right)\n",
    "&=\n",
    "    -\\log h^{(3)} \\cdot y\n",
    "\\\\\n",
    "&=\n",
    "    -\\log \\sum_{i = 0}^{9} h^{(3)}_i y_i\n",
    "\\\\\n",
    "&=\n",
    "    -\\log \\sum_{i = 0}^{9}\n",
    "        y_i\n",
    "        \\frac{\n",
    "            \\exp\\left(z^{(3)}_i\\right)\n",
    "        }{\n",
    "            \\sum_{j=0}^9\n",
    "            \\exp\\left(z^{(3)}_j\\right)\n",
    "        }\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**2. Only one term of the sum above matters. Which one? Why? Write the formula without the summation.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only term that matters is for $i = y^*$. That's because only the probability on the correct class matters. The other $y_i$ values are all zero.\n",
    "\n",
    "\\\\[\n",
    "\\begin{align}\n",
    "    CE\\left(h^{(3)}, y\\right)\n",
    "&=\n",
    "    -\\log \\sum_{i = 0}^{9}\n",
    "        y_i\n",
    "        \\frac{\n",
    "            \\exp\\left(z^{(3)}_i\\right)\n",
    "        }{\n",
    "            \\sum_{j=0}^9\n",
    "            \\exp\\left(z^{(3)}_j\\right)\n",
    "        }\n",
    "\\\\\n",
    "&=\n",
    "    -\\log\\left(\n",
    "        y_{y^*}\n",
    "        \\frac{\n",
    "            \\exp\\left(z^{(3)}_{y^*}\\right)\n",
    "        }{\n",
    "            \\sum_{j=0}^9\n",
    "            \\exp\\left(z^{(3)}_j\\right)\n",
    "        }\n",
    "    \\right)\n",
    "\\\\\n",
    "&=\n",
    "    -\\log\\left(\n",
    "        \\frac{\n",
    "            \\exp\\left(z^{(3)}_{y*}\\right)\n",
    "        }{\n",
    "            \\sum_{j=0}^9\n",
    "            \\exp\\left(z^{(3)}_j\\right)\n",
    "        }\n",
    "    \\right)\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Why are all the $z^{(3)}$ values still present in the formula?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because they are all part of the softmax calculation because they represents odds relative to each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Write this log in terms of a difference of logs.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    CE\\left(h^{(3)}, y\\right)\n",
    "&=\n",
    "    -\\log\\left(\n",
    "        \\frac{\n",
    "            \\exp\\left(z^{(3)}_{y*}\\right)\n",
    "        }{\n",
    "            \\sum_{j=0}^9\n",
    "            \\exp\\left(z^{(3)}_j\\right)\n",
    "        }\n",
    "    \\right)\n",
    "\\\\\n",
    "&=\n",
    "    -\\log \\left(\\exp\\left(z^{(3)}_{y*}\\right)\\right)\n",
    "    +\n",
    "    \\log \\left(\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)\\right)\n",
    "\\\\\n",
    "&=\n",
    "    -z^{(3)}_{y*}\n",
    "    +\n",
    "    \\log \\left(\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)\\right)\n",
    "\\end{align}\n",
    "\\\\]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. We're trying to calculate $\\grad_{z^{(3)}} CE(h^{(3)}, y)$. That means calculating each partial derivative $\\fpartial{}{z^{(3)}_i} CE(h^{(3)}, y)$. What is the partial of the first term when $i \\ne y^*$? What is the partial of the first term when $i = y^*$?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first term is just $-z^{(3)}_{y*}$, so if we differentiate with respect to some $z^{(3)}_i$ where $i$ is *not* the correct class, then this derivative is zero.\n",
    "\n",
    "Otherwise, the derivative is $-1$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Why is this term zero when $i\\ne y^*$? Why is it negative when $i = y^*$?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is saying that changing the $z^{(3)}_i$ values for the wrong classes doesn't change the numerator. But when you change it for the *right* class, it changes the numerator, which increases the probability of the correct class, which reduces the cross-entropy loss.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Now the second term! Using the rule that the derivative of $\\log a$ wrt $a$ is $\\frac{1}{a}$, and also the chain rule, do the first-step of the deriative of the second term wrt $z^{(3)}_i$. You don't have to differentiate the inside of the log yet.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    \\fpartial{}{z^{(3)}_i}\n",
    "        \\log \\left(\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)\\right)\n",
    "&=\n",
    "    \\frac{1}{\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)}\n",
    "    \\fpartial{}{z^{(3)}_i}\n",
    "        \\left(\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)\\right)\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. Next, use the rule that the derivative of $e^a$ wrt $a$ is also $e^a$. Also eliminate unnecessary terms in the sum.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    \\frac{1}{\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)}\n",
    "    \\fpartial{}{z^{(3)}_i}\n",
    "        \\left(\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)\\right)\n",
    "&=\n",
    "    \\frac{1}{\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)}\n",
    "    \\exp\\left(z^{(3)}_i\\right)\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9. Finally, use the definition of $h^{(3)}_i = SOFTMAX(z^{(3)})_i$ to simplify this.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    \\frac{1}{\\sum_{j=0}^9 \\exp\\left(z^{(3)}_j\\right)}\n",
    "    \\exp\\left(z^{(3)}_i\\right)\n",
    "&=\n",
    "    h^{(3)}_i\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**10. Why is the partial of the second term the same formula no matter whether $i$ is $y^*$?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the second term is about the change to the denominator, and you change the denominator regardless of whether you are changing the correct class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**11. Give the overall formula for the partial when $i \\ne y^*$ and for $i = y^*$.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h^{3}_i$ and $-1.0 + h^{3}_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**12. Let's use these to write the gradient as a vector formula. Note that you want to subtract $1.0$ from exactly one term of $h^{(3)}$ and nothing from the rest...**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\grad_{z^{(3)}} CE(h^{(3)}, y)\n",
    "=\n",
    "h^{(3)} - y\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**13. Let's do an intuition check. What entries of the gradient are positive? Which are negative? Why?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All entries except $y^*$ are positive, because increasing their log odds decreases the probability on the right answer. That increases the loss.\n",
    "\n",
    "Increasing $z^{(3)}_{y^*}$ increases the probability of the correct answer, so it reduces the loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backprop to $W^{(3)}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. To update the weights $W^{(3)}$ we must calculate $\\grad_{W^{(3)}} CE(h^{(3)}, y)$. What is the shape of this \"2 dimensional gradient\"?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$(256, 10)$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. What does each entry $\\left(\\grad_{W^{(3)}} CE(h^{(3)}, y)\\right)_{i, j}$ mean?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It means how much the loss will change if we changed $W_{i, j}$ a little.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Which value in the second hidden layer does $W_{i, j}$ connect to which pre-activation in the third layer?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h^{(2)}_i$ to $z^{(3)}_j$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. If for a given $j$ we have $\\fpartial{}{z^{(3)}_j} CE(h^{(3)}, y)$ is zero, what is $\\grad_{W^{(3)}} CE(h^{(3)}, y)$ at position $(i, j)$ for all $i$ and our given $j$? Why?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It must be zero. Because changing any $W_{i, j}$ may change $z^{(3)}_j$, but we know that has no impact on the loss.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. If for a given $i$ we have $h^{(2)}_i = 0$, what is $\\grad_{W^{(3)}} CE(h^{(3)}, y)$ at position $(i, j)$ for all $j$ and our given $i$? Why?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It must be zero. Changing any $W_{i, j}$ won't change $z^{(3)}_j$, because $z^{(3)}_j$ is a weighted sum of the $h^{(2)}$ values. If $h^{(2)}_i$ is zero, changing the associated weight doesn't change $z^{(3)}_j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Given the above, what two \"forces\" does $\\left(\\grad_{W^{(3)}} CE(h^{(3)}, y)\\right)_{i, j}$ need to combine?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the amount that changing $W^{(3)}_{i, j}$ changes $z^{(3)}_j$.\n",
    "\n",
    "Second, the amount that changing $z^{(3)}_j$ changes the loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Apply the chain rule to $\\fpartial{}{W^{(3)}_{i, j}} CE(h^{(3)}, y)$. Break this up into (a) how does $W^{(3)}_{i, j}$ change $z^{(3)}_j$ and (b) how does $z^{(3)}_j$ change the cross-entropy. Write a product of two partials, but you don't need to evaluate the partials.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "\\fpartial{}{W^{(3)}_{i, j}} CE(h^{(3)}, y)\n",
    "&=\n",
    "\\left(\n",
    "    \\fpartial{}{W^{(3)}_{i, j}} z^{(3)}_j\n",
    "\\right)\n",
    "\\left(\n",
    "    \\fpartial{}{z^{(3)}_j} CE(h^{(3)}, y)\n",
    "\\right)\n",
    "\\end{align}\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. We must calculate $\\fpartial{}{W^{(3)}_{i, j}} z^{(3)}_j$. What is it? Use the formula for $z^{(3)}_j$.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    \\fpartial{}{W^{(3)}_{i, j}} z^{(3)}_j\n",
    "&=\n",
    "    \\fpartial{}{W^{(3)}_{i, j}} \\left(\n",
    "        \\sum_{k=0}^{256} h^{(2)}_k W_{k, j}\n",
    "    \\right) + b^{(3)}_j\n",
    "\\\\\n",
    "&=\n",
    "    h^{(2)}_i\n",
    "\\end{align}\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9. We calculated $\\fpartial{}{z^{(3)}_j} CE(h^{(3)}, y)$ before. What is it?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "(h^{(3)} - y)_j\n",
    "\\\\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**10. Using the last two answers, what is $\\fpartial{}{W^{(3)}_{i, j}} CE(h^{(3)}, y)$ then?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\fpartial{}{W^{(3)}_{i, j}} CE(h^{(3)}, y)\n",
    "=\n",
    "    h^{(2)}_i\n",
    "    (h^{(3)} - y)_j\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**11. We want $\\grad_{W^{(3)}} CE(h^{(3)}, y)$, which is a matrix. Each entry of the matrix consists of a product of two factors as above. What is the first factor for every entry in row $i$?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h^{(2)}_i$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**11. What is the second factor for every entry in column $j$?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$(h^{(3)} - y)_j$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**12. If a vector $u$ has length $a$ and a vector $v$ has length $b$, what is the *outer product* $u \\otimes v$? What is its shape?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape is $(a, b)$ and entry $(i, j)$ is equal to $u_i v_j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**13. Can you write $\\grad_{W^{(3)}} CE(h^{(3)}, y)$ as an outer product of two vectors?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\grad_{W^{(3)}} CE(h^{(3)}, y)\n",
    "=\n",
    "h^{(2)} \\otimes (h^{(3)} - y)\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**14. You just backpropagated a derivative. Way to go!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks! I'm so proud!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backprop to $b^{(3)}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. We next want to find $\\grad_{b^{(3)}} CE(h^{(3)}, y)$. What is the shape of this gradient? Why?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is $(10,)$ since there are ten biases.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Let's consider a single $j$ index. Break $\\fpartial{}{b^{(3)}_j} CE(h^{(3)}, y)$ into two partial factors as before.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\fpartial{}{b^{(3)}_j} CE(h^{(3)}, y)\n",
    "=\n",
    "\\left(\n",
    "    \\fpartial{}{b^{(3)}_j} z^{(3)}_j\n",
    "\\right)\n",
    "\\left(\n",
    "    \\fpartial{}{z^{(3)}_j} CE(h^{(3)}, y)\n",
    "\\right)\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Evaluate the first partial.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\begin{align}\n",
    "    \\fpartial{}{b^{(3)}_j} z^{(3)}_j\n",
    "&=\n",
    "    \\fpartial{}{b^{(3)}_j} \\left(\n",
    "        \\sum_{k=0}^{256} h^{(2)}_k W_{k, j}\n",
    "    \\right) + b^{(3)}_j\n",
    "\\\\\n",
    "&=\n",
    "    0 + 1\n",
    "=\n",
    "    1\n",
    "\\end{align}\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Therefore, what is the partial overall?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\fpartial{}{b^{(3)}_j} CE(h^{(3)}, y)\n",
    "=\n",
    "1\n",
    "\\left(\n",
    "    h^{(3)} - y\n",
    "\\right)_j\n",
    "=\n",
    "\\left(\n",
    "    h^{(3)} - y\n",
    "\\right)_j\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Using this, give me a vector formula for the gradient $\\grad_{b^{(3)}} CE(h^{(3)}, y)$.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\\\[\n",
    "\\grad_{b^{(3)}} CE(h^{(3)}, y)\n",
    "=\n",
    "h^{(3)} - y\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. What must always be the relationship between $\\grad_{b^{i}} CE(h^{(3)}, y)$ and $\\grad_{z^{(i)}} CE(h^{(3)}, y)$? Why?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They must always be equal. Because the change in $b^{i}$ induces an identical to a change in $z^{(i)}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (default)",
   "language": "python",
   "name": "conda-default"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}