Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression in Python and Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In logistic regression, we are trying to model the outcome of a **binary variable** given a **linear combination of input features**. For example, we could try to predict the outcome of an election (win/lose) using information about how much money a candidate spent campaigning, how much time she/he spent campaigning, etc.\n",
    "\n",
    "### Model \n",
    "\n",
    "Logistic regression works as follows.\n",
    "\n",
    "**Given:** \n",
    "- dataset $\\{(\\boldsymbol{x}^{(1)}, y^{(1)}), ..., (\\boldsymbol{x}^{(m)}, y^{(m)})\\}$\n",
    "- with $\\boldsymbol{x}^{(i)}$ being a $d-$dimensional vector $\\boldsymbol{x}^{(i)} = (x^{(i)}_1, ..., x^{(i)}_d)$\n",
    "- $y^{(i)}$ being a binary target variable, $y^{(i)} \\in \\{0,1\\}$\n",
    "\n",
    "The logistic regression model can be interpreted as a very **simple neural network:**\n",
    "- it has a real-valued weight vector $\\boldsymbol{w}= (w^{(1)}, ..., w^{(d)})$\n",
    "- it has a real-valued bias $b$\n",
    "- it uses a sigmoid function as its activation function\n",
    "\n",
    "![title](figures/logistic_regression.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Different to [linear regression](linear_regression.ipynb), logistic regression has no closed form solution. But the cost function is convex, so we can train the model using gradient descent. In fact, **gradient descent** (or any other optimization algorithm) is guaranteed to find the global minimum (if the learning rate is small enough and enough training iterations are used).  \n",
    "\n",
    "Training a logistic regression model has different steps. In the beginning (step 0) the parameters are initialized. The other steps are repeated for a specified number of training iterations or until convergence of the parameters.\n",
    "\n",
    "* * * \n",
    "**Step 0: ** Initialize the weight vector and bias with zeros (or small random values).\n",
    "* * *\n",
    "\n",
    "**Step 1: ** Compute a linear combination of the input features and weights. This can be done in one step for all training examples, using vectorization and broadcasting:\n",
    "$\\boldsymbol{a} = \\boldsymbol{X} \\cdot \\boldsymbol{w} + b $\n",
    "\n",
    "where $\\boldsymbol{X}$ is a matrix of shape $(n_{samples}, n_{features})$ that holds all training examples, and $\\cdot$ denotes the dot product.\n",
    "* * *\n",
    "\n",
    "**Step 2: ** Apply the sigmoid activation function, which returns values between 0 and 1:\n",
    "\n",
    "$\\boldsymbol{\\hat{y}} = \\sigma(\\boldsymbol{a}) = \\frac{1}{1 + \\exp(-\\boldsymbol{a})}$\n",
    "* * *\n",
    "\n",
    "** Step 3: ** Compute the cost over the whole training set. We want to model the probability of the target values being 0 or 1. So during training we want to adapt our parameters such that our model outputs high values for examples with a positive label (true label being 1) and small values for examples with a negative label (true label being 0). This is reflected in the cost function:\n",
    "\n",
    "$J(\\boldsymbol{w},b) = - \\frac{1}{m} \\sum_{i=1}^m \\Big[ y^{(i)} \\log(\\hat{y}^{(i)}) + (1 - y^{(i)}) \\log(1 - \\hat{y}^{(i)}) \\Big]$\n",
    "* * *\n",
    "\n",
    "** Step 4: ** Compute the gradient of the cost function with respect to the weight vector and bias. A detailed explanation of this derivation can be found [here](https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated).\n",
    "\n",
    "The general formula is given by:\n",
    "\n",
    "$ \\frac{\\partial J}{\\partial w_j} = \\frac{1}{m}\\sum_{i=1}^m\\left[\\hat{y}^{(i)}-y^{(i)}\\right]\\,x_j^{(i)}$\n",
    "\n",
    "For the bias, the inputs $x_j^{(i)}$ will be given 1.\n",
    "* * *\n",
    "\n",
    "** Step 5: ** Update the weights and bias\n",
    "\n",
    "$\\boldsymbol{w} = \\boldsymbol{w} - \\eta \\, \\nabla_w J$  \n",
    "\n",
    "$b = b - \\eta \\, \\nabla_b J$\n",
    "\n",
    "where $\\eta$ is the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "class LogisticRegression:\n",
    "    \n",
    "    def fit(self, X, y, lr = 0.001, epochs=10000, verbose=True, batch_size=1):\n",
    "        self.classes = np.unique(y)\n",
    "        y = (y==self.classes[1]) * 1\n",
    "        X = self.add_bias(X)\n",
    "        self.weights = np.zeros(X.shape[1])\n",
    "        self.loss = []\n",
    "        for i in range(epochs):\n",
    "            self.loss.append(self.cross_entropy(X,y))\n",
    "            if i % 1000 == 0 and verbose: \n",
    "                print('Iterations: %d - Error : %.4f' %(i, self.loss[i]))\n",
    "            idx = np.random.choice(X.shape[0], batch_size)\n",
    "            X_batch, y_batch =  X[idx], y[idx]\n",
    "            self.weights -= lr * self.get_gradient(X_batch, y_batch)\n",
    "        return self\n",
    "    \n",
    "    def get_gradient(self, X, y):\n",
    "        return -1.0 * (y - self.predict_(X)).dot(X) / len(X)\n",
    "    \n",
    "    def predict_(self, X):\n",
    "        return self.sigmoid(np.dot(X, self.weights))\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return self.predict_(self.add_bias(X))\n",
    "    \n",
    "    def sigmoid(self, z):\n",
    "        return 1.0/(1 + np.exp(-z))\n",
    "    \n",
    "    def predict_classes(self, X):\n",
    "        return self.predict_classes_(self.add_bias(X))\n",
    "\n",
    "    def predict_classes_(self, X):\n",
    "        return np.vectorize(lambda c: self.classes[1] if c>=0.5 else self.classes[0])(self.predict_(X))\n",
    "    \n",
    "    def cross_entropy(self, X, y):\n",
    "        p = self.predict_(X)\n",
    "        return (-1 / len(y)) * (y * np.log(p)).sum()\n",
    "\n",
    "    def add_bias(self,X):\n",
    "        return np.insert(X, 0, 1, axis=1)\n",
    "\n",
    "    def score(self, X, y):\n",
    "        return self.cross_entropy(self.add_bias(X), y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "def train_model(X, y, model):\n",
    "    model.fit(X, y, lr=0.1)\n",
    "    pre = model.predict_classes(X)\n",
    "    print('Accuracy :: ', accuracy_score(y, pre))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iterations: 0 - Error : 69.3147\n",
      "Iterations: 1000 - Error : 0.4746\n",
      "Iterations: 2000 - Error : 0.3847\n",
      "Iterations: 3000 - Error : 0.1645\n",
      "Iterations: 4000 - Error : 0.1280\n",
      "Iterations: 5000 - Error : 0.1126\n",
      "Iterations: 6000 - Error : 0.0783\n",
      "Iterations: 7000 - Error : 0.0674\n",
      "Iterations: 8000 - Error : 0.0621\n",
      "Iterations: 9000 - Error : 0.0664\n",
      "('Accuracy :: ', 1.0)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "X, y = load_iris(return_X_y=True)\n",
    "lr = LogisticRegression()\n",
    "train_model(X,(y !=0 )*1, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAGDCAYAAADHzQJ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3XmcJXV97//Xu7tngQEZBkbCpoOIJi6J4mhQs+BylRgVc39G8XoVo5FrzGqSqxIT482qMVFjTMw1omhUlKCJmLgRhBhvFBxcWF1GdhxgWIRhmbU/vz/q23BseqZ7Zrr7TBev5+NxHl31rW/V+Zzqmn6fWqYqVYUkSeqXkWEXIEmSZp8BL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8NJ2JDk3yS8Pu475lOThSb6RZEOS3xh2PQtdkgcluSPJ6Gz2lWbCgNceLcmVSe5ugfODJP+V5FVJ3HanMAtfSl4LnFNV+1bVO2erru1JcmqSP5nr99kVSV6W5Eu7s4yqurqq9qmqbbPZV5oJ/0hqIXhOVe0LPBh4M/A64JThltRbDwYuGXYR8y3J2C7O59629lgGvBaMqrqtqs4EXgicmORRAEmWJPnLJFcnuSHJ3yfZq027LMmzJ5aRZCzJ+iRHt/Fj2lGBHyT5ZpJjp3rvJCNJfj/JVUluTPLBJPu1aauSVJKTknw/ybokvzsw75uS/FOSD7UjERcleViSk9uyrknyjIH++yU5pS3nuiR/MhEkE3uV7fPemuSKJD/Xpv0p8NPAu9qh3ndt57M8N8kl7TOfm+THWvsXgKcMzP+wKeZdkeT97XPemuRfBqa9MsnaJLckOTPJIa09Sd7ePuvt7fM/KslJwIuB17b3+9R26n1Skq8mua39fFJrf2GSNZP6vibJmTPYLo5Ncm2S1yW5Hnj/pOX8GPD3wBNbbT9o7acmeXeSTye5E3hKkp9P8vX22a5J8qaB5UxsG2Nt/Nwkf5zk/7Vt4fNJDtzZvm36S9v2eHOSP0h3tOvpU61D3U9VlS9fe+wLuBJ4+hTtVwO/0obfDpwJrAD2BT4F/Hmb9kbgwwPz/TxwWRs+FLgZeBbdl93/1sZXtunnAr/chl8OrAUeAuwDfAL4xzZtFVDAacAy4NHA+om6gTcBG4FnAmPAB4ErgDcAi4BXAlcM1PjPwP9ty3ogcD7wv9q0lwFb2jyjwK8A3wcyuebtrM+HAXe2z7qI7pD8WmDxDOf/N+BjwP5t/p9t7U8FbgKOBpYAfwN8sU17JnABsBwI8GPAwW3aqcCf7OD9VgC3Ai9p6+5FbfwAYG9gA3DUQP+vAifMYLs4FtgKvKXVu9cU7/0y4EuT2k4FbgOeTLfNLG3LenQb/3HgBuB5k7aNsYH1+732e9irjb95F/o+ArgD+ClgMfCXbbu4z78VX/ff19AL8OVrRy+2H/BfoQvItMA6cmDaE2mBCTy0hcDebfzDwBvb8OtoIT0w7+eAE9vwudwb8GcDrx7o9/D2B3Vs4A/zjw5M/wvglDb8JuCsgWnPaX+cR9v4vm3+5cBBwKbBwKELtXPa8MuAtQPT9m7z/sjkmrezPv8AOH1gfAS4Djh2uvmBg4FxYP8ppp0C/MXA+D5t/ayiC//vAMcAI5PmO5UdB/xLgPMntX0ZeFkb/tDA7/Ooid/1DLaLY4HNwNIdvPfLmDrgPzjNNvsO4O1teGLbGAzt3x/o+2rgs7vQ943AaZO2g80Y8L4GXrt03knaAxwK3AKspPvjdkGSiWmh27ulqtYmuQx4TjsE/Fzgsa3fg4FfTPKcgeUuAs6Z4v0OAa4aGL+KLtwPGmi7ZtL0Rw+M3zAwfDdwU917MdXd7ec+7X0WAesGPs/IpGVfPzFQVXe1fvtMUfNUfuhzVNV4kmvo1ud0Dgduqapbt7Pcrw0s944kNwOHVtUX2umCvwUenOQTwO9W1e07W29z1UC9HwH+Cvgj4H8A/9LWyQPZwXbRrK+qjTOoYbLB3wVJfpLu2pBH0e1NLwH+aQfzXz8wfBc7/t1tr+8hg3W0z3zztJXrfsVz8Fpwkjye7g/8l+gOC98NPLKqlrfXflU1+EfzNLq94OOBS6tqbWu/hm4PfvnAa1lVvXmKt/0+3ReCCQ+iO8Q7GNyHT5r+/V34eNfQ7cEfOFDTA6rqkTOcf7rHQ/7Q50iXfofT7cXPpLYVSZbPYLnL6A6jXwdQVe+sqsfRHVp+GPC/d6Xe5kED9Z4FrEzyGLrf8Uda+0y2i+nee3vTJ7d/hO5UwOFVtR/dufvcZ67ZtQ44bGKkXVtwwBy/pxYYA14LRpIHpLtg7qPAh6rqoqoaB/4BeHvbayPJoUmeOTDrR4Fn0J2v/shA+4fo9uyfmWQ0ydJ28dVh3NdpwGuSHJFkH+DPgI9V1daBPn+QZO8kjwR+ie5c9U6pqnXA54G/ap93JMmRSX52hou4ge46ge05Hfj5JE9Lsgj4HbovFP81w9o+A/xdkv2TLEryM23yacAvJXlMkiV06+e8qroyyeOT/GR7vzvprkcYn2G9nwYeluR/pLtA8oV0XxL+tdW0hW5v+a1059rPau0z2S6mcwNwWJLF0/Tbl+7IxsYkT6A7kjDXzqDbdp/U6nsTc/+lQguMAa+F4FNJNtDtQb4BeBtdgE54Hd2FYl9Jcjvw73TnyIF7gunLwJMYCN2quoZur/736C6Ku4Zuz3KqfxfvA/4R+CLdBXIbgV+f1Oc/Wh1nA39ZVZ/ftY/LS+kO9V5Kd0HZGXTnv2fir4Hnp7vC/T7/j72qvg38T7qL4G6iux7gOVW1eYbLfwndufVvATcCv9WW++905/c/Trd3eSRwQpvnAXRheyvd4fWb6QIZunP3j0h3Rf89V+QP1Hsz8Gy6LyI3010U+Oyqummg20eApwP/NOkL1w63ixn4At1/Gbw+yU076Pdq4I/aNvpGui9Rc6qqLqHb/j5Kt77voPt9bJrr99bCMXHlraRdlGQVXegvmhQw0rxoR5V+QPc/Cq4Ydj3aM7gHL0kLUJLntFNCy+j+m9xFdP/rRAIMeElaqI6nuwjx+3T/RfCE8pCsBniIXpKkHnIPXpKkHjLgJUnqoQV9J7sDDzywVq1aNewyJEmaNxdccMFNVbVyun4LOuBXrVrFmjVrpu8oSVJPJJl8++YpeYhekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQemrOAT/K+JDcmuXiKab+TpJIc2MaT5J1J1ia5MMnRc1WXJEn3B3O5B38qcNzkxiSHA88Arh5o/jm6pyEdBZwEvHsO65IkqffmLOCr6ovALVNMejvwWmDwMXbHAx+szleA5UkOnqvaJEnqu3k9B5/keOC6qvrmpEmHAtcMjF/b2iRJ0i6Yt4BPsjfwe8Abd3M5JyVZk2TN+vXrZ6e4Zu2Nd1BV03eUJGkPN5978EcCRwDfTHIlcBjwtSQ/AlwHHD7Q97DWdh9V9Z6qWl1Vq1eunPZhOjN2wVW38vS3/Qen/teVs7ZMSZKGZd4CvqouqqoHVtWqqlpFdxj+6Kq6HjgTeGm7mv4Y4LaqWjdftQFcdfOdAFx47W3z+baSJM2JufxvcqcBXwYenuTaJK/YQfdPA5cDa4F/AF49V3VJknR/MGfPg6+qF00zfdXAcAG/Ole1SJJ0f+Od7CRJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAnqaphlyBJ0m4z4Jtk2BVIkjR7DHhJknrIgG88Mi9J6hMDfpJ4rF6S1AMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPTRnAZ/kfUluTHLxQNtbk3wryYVJ/jnJ8oFpJydZm+TbSZ45V3VJknR/MJd78KcCx01qOwt4VFX9OPAd4GSAJI8ATgAe2eb5uySjc1ibJEm9NmcBX1VfBG6Z1Pb5qtraRr8CHNaGjwc+WlWbquoKYC3whLmqTZKkvhvmOfiXA59pw4cC1wxMu7a13UeSk5KsSbJm/fr1c1yiJEkL01ACPskbgK3Ah3d23qp6T1WtrqrVK1eunP3iJEnqgbH5fsMkLwOeDTyt6p5nuF0HHD7Q7bDWJkmSdsG87sEnOQ54LfDcqrprYNKZwAlJliQ5AjgKOH8+a5MkqU/mbA8+yWnAscCBSa4F/pDuqvklwFntsaxfqapXVdUlSU4HLqU7dP+rVbVtrmqTJKnv5izgq+pFUzSfsoP+fwr86VzVI0nS/Yl3spMkqYcMeEmSesiAlySphwx4SZJ6yICXJKmHDHhJknrIgJ/k3pvrSZK0cBnwTXffHUmS+sGAlySphwx4SZJ6yICXJKmHDPjGa+skSX1iwE8Sr7aTJPWAAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JNU1bBLkCRptxnwTTLsCiRJmj1zFvBJ3pfkxiQXD7StSHJWku+2n/u39iR5Z5K1SS5McvRc1SVJ0v3BXO7BnwocN6nt9cDZVXUUcHYbB/g54Kj2Ogl49xzWJUlS781ZwFfVF4FbJjUfD3ygDX8AeN5A+wer8xVgeZKD56q2qYyPdz8vXXf7fL6tJElzYr7PwR9UVeva8PXAQW34UOCagX7Xtrb7SHJSkjVJ1qxfv37WCrv+9o0AfOeGO2ZtmZIkDcvQLrKr7nL1nb5kvareU1Wrq2r1ypUr56AySZIWvvkO+BsmDr23nze29uuAwwf6HdbaJEnSLpjvgD8TOLENnwh8cqD9pe1q+mOA2wYO5UuSpJ00NlcLTnIacCxwYJJrgT8E3gycnuQVwFXAC1r3TwPPAtYCdwG/NFd1SZJ0fzBnAV9VL9rOpKdN0beAX52rWiRJur/xTnaSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfBN97wbSZL6wYCXJKmHDHhJknrIgJckqYcM+CbJsEuQJGnWGPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8E1VDbsESZJmjQEvSVIPDSXgk7wmySVJLk5yWpKlSY5Icl6StUk+lmTxMGqTJKkP5j3gkxwK/AawuqoeBYwCJwBvAd5eVQ8FbgVeMd+1SZLUF8M6RD8G7JVkDNgbWAc8FTijTf8A8Lwh1SZJ0oI37wFfVdcBfwlcTRfstwEXAD+oqq2t27XAofNdmyRJfTGMQ/T7A8cDRwCHAMuA43Zi/pOSrEmyZv369bNZ16wtS5KkYRvGIfqnA1dU1fqq2gJ8AngysLwdsgc4DLhuqpmr6j1VtbqqVq9cuXJ+KpYkaYEZRsBfDRyTZO90u81PAy4FzgGe3/qcCHxyCLVJktQLwzgHfx7dxXRfAy5qNbwHeB3w20nWAgcAp8x3bZIk9cXY9F1mX1X9IfCHk5ovB54whHIkSeod72QnSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPXQjAI+yT/OpE2SJO0ZZroH/8jBkSSjwONmvxxJkjQbdhjwSU5OsgH48SS3t9cG4EZ8GIwkSXusHQZ8Vf15Ve0LvLWqHtBe+1bVAVV18jzVKEmSdtJMD9H/a5JlAEn+Z5K3JXnwHNYlSZJ2w0wD/t3AXUl+Avgd4HvAB+esKkmStFtmGvBbq6qA44F3VdXfAvvOXVnzr/t4kiT1w0yfB78hycnAS4CfTjICLJq7siRJ0u6Y6R78C4FNwMur6nrgMOCtc1bVENy4YdOwS5AkadbMKOBbqH8Y2C/Js4GNVdWrc/DfW3/HsEuQJGnWzPROdi8Azgd+EXgBcF6S589lYZIkadfN9Bz8G4DHV9WNAElWAv8OnDFXhUmSpF0303PwIxPh3ty8E/NKkqR5NtM9+M8m+RxwWht/IfDpuSlJkiTtrh0GfJKHAgdV1f9O8t+Bn2qTvkx30Z0kSdoDTbcH/w7gZICq+gTwCYAkj27TnjOn1UmSpF0y3Xn0g6rqosmNrW3VnFQkSZJ223QBv3wH0/aazUIkSdLsmS7g1yR55eTGJL8MXDA3JUmSpN013Tn43wL+OcmLuTfQVwOLgV+Yy8IkSdKu22HAV9UNwJOSPAV4VGv+t6r6wpxXJkmSdtmM/h98VZ0DnDPHtUiSpFni3egkSeohA16SpB4y4JuQYZcgSdKsMeAlSeqhoQR8kuVJzkjyrSSXJXlikhVJzkry3fZz/2HUJklSHwxrD/6vgc9W1Y8CPwFcBrweOLuqjgLObuOSJGkXzHvAJ9kP+BngFICq2lxVPwCOBz7Qun0AeN581yZJUl8MYw/+CGA98P4kX0/y3iTL6B5ss671uR44aKqZk5yUZE2SNevXr5+1ouI1dpKkHhlGwI8BRwPvrqrHAncy6XB8VRVQU81cVe+pqtVVtXrlypVzXqwkSQvRMAL+WuDaqjqvjZ9BF/g3JDkYoP28cQi1SZLUC/Me8FV1PXBNkoe3pqcBlwJnAie2thOBT853bZIk9cWM7kU/B34d+HCSxcDlwC/Rfdk4PckrgKuAFwypNkmSFryhBHxVfYPusbOTPW2+a5EkqY+8k11TU17SJ0nSwmTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwDfJsCuQJGn2GPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAHfxMvoJUk9YsBLktRDBrwkST1kwEuS1EMGvCRJPWTAN1U17BIkSZo1BrwkST1kwEuS1EMGvCRJPWTAS5LUQwb8FO7evG3YJUiStFsM+Cls2Lhl2CVIkrRbDHhJknrIgJckqYcMeEmSesiAlySph4YW8ElGk3w9yb+28SOSnJdkbZKPJVk8rNrw0fCSpAVumHvwvwlcNjD+FuDtVfVQ4FbgFUOpCsDb0kuSFrihBHySw4CfB97bxgM8FTijdfkA8Lxh1CZJUh8Maw/+HcBrgfE2fgDwg6ra2savBQ6dz4K67xiSJPXDvAd8kmcDN1bVBbs4/0lJ1iRZs379+tmra7sjkiQtPMPYg38y8NwkVwIfpTs0/9fA8iRjrc9hwHVTzVxV76mq1VW1euXKlfNRryRJC868B3xVnVxVh1XVKuAE4AtV9WLgHOD5rduJwCfnuzZJkvpiT/p/8K8DfjvJWrpz8qcMrRKvopckLXBj03eZO1V1LnBuG74ceMLQahnWG0uSNAf2pD34PYcX2UmSFjgDXpKkHjLgJUnqIQNekqQeMuAbT7tLkvrEgJckqYcMeEmSesiAlySphwx4SZJ6yICfQrzkTpK0wBnwkiT1kAE/hfLO9JKkBc6AlySphwx4SZJ6yICfghfZSZIWOgO+iZkuSeoRA16SpB4y4JvywnlJUo8Y8JIk9ZABL0lSDxnwkiT1kAHfeBW9JKlPDHhJknrIgJ+Ce/OSpIXOgJ+C/2VOkrTQGfCSJPWQAT+FW+/aPOwSJEnaLQb8FL703ZuGXYIkSbvFgJckqYcM+Cl4jZ0kaaEz4CVJ6iEDXpKkHjLgJUnqIQO+8eZ1kqQ+mfeAT3J4knOSXJrkkiS/2dpXJDkryXfbz/3nuzZJkvpiGHvwW4HfqapHAMcAv5rkEcDrgbOr6ijg7DYuSZJ2wbwHfFWtq6qvteENwGXAocDxwAdatw8Az5vPuuITZiRJPTLUc/BJVgGPBc4DDqqqdW3S9cBB25nnpCRrkqxZv379rNVSA0+Y2bx1fNaWK0nSMAwt4JPsA3wc+K2qun1wWnVpO+X9ZqrqPVW1uqpWr1y5ck5qe8tnv8WnL1o3fUdJkvZQQwn4JIvowv3DVfWJ1nxDkoPb9IOBG4dR24TXfOwbw3x7SZJ2yzCuog9wCnBZVb1tYNKZwIlt+ETgk/Nd26BNHqaXJC1gY0N4zycDLwEuSjKxm/x7wJuB05O8ArgKeMEQapMkqRfmPeCr6kts/74yT5vPWiRJ6ivvZCdJUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCN96KXJPWJAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8I3PkpMk9YkBL0lSDxnwO3DLnZuHXYIkSbvEgN+Bo//4LM67/Ga+fvWtVNWwy5EkacYM+Gn8n09dyi/83X/xqQvXDbsUSZJmzICfxhU33QnA9268Y8iVSJI0cwb8NEZHuuvrz/zm99mybXzI1UiSNDMG/DRavnPFTXfy7nO/N9xiJEmaIQN+Grdv3HrP8I0bNg6xEkmSZs6A3wkf+srV/Fu72O5tn/82F11725ArkiRpagb8Tvrt07/Bqtf/G+/8wlqe864vDbscSZKmZMDvpE1bvdBOkrTnM+CbRaO7tipO/+o1s1yJJEm7z4Bvfu7RP7JL87324xdy1+at92nfuGXb7pYkSdIuM+CbZz364F2e91Uf+hrv+sJ32bJtnH/88pWc/ImL+NE/+Cz/9b2b7ulTVVN+EZAkaS5kT7vHepLjgL8GRoH3VtWbt9d39erVtWbNmll5323jxZG/92kAPvObP80D913Cqz50AV+98lZee9zDeeVPP4Tvrb+D497xnzu97O/92bPuWfbLn3wERz5wGc9/3GHcsXEr+y5dxJ99+jLWb9jETx91IE888gAeuO9SbrpjE4ev2HtWPpskqT+SXFBVq6fttycFfJJR4DvAfwOuBb4KvKiqLp2q/2wGPMA/fPFynvzQA3nEIQ8A4O7N2/jMxev4hcceSnLvE+PHx4tnvfM/+db1G2btvaeyeGyEhx20D9vG4XmPOYRvXPMDbtywiX2XjrFi2WKuv20jd27exmXfv50fO+QBrH7w/ixdNMLVt9zNJd+/jUcesh9POGIFowlX3XInS8ZG2TY+zs13bObym+7k/CtuAeCYh6zgd57xcEYSNm3dxt2bt3HHpq3dPQCqWDQ6wkjC+js2cdj+e3HwfnvdcwOgzVvHuebWu7hr8zauvuUuqmC/vRaxeGyEJWMjbBsvNm4ZZ8WyRTz0gfuyeCxs2VZs3VYsWzLKAcuWsGTRCPvttah9nq1s3DLO6Ei4a9NWliwaYemiURaNjrBh41a2bBvnwQfszf57L6YKbrpjE9vGi/Eqxgug+zleRQ38HB0JDztoX7aNF0XXVgXV+lcVBdQ490wfb23jXcd7hqu6zz2SMDYa9l48yvK9F9/ze+uO1mzj1rs2MzoSRhIesHQRey0endPtRdL9w0IN+CcCb6qqZ7bxkwGq6s+n6j/bAb8rrrnlLv703y5jn6VjHLp8Lx6/agVv+tQlfPxVT+K7N27g+X//5Xv6vvrYI/m7dje8/fdexK13bQHgCatWcP6Vt/zQckdHwrbxH/7d7Lt0jAet2Jvv3LCBbePF0kWjPGjF3rv8RWPvxaPctdlrBWbD0kUjjI2MUFVs2jrO1vH7/rtaumiEJWOj7LVolK3jRdW9X0wm/h2OtYs97/nC0RYzMT7xReM+0ye+tNzTpxvfVkWAkXRfNEh3d8aJ8QAJjIwMjoeR3Hub5h2Zvgc/9OV4V8xk9un6ZAaVTr+MmdSx417TLmMmn3U3a5jZMmZSxzSfdfd+7W0Zu7k+Z1DHfHzW0155DMuWjE3/RjO0UAP++cBxVfXLbfwlwE9W1a8N9DkJOAngQQ960OOuuuqqodS6M6pqyg11fLxItr8Rbxsvto0XN9+5ifGCg/ZdwtjoCBu3bCOBJWOj9yzn6lvuYnQk/Mh+S1k0OsJdm7dy2brbOWDZEhaPjbBl2zgH7rOEvRePMl73/vHeuGUbl6+/k3W33c3oSLoAWjzKvkvH2GfJGCMJt9y5mUWj4YB9lnDdrXdz852b7tnDXTw6wuEr9mbpolFWLFvM6EgYHy82bxtn87Zx7trUHQ245Pu3ccCyJWyrYtFI2FbV9r5hw8YtbNi4lUOX78WyJWMsGRthvIq9Fo+yZVuxccs2Nm8dZ9+lXT3fvn4DW8bHCWHFskUsGRu9Zz1OhFkycZvhLqzW3bbxns+RNj3c229i+N5l3Bt0TFpu6I6uVMHW8XHW3baRDQN3PFwy1h2R2G+vRQBsGS9u2rCJu7dsY+OW7gjJ2OgIoyMDwdtMPO9gsL6J0B1sv3eYH6qbSfONjuSHjlRMHIGo2v74eHXb1HjVDv9wzeRPx3RdpltGTbuE6d9kJn/hpvs7OLNl7F4dM/lbPG2PGa2uaT7rTJYxC7+33V9f077FtEuZnW14+oX87YuPZu/FBvy0AT9oT9iDlyRpPs004Pe0q+ivAw4fGD+stUmSpJ2wpwX8V4GjkhyRZDFwAnDmkGuSJGnBmb2TArOgqrYm+TXgc3T/Te59VXXJkMuSJGnB2aMCHqCqPg18eth1SJK0kO1ph+glSdIsMOAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB7ao54mt7OSrAdm83mxBwI3zeLy7q9cj7vPdbj7XIe7z3W4++ZiHT64qlZO12lBB/xsS7JmJo/g0465Hnef63D3uQ53n+tw9w1zHXqIXpKkHjLgJUnqIQP+h71n2AX0hOtx97kOd5/rcPe5Dnff0Nah5+AlSeoh9+AlSeohA75JclySbydZm+T1w65nT5Lk8CTnJLk0ySVJfrO1r0hyVpLvtp/7t/YkeWdblxcmOXpgWSe2/t9NcuKwPtOwJBlN8vUk/9rGj0hyXltXH0uyuLUvaeNr2/RVA8s4ubV/O8kzh/NJhiPJ8iRnJPlWksuSPNGQRT1RAAAIGElEQVTtcOckeU37d3xxktOSLHU7nF6S9yW5McnFA22ztu0leVySi9o870yS3S66qu73L2AU+B7wEGAx8E3gEcOua095AQcDR7fhfYHvAI8A/gJ4fWt/PfCWNvws4DNAgGOA81r7CuDy9nP/Nrz/sD/fPK/L3wY+AvxrGz8dOKEN/z3wK2341cDft+ETgI+14Ue07XMJcETbbkeH/bnmcf19APjlNrwYWO52uFPr71DgCmCvge3vZW6HM1p3PwMcDVw80DZr2x5wfuubNu/P7W7N7sF3ngCsrarLq2oz8FHg+CHXtMeoqnVV9bU2vAG4jO4PxfF0f3BpP5/Xho8HPlidrwDLkxwMPBM4q6puqapbgbOA4+bxowxVksOAnwfe28YDPBU4o3WZvA4n1u0ZwNNa/+OBj1bVpqq6AlhLt/32XpL96P7IngJQVZur6ge4He6sMWCvJGPA3sA63A6nVVVfBG6Z1Dwr216b9oCq+kp1af/BgWXtMgO+cyhwzcD4ta1Nk7RDdI8FzgMOqqp1bdL1wEFteHvr8/6+nt8BvBYYb+MHAD+oqq1tfHB93LOu2vTbWv/78zo8AlgPvL+d5nhvkmW4Hc5YVV0H/CVwNV2w3wZcgNvhrpqtbe/QNjy5fbcY8JqxJPsAHwd+q6puH5zWvnX6XzK2I8mzgRur6oJh17KAjdEdIn13VT0WuJPusOg93A53rJ0jPp7uy9IhwDLuX0cv5syeuO0Z8J3rgMMHxg9rbWqSLKIL9w9X1Sda8w3t0BLt542tfXvr8/68np8MPDfJlXSngJ4K/DXdobux1mdwfdyzrtr0/YCbuX+vw2uBa6vqvDZ+Bl3gux3O3NOBK6pqfVVtAT5Bt226He6a2dr2rmvDk9t3iwHf+SpwVLuSdDHdxSRnDrmmPUY753YKcFlVvW1g0pnAxFWgJwKfHGh/abuS9BjgtnYY63PAM5Ls3/YkntHaeq+qTq6qw6pqFd329YWqejFwDvD81m3yOpxYt89v/au1n9Cubj4COIru4pzeq6rrgWuSPLw1PQ24FLfDnXE1cEySvdu/64l16Ha4a2Zl22vTbk9yTPu9vHRgWbtu2Fcm7ikvuqsev0N3Negbhl3PnvQCforu0NOFwDfa61l05+LOBr4L/DuwovUP8LdtXV4ErB5Y1svpLshZC/zSsD/bkNbnsdx7Ff1D6P4wrgX+CVjS2pe28bVt+kMG5n9DW7ffZhautF1IL+AxwJq2Lf4L3ZXIboc7tw7/D/At4GLgH+muhHc7nH69nUZ33cIWuqNJr5jNbQ9Y3X4n3wPeRbsR3e68vJOdJEk95CF6SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAl+ZRkkryVwPjv5vkTbO07FOTPH/6nrv9Pr+Y7klu50xqPyTJGW34MUmeNYvvuTzJq6d6L0lTM+Cl+bUJ+O9JDhx2IYMG7mI2E68AXllVTxlsrKrvV9XEF4zH0N0rYbZqWE73ZLOp3kvSFAx4aX5tBd4DvGbyhMl74EnuaD+PTfIfST6Z5PIkb07y4iTnt+dHHzmwmKcnWZPkO+3+9xPPoH9rkq+2Z1P/r4Hl/meSM+nuZja5nhe15V+c5C2t7Y10Nz46JclbJ/Vf1fouBv4IeGGSbyR5YZJl6Z6nfX57UMzxbZ6XJTkzyReAs5Psk+TsJF9r7z3xVMc3A0e25b114r3aMpYmeX/r//UkTxlY9ieSfDbds7f/YmB9nNpqvSjJfX4XUh/szLd2SbPjb4ELJwJnhn4C+DG6x1VeDry3qp6Q5DeBXwd+q/VbRffYziOBc5I8lO62l7dV1eOTLAH+X5LPt/5HA4+q7pGf90hyCPAW4HHArcDnkzyvqv4oyVOB362qNVMVWlWb2xeB1VX1a215f0Z3m9OXJ1kOnJ/k3wdq+PGquqXtxf9CVd3ejnJ8pX0BeX2r8zFteasG3vJXu7etRyf50Vbrw9q0x9A9/XAT8O0kfwM8EDi0qh7VlrV8mnUvLUjuwUvzrLon8X0Q+I2dmO2rVbWuqjbR3cpyIqAvogv1CadX1XhVfZfui8CP0t3v+qVJvkH3mN8D6O4dDnD+5HBvHg+cW91DSbYCH6Z7Fvuuegbw+lbDuXS3QH1Qm3ZWVU08ZzvAnyW5kO7Wn4dy7yM4t+engA8BVNW3gKuAiYA/u6puq6qNdEcpHky3Xh6S5G+SHAfcPsUypQXPPXhpON4BfA14/0DbVtqX7iQjwOKBaZsGhscHxsf54X/Hk+89XXSh+etV9UMPVElyLN0jV+dDgP+vqr49qYafnFTDi4GVwOOqaku6p+8t3Y33HVxv24Cxqro1yU8AzwReBbyA7v7gUq+4By8NQdtjPZ3ugrUJV9IdEgd4LrBoFxb9i0lG2nn5h9A9CORzwK+ke+QvSR6WZNk0yzkf+NkkByYZBV4E/MdO1LEB2Hdg/HPAr7cnZZHksduZbz/gxhbuT6Hb455qeYP+k+6LAe3Q/IPoPveU2qH/kar6OPD7dKcIpN4x4KXh+Stg8Gr6f6AL1W8CT2TX9q6vpgvnzwCvaoem30t3ePpr7cK0/8s0R++qe3zl6+keI/pN4IKq2pnHV54DPGLiIjvgj+m+sFyY5JI2PpUPA6uTXER37cC3Wj030107cPHki/uAvwNG2jwfA17WTmVsz6HAue10wYeAk3fic0kLhk+TkySph9yDlySphwx4SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAlySphwx4SZJ66P8HvudQiwOC+6QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "fig = plt.figure(figsize=(8,6))\n",
    "plt.plot(np.arange(len(lr.loss)), lr.loss)\n",
    "plt.title(\"Development of cost over training\")\n",
    "plt.xlabel(\"Number of iterations\")\n",
    "plt.ylabel(\"Cost\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}