HW2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 2\n",
    "## Hoeffding Inequality\n",
    "Run a computer simulation for fipping 1,000 virtual fair coins. Flip each coin independently 10 times. Focus on 3 coins as follows: $c_1$ is the first coin flipped, $c_{rand}$ is a coin chosen randomly from the 1,000, and $c_{min}$ is the coin which had the minimum frequency of heads (pick the earlier one in case of a tie). Let $v_1$, $v_{rand}$, and $v_{min}$ be the fraction of heads obtained for the 3 respective coins out of the 10 tosses.  \n",
    "\n",
    "Run the experiment 100,000 times in order to get a full distribution of $v_1$, $v_{rand}$, and $v_min$ (note that crand and cmin will change from run to run).\n",
    "### Question 1\n",
    "The average value of $v_{min}$ is closest to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average value of v_min is closest to:  0.01  with v_min equal to  0.037624\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "no_coins = 1000 # number of coins to flip\n",
    "no_flips = 10 # number of times to flip one coin\n",
    "no_trials = 100000 # number of times to repeat the experiment\n",
    "v_1, v_rand, v_min = [], [], [] # where to save the values for each trial run\n",
    "\n",
    "for t in range(no_trials):\n",
    "    sample = np.random.randint(2, size=(no_coins, no_flips))\n",
    "    # Let 1’s be heads and 0’s tails\n",
    "    heads = sample.sum(axis=1)/no_flips # sum heads over rows (per each coin)\n",
    "    \n",
    "    v_1.append(heads[1])\n",
    "    # choose index of the random coin\n",
    "    c_rand = random.randint(1,10)\n",
    "    v_rand.append(heads[c_rand])\n",
    "    v_min.append(heads[np.argmin(heads)])\n",
    "\n",
    "# average value of v_min\n",
    "v_min_avg = np.average(v_min)\n",
    "print('The average value of v_min is closest to: ', \n",
    "      min([0, 0.01, 0.1, 0.5, 0.67], key=lambda x:abs(x-v_min_avg)),\n",
    "     ' with v_min equal to ', v_min_avg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer (b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2  \n",
    "Which coin(s) has a distribution of $v$ that satisfes the (single-bin) Hoeffding Inequality?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: Single bin relates to only one trial where we flip coins to get H or T. The probability of heads or tails on one trial is 0.5. Hence, $\\mu=0.5$ which means we need a coin which will give us probabilities close to 0.5 which will in turn make the epsilon small. Using the previous code and results we see that $v_{rand}$ and $v_1$ satisfy this logic as they are both close to 0.5 as expected. The coins $c_{min}$ will give lower probabilities since we deliberately made them small by definition.  \n",
    "\n",
    "Answer is (d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average of v_rand is  0.49915\n",
      "Average of v_1 is  0.499095\n",
      "Average of v_min is  0.037624\n"
     ]
    }
   ],
   "source": [
    "print('Average of v_rand is ', np.average(v_rand))\n",
    "print('Average of v_1 is ', np.average(v_1))\n",
    "print('Average of v_min is ', np.average(v_min))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error and Noise\n",
    "Consider the bin model for a hypothesis $h$ that makes an error with probability $\\mu$ in\n",
    "approximating a deterministic target function $f$ (both $h$ and $f$ are binary functions).\n",
    "If we use the same $h$ to approximate a noisy version of $f$ given by:\n",
    "$$P(y|\\mathbf x) = \\cases{\n",
    "\\lambda  & \\text{if   } y=f(\\mathbf x)\\cr\n",
    "(1-\\lambda) & \\text{if   } y \\neq f(\\mathbf x)\n",
    "}$$\n",
    "\n",
    "### Question 3\n",
    "What is the probability of error that $h$ makes in approximating $y$? Hint: Two wrongs can make a right!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: If $h$ makes an error approximating $f$ then that error can arise from two cases namely\n",
    "1. $P[h(x)=f(x)] \\text{   and   } y\\neq f(x)$\n",
    "2. $P[h(x)\\neq f(x)] \\text{   and   } y=f(x)$  \n",
    "\n",
    "Hence, combining the two cases we can write\n",
    "$$\n",
    "P[h(x)\\neq y] = (1-\\mu)(1-\\lambda) + \\mu\\lambda\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "At what value of $\\lambda$ will the performance of $h$ be independent of $\\mu$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: We have from previous exercise that \n",
    "$$\\begin{align}\n",
    "P[h(x)\\neq y] &= (1-\\mu)(1-\\lambda) + \\mu\\lambda \\\\\n",
    "&=1-\\lambda-\\mu+\\mu\\lambda+\\mu\\lambda \\\\\n",
    "&=1-\\lambda-\\mu+2\\mu\\lambda \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Hence, if $\\lambda=\\frac {1}{2}$, $P[h(x)\\neq y]$ becomes independent of $\\mu$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression\n",
    "In these problems, we will explore how Linear Regression for classification works. As with the Perceptron Learning Algorithm in Homework # 1, you will create your own target function $f$ and data set $D$. Take $d = 2$ so you can visualize the problem, and assume $X = [-1; 1] x [-1; 1]$ with uniform probability of picking each $\\mathbf x \\in \\mathcal X$. In each run, choose a random line in the plane as your target function $f$ (do this by taking two random, uniformly distributed points in $[-1; 1] x [-1; 1]$ and taking the line passing through them), where one side of the line maps to +1 and the other maps to -1. Choose the inputs $x_n$ of the data set as random points (uniformly in X), and evaluate the target function on each $x_n$ to get the corresponding output $y_n$.  \n",
    "### Question 5  \n",
    "\n",
    "Take $N = 100$. Use Linear Regression to find $g$ and evaluate $E_{in}$, the fraction of in-sample points which got classified incorrectly. Repeat the experiment 1000 times and take the average (keep the $f$'s and $g$'s as they will be used again in Problem 6). Which of the following values is closest to the average $E_{in}$? (Closest is the option that makes the expression |your answer - given option| closest to 0. Use this definition of closest here and throughout.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: The process is as follows:\n",
    "1. First we need to generate the input data set first and assign the target value for each $i=1,...,n$ in the input data set. As specified in the question we need to save the $f$'s in order to reuse it later. So in the code below we use the same **l** value denoting the line function i.e. our $f$. \n",
    "2. Write the linear regression algorithm by caching the $g$ values for each trial\n",
    "3. Compute the $E_{in}$\n",
    "4. Repeat for the number of trials needed and average the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 2. Linear regression\n",
    "def linregr(N, Nt, dim, no_trials):\n",
    "    \n",
    "    Ein_t = [] # save the in-sample error for each trial\n",
    "    Eout_t = [] # save out-out-of sample error for each trial\n",
    "    w = [] # store the weights\n",
    "    \n",
    "    for t in range(no_trials):\n",
    "        \n",
    "        # Generate random values for the line segment\n",
    "        xa,ya,xb,yb = [random.uniform(-1, 1) for i in range(4)]\n",
    "        l = np.array([xb*ya-xa*yb, yb-ya, xa-xb])\n",
    "        l = l.reshape((l.shape[0],1))\n",
    "        \n",
    "        # Generate train data\n",
    "        X = np.column_stack((np.ones(N), \n",
    "                             np.random.uniform(-1.,1., size=(N, dim))))\n",
    "        Y = np.where(np.dot(X,l)>=0,1,-1)\n",
    "        \n",
    "        # Compute the weights\n",
    "        w = np.dot(np.dot(np.linalg.inv(np.dot(X.T,X)),X.T),Y)\n",
    "        \n",
    "        # Estimate hypothesis\n",
    "        g = np.where(np.dot(X,w)>=0, 1, -1)\n",
    "        \n",
    "        # Compute in-sample error\n",
    "        Error = g - Y\n",
    "        Ein = np.count_nonzero(Error)/N\n",
    "        Ein_t.append(Ein)\n",
    "        \n",
    "        # Compute out-of-sample error\n",
    "        Xt = np.column_stack((np.ones(Nt),\n",
    "                              np.random.uniform(-1.,1., size=(Nt, dim))))\n",
    "        Yt = np.where(np.dot(Xt,l)>=0,1,-1)\n",
    "        \n",
    "        gt = np.where(np.dot(Xt,w)>=0, 1, -1)\n",
    "        Error_t = gt - Yt\n",
    "        Eout = np.count_nonzero(Error_t)/Nt\n",
    "        Eout_t.append(Eout)\n",
    "        \n",
    "        lr_out = {'w':w,\n",
    "                  'Ein_t': Ein_t,\n",
    "                  'Eout_t': Eout_t}\n",
    "    \n",
    "    return lr_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value closes to the E_in is:  0.01  with E_in equal to  0.03885\n"
     ]
    }
   ],
   "source": [
    "# INPUTS\n",
    "N = 100 # number of training examples\n",
    "Nt = 1000 # number of test examples\n",
    "dim = 2 # dimension of X\n",
    "no_trials = 1000 # number of times to run the experiment\n",
    "\n",
    "lreg_out = linregr(N, Nt, dim, no_trials)\n",
    "\n",
    "print('The value closes to the E_in is: ', \n",
    "      min([0, 0.001, 0.01, 0.1, 0.5], key=lambda x:abs(x-np.average(lreg_out['Ein_t']))),\n",
    "     ' with E_in equal to ', np.average(lreg_out['Ein_t']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6  \n",
    "Now, generate 1000 fresh points and use them to estimate the out-of-sample error $E_{out}$ of the $g$'s that you got in Problem 5 (number of misclassified out-of-sample points / total number of out-of-sample points). Again, run the experiment 1000 times and take the average. Which value is closest to the average $E_{out}$?  \n",
    "\n",
    "Answer: Using the code from Question 5 we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value closes to the E_out is:  0.01  with E_out equal to  0.049463\n"
     ]
    }
   ],
   "source": [
    "print('The value closes to the E_out is: ', \n",
    "      min([0, 0.001, 0.01, 0.1, 0.5], key=lambda x:abs(x-np.average(lreg_out['Eout_t']))),\n",
    "     ' with E_out equal to ', np.average(lreg_out['Eout_t']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 7  \n",
    "Now, take $N = 10$. After finding the weights using Linear Regression, use them as a vector of initial weights for the Perceptron Learning Algorithm. Run PLA until it converges to a final vector of weights that completely separates all the in-sample points. Among the choices below, what is the closest value to the average number of iterations (over 1000 runs) that PLA takes to converge? (When implementing PLA, have the algorithm choose a point randomly from\n",
    "the set of misclassified points at each iteration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perceptron(N, no_trials, learn_rate):\n",
    "    \n",
    "    iter_cnt = []\n",
    "    for t in range(no_trials):\n",
    "    \n",
    "        # Generate random values for the line segment\n",
    "        xa,ya,xb,yb = [random.uniform(-1, 1) for i in range(4)]\n",
    "        l = np.array([xb*ya-xa*yb, yb-ya, xa-xb])\n",
    "        l = l.reshape((l.shape[0],1))\n",
    "        \n",
    "        # Generate train data\n",
    "        X = np.column_stack((np.ones(N), \n",
    "                             np.random.uniform(-1.,1., size=(N, dim))))\n",
    "        Y = np.where(np.dot(X,l)>=0,1,-1)\n",
    "        \n",
    "        # Compute the weights using linear regression\n",
    "        w = np.dot(np.dot(np.linalg.inv(np.dot(X.T,X)),X.T),Y)\n",
    "        \n",
    "        cnt_iter = 0 # initialize counter for the convergence\n",
    "        while True:\n",
    "            g = np.where(np.dot(X,w) >= 0, 1, -1)\n",
    "            err = g - Y\n",
    "            misclass = np.nonzero(err)[0]\n",
    "            if len(misclass) == 0:\n",
    "                break\n",
    "            else:\n",
    "                point = random.choice(misclass) # choose a random point from the vector of misclassifications\n",
    "                update = np.reshape(learn_rate*Y[point]*X[point],(len(w),1))\n",
    "                w = w + update # update weights\n",
    "                cnt_iter += 1\n",
    "        iter_cnt.append(cnt_iter)\n",
    "        \n",
    "    return iter_cnt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The closest value to the average number of iterations (over 1000 runs) that the PLA takes to converge is :  1  with average value equal to  6.094\n"
     ]
    }
   ],
   "source": [
    "# Inputs\n",
    "N = 10\n",
    "no_trials = 1000\n",
    "learn_rate = 0.01\n",
    "\n",
    "iter_cnt = perceptron(N, no_trials, learn_rate)\n",
    "\n",
    "print('The closest value to the average number of iterations (over 1000 runs) that the PLA takes to converge is : ', \n",
    "      min([1, 15, 300, 5000, 10000], key=lambda x:abs(x-np.average(iter_cnt))),\n",
    "     ' with average value equal to ', np.average(iter_cnt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (a) as expected with such small N value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nonlinear Transformation  \n",
    "In these problems, we again apply Linear Regression for classification. Consider the target function:  \n",
    "\n",
    "$$f(x_1, x_2) = sign(x^2_1 + x^2_2 - 0.6)$$  \n",
    "\n",
    "Generate a training set of $N = 1000$ points on $X = [-1; 1] x [-1; 1]$ with a uniform probability of picking each $\\mathbf x \\in \\mathcal X$. Generate simulated noise by flipping the sign of the output in a randomly selected 10% subset of the generated training set.  \n",
    "### Question 8  \n",
    "Carry out Linear Regression without transformation, i.e., with feature vector:\n",
    "$$(1, x_1, x_2)$$\n",
    "to find the weight $w$. What is the closest value to the classification in-sample error $E_{in}$? (Run the experiment 1000 times and take the average $E_{in}$ to reduce variation in your results.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# NonLinear regression\n",
    "def nonlinregr(N, Nt, dim, no_trials):\n",
    "    \n",
    "    nEin_t = [] # save the in-sample error for each trial\n",
    "    nEout_t = [] # save out-out-of sample error for each trial\n",
    "    nw = [] # store the weights\n",
    "    pnoise = 10 # percent of noise\n",
    "\n",
    "    \n",
    "    for t in range(no_trials):\n",
    "        # Generate train data        \n",
    "        X = np.column_stack((np.ones(N), \n",
    "                             np.random.uniform(-1.,1., size=(N, dim))))\n",
    "        Y = np.where((np.square(X[:,1]) + np.square(X[:,2]) - 0.6) >= 0,1,-1)\n",
    "        points = random.sample(range(1, N), int(N*pnoise/100)) # outputs to be altered\n",
    "        Y[points] = Y[points]*(-1) # add noise to the output\n",
    "        \n",
    "        # Compute the weights\n",
    "        nw = np.dot(np.dot(np.linalg.inv(np.dot(X.T,X)),X.T),Y)\n",
    "        \n",
    "        # Estimate hypothesis\n",
    "        g = np.where(np.dot(X,nw)>=0, 1, -1)\n",
    "        \n",
    "        # Compute in-sample error\n",
    "        Error = g - Y\n",
    "        Ein = np.count_nonzero(Error)/N\n",
    "        nEin_t.append(Ein)\n",
    "        \n",
    "        # Compute out-of-sample error\n",
    "        Xt = np.column_stack((np.ones(Nt),\n",
    "                              np.random.uniform(-1.,1., size=(Nt, dim))))\n",
    "        Yt = np.where((np.square(Xt[:,1]) + np.square(Xt[:,2]) - 0.6) >= 0,1,-1)\n",
    "        Yt[points] = Yt[points]*(-1)\n",
    "            \n",
    "        gt = np.where(np.dot(Xt,nw)>=0, 1, -1)\n",
    "        Error_t = gt - Yt\n",
    "        Eout = np.count_nonzero(Error_t)/Nt\n",
    "        nEout_t.append(Eout)\n",
    "        \n",
    "        nlr_out = {'nw':nw,\n",
    "                  'nEin_t': nEin_t,\n",
    "                  'nEout_t': nEout_t}\n",
    "    \n",
    "    return nlr_out   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value closes to the E_in is:  0.5  with E_in equal to  0.505502\n"
     ]
    }
   ],
   "source": [
    "# INPUTS\n",
    "N = 1000 # number of training examples\n",
    "Nt = 1000 # number of test examples\n",
    "dim = 2 # dimension of X\n",
    "no_trials = 1000 # number of times to run the experiment\n",
    "\n",
    "nlr_out = nonlinregr(N, Nt, dim, no_trials)\n",
    "\n",
    "print('The value closes to the E_in is: ', \n",
    "      min([0, 0.1, 0.3, 0.5, 0.8], key=lambda x:abs(x-np.average(nlr_out['nEin_t']))),\n",
    "     ' with E_in equal to ', np.average(nlr_out['nEin_t']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9  \n",
    "Now, transform the $N = 1000$ training data into the following nonlinear feature vector:\n",
    "$$(1, x_1, x_2, x_1x_2, x^2_1, x^2_2)$$\n",
    "Find the vector $\\widetilde{w}$ that corresponds to the solution of Linear Regression. Which\n",
    "of the following hypotheses is closest to the one you find? Closest here means agrees the most with your hypothesis (has the highest probability of agreeing on a randomly selected point). Average over a few runs to make sure your answer is stable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# NonLinear regression\n",
    "def nonlinregr2(N, Nt, dim, no_trials):\n",
    "    \n",
    "    nEin_t = [] # save the in-sample error for each trial\n",
    "    nEout_t = [] # save out-out-of sample error for each trial\n",
    "    nwt = [] # store the weights\n",
    "    pnoise = 10 # percent of noise\n",
    "    \n",
    "    for t in range(no_trials):\n",
    "        # Generate train data        \n",
    "        Xr = np.column_stack((np.ones(N), \n",
    "                             np.random.uniform(-1.,1., size=(N, dim))))\n",
    "        X = np.column_stack((Xr, Xr[:,1]*Xr[:,2],\n",
    "                            np.square(Xr[:,1]), np.square(Xr[:,2])))\n",
    "        Y = np.where((np.square(X[:,1]) + np.square(X[:,2]) - 0.6) >= 0,1,-1)\n",
    "        points = random.sample(range(1, N), int(N*pnoise/100)) # outputs to be altered\n",
    "        Y[points] = Y[points]*(-1) # add noise to the output\n",
    "        \n",
    "        # Compute the weights\n",
    "        nw = np.dot(np.dot(np.linalg.inv(np.dot(X.T,X)),X.T),Y)\n",
    "        nwt.append(nw)\n",
    "        \n",
    "        # Estimate hypothesis\n",
    "        g = np.where(np.dot(X,nw)>=0, 1, -1)\n",
    "        \n",
    "        # Compute in-sample error\n",
    "        Error = g - Y\n",
    "        Ein = np.count_nonzero(Error)/N\n",
    "        nEin_t.append(Ein)\n",
    "        \n",
    "        # Compute out-of-sample error\n",
    "        Xtr = np.column_stack((np.ones(Nt),\n",
    "                              np.random.uniform(-1.,1., size=(Nt, dim))))\n",
    "        Xt = np.column_stack((Xtr, Xtr[:,1]*Xtr[:,2],\n",
    "                            np.square(Xtr[:,1]), np.square(Xtr[:,2])))\n",
    "        Yt = np.where((np.square(Xt[:,1]) + np.square(Xt[:,2]) - 0.6) >= 0,1,-1)\n",
    "        Yt[points] = Yt[points]*(-1)\n",
    "        \n",
    "        gt = np.where(np.dot(Xt,nw)>=0, 1, -1)\n",
    "        Error_t = gt - Yt\n",
    "        Eout = np.count_nonzero(Error_t)/Nt\n",
    "        nEout_t.append(Eout)\n",
    "        \n",
    "        nlr_out2 = {'nwt':nwt,\n",
    "                  'nEin_t': nEin_t,\n",
    "                  'nEout_t': nEout_t}\n",
    "    \n",
    "    return nlr_out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -9.94e-01,   1.63e-03,   6.32e-04,   4.46e-03,   1.56e+00,\n",
       "         1.56e+00])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# INPUTS\n",
    "N = 1000 # number of training examples\n",
    "Nt = 1000 # number of test examples\n",
    "dim = 2 # dimension of X\n",
    "no_trials = 1000 # number of times to run the experiment\n",
    "\n",
    "nlr_out2 = nonlinregr2(N, Nt, dim, no_trials)\n",
    "np.set_printoptions(precision=2)\n",
    "proba = np.average(np.array(nlr_out2['nwt']), axis = 0)\n",
    "proba"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10  \n",
    "What is the closest value to the classification out-of-sample error $E_{out}$ of your hypothesis from Problem 9? (Estimate it by generating a new set of 1000 points and adding noise, as before. Average over 1000 runs to reduce the variation in your results.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value closes to the E_out is:  0.1  with E_out equal to  0.126368\n"
     ]
    }
   ],
   "source": [
    "print('The value closes to the E_out is: ', \n",
    "      min([0, 0.1, 0.3, 0.5, 0.8], key=lambda x:abs(x-np.average(nlr_out2['nEout_t']))),\n",
    "     ' with E_out equal to ', np.average(nlr_out2['nEout_t']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: (b)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}