notebooks/L17_more-reg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More regularization\n",
    "\n",
    "CPSC 340: Machine Learning and Data Mining\n",
    "\n",
    "The University of British Columbia\n",
    "\n",
    "2018 Winter Term 1\n",
    "\n",
    "Mike Gelbart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "from sklearn.linear_model import LinearRegression, HuberRegressor, Ridge, Lasso\n",
    "from scipy.optimize import minimize\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "matplotlib.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topics:\n",
    "\n",
    "- Why L1 leads to sparsity\n",
    "- Regularization strength intuition, regularization paths, L1 vs. L2\n",
    "- Why not use Huber regularization?\n",
    "- Uniqueness of solution: L1 and L2\n",
    "- Intercept and the mean value\n",
    "- Scaling features\n",
    "- Scaling targets\n",
    "- Collinearity and conditioning\n",
    "- Overfitting and $d>n$\n",
    "- Loss function Venn diagram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why L1 regularization leads to sparsity\n",
    "\n",
    "Consider this 1D case of a feature whose \"true\" weight is close to zero. Starting with no regularization:\n",
    "\n",
    "$$f(w)=\\sum_{i=1}^n (wx_i-y_i)^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minimizer: w=-0.25\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAElCAYAAADOTWQ3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VFX6x/HPE5KQAoSE3nsXAQlNUVBYFZayioIFFRvFytpx1Z+uXVFZl8WuqLgIiA1FXKMiIh2kl9B7SwIJIT05vz/uRMc4IZnMZO6U5/16zSvkzr1zv3NJ5sm5595zxBiDUkopVVFhdgdQSikV2LSQKKWU8ogWEqWUUh7RQqKUUsojWkiUUkp5RAuJUkopj2ghUcpHRKS5iBgRmR7I+3AjyxhHljF2Z1GVSwuJqjCnDy0jInNKWaf4w2Sir/Opyicie0Rkj905lL3C7Q6ggsYIEUk0xqyyO0iIOwh0ANLtDgJ8BiwDDtsdRFUubZEob9jt+PqsrSkUxph8Y8xWY4ztH97GmHRHFn8oaqoSaSFR3rAemAsMFJEB5d1IRC4QkQUickJEskVkg4jcJyLlaik79weISGcR+crxWqbEeheKyHwRSRWRHBHZLCIPudqPiNQVkfdEJEVETovIL47tH3fsq7/TuqX2AbjTPyAiFzn2mezY5ykRWSIio9x9z676SJyyl/ZY6LRuWxF5UUTWOl43R0Q2icgjIhJRMgfQDGhW4vXGlOP43CwiKx3vN0NEFonIcBfr/XbcReRaEVnnyHRARJ4SkSplHV9V+fTUlvKWR4DLsFolPctaWURGAv8FsoBZwAlgMPAi0FdELjPlHwiuNfALsBp4G2jstJ87gFeBFOALx376OuW83Gnd6sAioB3wA7Dc8doLgB/LmaUiHgBaOvZ3EEgAhgEfi0gDY8wUF9uU+p5dWFjK8h5YxzzLadnlwI1Y7z8JqAr0A54EEoG/OdY7CTwBFPd9OWdce4YsiMi/gTuAvcAbQCRwJfC5iNxrjHnZxWZ3An/B+j/8Eev4/APrM+yhM+1P+YAxRh/6qNADaA4Y4HPH9285vh/htM4Yx7KJTstqYH0QZQIdnJaHA9861r/ejf0b4GEXz3cC8rHO08c5LRdgqmO7K5yWP+1Y9mKJ1xnttJ/+Lt7bGBf7/tNzTnmnl1i3hYvtY4F1WH0dMW68Z5f7KGW9Y0Aa0NZpeSMgssS64vR/27fEc3uAPaXsw9Ux6OdYtg6o5rS8PlYRzQdaOi1/3LF+GtDaaXkCkAqcKplXH75/6Kkt5U2PAzlAWacc/gbEAW8aY7YULzTGFAAPOr69wY39HsZqyZQ0Dqs43WGcztMb65PoYawPqKuc1r8W66/zZ0q8zkfAZjfyuMUYs9vFstPA+1hF11ULr7T3XCYRqQF8BcQDVxpjkp32e9AYk1ciiwFec3w7sCL7dFL8//qoMSbTaR9HsN5PONb/Q0mvGmN2OK2fBnwJVMNqQSob6akt5TXGmIMiMhW4D+sD491SVu3q+LrQxWusFZF0p3XKY50xJt/F8l5YxWKYiAxx8Xw20B5AROKwzvevNsacKJHJiMgyoKMbmcrN8cH+ADAc6xRXTIlVGrjYrLT3XNa+qgAzsVprE4wx35d4Pgy4Gas10QmrkEkZWdxR6v+90zJX//e/ulh20PG1pmeRlKe0kChvexa4FXhcRD4qZZ0ajq9HS3n+CNDKjX0eK2V5AtaH4KNn2DbW8bW64+txN/fhERGJBH7C+vBcDUzHOo1T6Fg2HKufwlt5XsLqF3nVGPO6i+f/DdyG1X/xKdb/RR7Wh/XdpWRxRw0gxxiT4eK5I07rlOTqyq8Cx1ftcLeZFhLlVcaYNBGZjNU5ezvWh2JJxR8i9Up5mXpO65Rrt6Usz8D6QI41xuSW8RqnHF/rlPJ8XRfLihxfXX2QufowdGU4VsF40xgzzvkJEXnQ8bwrbs9IJyLjsIrBt8A9Lp6vB0zA6r/oY4zJdnqul2NbT2UArUSkhotiUs9pHRVAtI9EVYZXsFobk3D9gVp8Vc8FJZ8QkbOx/vo945U/5bQC60O+R1krOvpQ9gIdROQPp0pERIDeLjY76fjayMVz3cqZsbjlNc/Fc+eV8zXKJCIXYV1gsBkYZYwpdLFaC6wWXJJzESkjSyHutQhK/b/H6oh3XkcFCC0kyuscHcVPAbWBu1ys8gXWX51jRaR18ULH+fvnHd9+4IUo07A+6P4jIn86ty8i9USkg9OimVj9E/8oseo1uO4fWYPVMhglIr+d8hGRnrjuMHZln+PrHz6oReRyYGg5X+OMRKQt8AnW6aGhpvQbBIuz9HEUT+ftJ5WyTRpQ2/n9l6H4//UJEfmtL0hE6mL1ExVgXRauAoie2lKV5Q2s0yd/6uswxqSLyHhgBrBaRD7G+ut+MHAW1hVFHhcSY8wGEbkT6y/xZBGZj3W5ajzQBut+kkeB4ivHngVGAPeJSDesFk1rrHsWvgUu4ffTWRhjDojIXOAKYKWIfIfVOhkOfM3v91ycyTysD/AHRaQTsBWrk/tSrCFGLqvo+3cyBes9fwNc71Qjiu0xxkw3xhwSkeJ9rhSRH4GG/P7+R7h47R+x7i/5QkR+wbp892tjzAZXQYwxC0XkNaxTaBsd+4sERmKdPnzAGLPTs7erfE0LiaoUxph8EXkM+LCU52eKyGGsv3RHAVHATqyby152XHLqjRyvicharKJ2AdaHZBpWQXkSp79+jTEZInI+VqtoKNAH6zTLIKwb9S7h976UYmOwbnYcgdVJvd6xj/qUo5AYY045TjtNxipsFzr2ORjrCilvFJLiv/wHOR4l/YTVyQ/W1Xb7HPu9E+v/5B9YBc9VIXkKqAX8FeuGwTCsTnOXhcThdqyrsMZjFZQix/cTjDGflvM9KT8iXvp9VSqoicjPwLlYNzZmlrW+UqFE+0iUclJKX8pVWK2FH7SIKPVn2iJRyomIbMDqr1mHdf/E2cAArOFczjfG6BVFSpWghUQpJyJyL9ZVWi2xht9Ixbrj+kljzCYboynlt7SQKKWU8khIXLVVu3Zt07x5c7tjKKVUwFi9enWKMaa0kR7+ICQKSfPmzVm1SmeAVUqp8hKRveVdV6/aUkop5REtJEoppTyihUQppZRHtJAopZTyiBYSpZRSHtFCopRSyiNaSJRSSnlEC4lSSimPaCFRSinlES0kKmQkJSWRlJRkdwylgk5IDJGiFMBTTz0FwMCBA21OolRw0RaJUkopj2ghUUop5REtJEoppTyihUQppZRHtLNdhYw33njD7ghKBSUtJCpktGvXzu4ISvnMwm3H2HEsk+v7NCcyvHJPPumpLRUy5s2bx7x58+yOoVSlKywyPP31Fv67fB9hUvn70xaJChkvvfQSAEOHDrU5iVKV6/NfD7L9WCb/ueYcwqtUfntBWyRKKRVEcgsKefm7ZM5qVINBZ9X3yT61kCilVBCZuXwfB09mc/8l7QnzxXkttJAopVTQOJ1bwNQfd9C7ZQIXtKnts/1qIVFKqSDx7uLdpGTm8cCl7RHxTWsEtLNdhZAPP/zQ7ghKVZoTp/N4c9Eu/tKxHuc0jffpvrWQqJDRpEkTuyMoVWle+2knmXkF3Hex7++X0lNbKmTMmjWLWbNm2R1DKa87kp7D+0v2cFm3RrSrX93n+7e1kIjIeSLyPxE5JiIZIrJGRG4qsU6UiLwoIodFJFtElorIBXZlVoHrtdde47XXXrM7hlJeNyUpmSJj+PvAtrbs37ZCIiJnA0lABHArMAJYCbwjIhOcVn3H8fxjwBDgMPCtiHT1bWKllPI/24+eYvaq/Yzu3YwmCTG2ZLCzj+QqoAow1BiT6Vj2nYh0Aa4HXnP8+xrgJmPMewAi8hOwCfgnMMz3sZVSyn88v2ArsZHh3HlRG9sy2HlqKxLIB7JLLD/J77mGOdb57cS2MaYA+Bi4RESq+iCnUkr5pRW700jacozx/VuREBtpWw47C8l0x9dXRaShiNQUkVuBAcArjuc6AbuNMVkltt2EVYha+ySpUkr5GWMMz36zhXo1qnLTeS1szWLbqS1jzEYR6Q98BtzmWJwPjDfGfOz4PgE44WLzNKfnXRKRscBYgKZNm3ojsgpwn3zyid0RlPKaBRuP8Ou+kzx3eWeiI6vYmsW2QiIibYC5WK2L8VinuIYDr4tIjjHmI0AA42rzsl7fGPMm8CZAYmKiq9dQIaZ2bd8NGaFUZcovLOKFb7fRpm41ruje2O44tna2P4PVAhlijMl3LPteRGoB/xKRmVgtD1fNieLbNtNcPKeUS9OnTwdgzJgxtuZQylMfr9zP7pTTvH19ok+GiS+LnQk6A+ucikixFUAtoC5Wa6WFiJS8pq0jkAfsqPSUKmhMnz79t2KiVKA6nVvAv5KS6dk8gQEd6todB7C3kBwBuopIyUsNegE5WK2NL7HuM7my+EkRCQdGAf8zxuT6KKtSSvmFN37aSUpmHg8N9u3AjGdi56mtqcAcYJ6ITMPqIxkGXA28YozJA9aKyCxgiohEALuBCUAL4Fp7YiullD0Op2fz5s+7+OvZDXw+MOOZ2HnV1iciMhh4EHgbiAJ2ArcDbziteiPwNPAUUBNYB1xqjFnj28RKKWWvyd8mU1QED13a3u4of2Dr6L/GmG+Ab8pYJxu4x/FQSqmQtPFgOp/+eoBbz29p21AopdFh5FXImD9/vt0RlKoQYwzPzN9CzegIbr/Q/+7Dtv+6MaV8JCYmhpgY//pLTqny+GHrMZbsTOXuAW2Ii46wO86faCFRIWPatGlMmzbN7hhKuSW/sIhn5m+hZe1Yru3dzO44LmkhUSFj9uzZzJ492+4YSrnl4xX72Hn8NA8Nak+EH9x86Ip/plJKKUVGTj5TkrbTq0UCf+lYz+44pdJCopRSfmrqDztIy8rjkb929JubD13RQqKUUn5od8pp3vtlN1d2b0znxnF2xzkjLSRKKeWHnv56C1XDq3DfJe3sjlImvY9EhYyFCxfaHUGpclm8PYWkLUd58NL21K0eZXecMmmLRCml/EhBYRH//GoTTRKiufG85nbHKRctJCpkTJ48mcmTJ9sdQ6kzmrliH8lHM/nH4A5ERdg782F5aSFRIeOrr77iq6++sjuGUqVKz8rn5e+S6d0ygUs61bc7TrlpIVFKKT8x5ftk0rPzeWxIJ7++3LckLSRKKeUHth89xYdL9zKqR1M6Nqxhdxy3aCFRSimbGWN4fN4mYquGc38AXO5bkl7+q0JGdHS03RGUcmnBxiP8siOVfw7vREJsydnH/Z8WEhUyvvnmjHOoKWWL7LxCnvp6C+3rV+eank3tjlMhWkiUUspGry3cwcGT2cwa25twPx3dtyyBmVqpCnjyySd58skn7Y6h1G/2pWbx+qJdDOvSkF4ta9kdp8K0kKiQ8f333/P999/bHUOp3zz59WbCw4SHB3ewO4pHtJAopZQNfko+znebj3LnRW2oH+f/42mdiRYSpZTysdyCQp74chMtasdyU9/mdsfxmHa2K6WUj721aBe7Uk7zwU09qRoeGONpnYkWEhUyatUK3M5MFTz2p2Xx7x928NfODbigbR2743iFFhIVMubOnWt3BBXijDH835ebCA8THh3S0e44XqN9JEop5SPfbT7KD1uPMXFg24DvYHemhUSFjEmTJjFp0iS7Y6gQlZVXwBPzNtOuXnXGBMiEVeWlp7ZUyFi6dKndEVQIm/qDdQf7nPF9iAjQO9hLE1zvRiml/NCOY6d46+ddXNG9MT2aJ9gdx+u0kCilVCUyxvDI5xuJjqjCQ4Pa2x2nUmghUUqpSjRn9QGW7Upj0uAO1K5W1e44lUL7SFTIaNy4sd0RVIhJzczlmflb6NE8nlGJTeyOU2m0kKiQMWPGDLsjqBDz9NdbOJ1bwDOXdSYsLHDmYHeXntpSSqlKsHh7Cp/+epDx/VrRpl51u+NUKi0kKmRMnDiRiRMn2h1DhYCc/EIe+XwDzWvFcPuFre2OU+n01JYKGWvXrrU7ggoRU3/YwZ7ULD66pRdREYE/KGNZtEWilFJelHz0FG8s2snl3RpxXuvadsfxCS0kSinlJYVFhgfnrie2ajj/+Gtgz3roDi0kSinlJR8s3cOv+07yf0M7UitI7xlxRftIVMho27at3RFUENuflsULC7bRv10d/ta1kd1xfEoLiQoZb775pt0RVJAyxvDwZxsIE3j6ss6IBO89I67oqS2llPLQJ6sP8PP2FB4a1J5GNaPtjuNzWkhUyBg7dixjx461O4YKMsdO5fDU19YwKNf2amZ3HFvoqS0VMpKTk+2OoILQ419uIju/kOdGnB3Uw6CcibZIlFKqgr7ZcJj5G45w94A2tKpTze44ttFCopRSFZCamcsjn2/krEY1GHtBS7vj2EpPbSmlVAU89uUmMnLy+ejKXkE3da67bH/3IjJYRBaJSKaIZIjIKhG5yOn5eBF5W0RSROS0iCSJSGc7M6vA1LVrV7p27Wp3DBUE5m84zNfrD3P3gDa0r1/D7ji2s7VFIiLjgKmOx5NYha0rEON4XoAvgRbAncAJYBLwo4h0NcYcsCO3CkxTpkyxO4IKAqmZuTz6+UY6N4pjfL9WdsfxC7YVEhFpDkwB7jfGOP+Gf+v072FAX+AiY8yPju2WAruBB4C7fBJWKaUcHvtiE6dyCph8ZRfCQ/yUVjE7j8JNQBHw+hnWGQYcKi4iAMaYdGAeMLxy46lgM3r0aEaPHm13DBXAvl5/mK83HObugW1oVz+4J6tyh52FpC+wFbhKRHaKSIGI7BCR253W6QRsdLHtJqCpiITu9XbKbQcOHODAAT0bqirm+KlcHv1iI10axzEuxK/SKsnOQtIQaAO8CDwHXAx8B0wVkbsd6yRg9YuUlOb4Gl/ai4vIWEfH/arjx497L7VSKuQYY5j06Xoyc/WUlit2Ho0woDowzhjzljHmB2PMBGABMMnR0S6AcbFtmbePGmPeNMYkGmMS69Sp49XgSqnQMnvVfpK2HOOBS9oF/fzrFWFnIUl1fP2uxPL/AfWABlgtjwQX2xa3RFy1VpRSymv2pWbxz3mb6dOyFjed18LuOH7Jzst/NwG9XSwvbm0UOda52MU6HYF9xpjMSsqmglCfPn3sjqACTGGR4d45awkTYfLILiE7llZZ3CokItIbuBSrADQEooEUYBvwE/C5Maa8rYTPgJuBS4BPnJZfAhwwxhwRkS+BG0WknzHmJ0eGGsBQ4L/uZFfq2WeftTuCCjBv/byLlXtO8NKVXUJyePjyKlchEZEbgPuwrqLKANYD24FsrFNPvYDrgP+IyGzgCWPM7jJedj7wI/CGiNQGdgFXYLVAbnSs8yWwFJghIvfz+w2JArxQzveolFJu23I4g5f/l8ylnepz+TmhNeOhu8osJCKyDqgLfABcD6w1xvypA1xE4oAhwLXAJhG50Rgzq7TXNcYYEfkb8CzwBFa/x1bgWmPMfx3rFInIEGAyMA2IwiosFxpj9rv1TlXIGzFiBABz5861OYnydzn5hfx91lpqREfwzOWhN+Ohu8rTInkPeN0Yk3OmlRw3Cn4EfCQiXYD6Zb2wMSYDuN3xKG2dNKybF28qR1alSpWamlr2SkoBzy/YytYjp3hvTA8SYiPtjuP3yiwkzsOXiIi4ao242GYdsM7DbEop5XMLtx3jvV/2MObc5lzYvq7dcQKCu5f/7hCRu/SOcqVUMErJzOW+OetpV686Dw1qb3ecgOFuIfkM+D9gv4i8ICJNKiGTUkr5nDGG++esIyMnn1ev7kZURBW7IwUMtwqJMeY+oDHWlVODgJ0i8rGI9KiMcEp504ABAxgwYIDdMZSf+mDpXn7cdpyHB7XXARndJOXo8ih9Y5EBWEO5DwaWAS8bYz7zUjavSUxMNKtWrbI7hlLKT209ksGwqb9wXqtavDumh16lBYjIamNMYnnWdfeGxIZYl+k6P37AurfkKqwbC7U9qJQKGNl5hdw181dqRIXz4pVdtIhUgLtDpBzAGrokEzgFpGMVkXTgU8dXpfzSoEGDAPjmm29sTqL8yRPzNpF8NJMPbupJ7WpV7Y4TkNwtJL8CbYHpwBRjzB5vB1KqsmRnZ9sdQfmZL9Ye5OOV+7mtfysuaKujhFeUu53t3bFmLWwNbBORWSLSs1KSKaVUJdqdcpqHP91AYrN47vlLW7vjBDS3h5E3xvxojBkCdMU6vfWTiCwWkb+JnlxUSgWA3IJC7vjvGiLCw3j16m46UZWHKnz0jDFbgFuBLsAOYDaQ7KVcSilVaZ6dv5VNhzKYfEUXGuqovh5z96qt1UA1rJkNa2ANIw+/zyGiExkrvzVkyBC7Iyg/sGDjEaYv2cPNfVswsGM9u+MEBXc729dgDeV+podSfum+++6zO4Ky2Z6U09w/Zx1dGsfx4KU6BIq3uFVIjDG3VlYQpZSqTNl5hYyfsZoqVYT/XHsOkeHaL+Itdk61q5RP9e/fH4CFCxfamkP5njGGR7/YyLajp3h3TA8ax8fYHSmolFmSReQLEelW3hcUkSgRuUdExnsWTSmlvGPWyv18svoAd17Ymgvb6dDw3laett1eYJmILBeRu0XkHBH5Q0tGRBo6Lv99BziMNQnVmkrIq5RSbtl4MJ3HvtzE+W1qc/dAvV+kMpTn1FYecCFwNdYQ8nGAEZEMIBdrvK0IrCu3VgATgQ+NMUWVklgppcopPSufCR+tplZsJFNGdaVKmN7qVhnKU0gmArONMXeKyClgAdAHaIA1h3oq1lzri4wxeystqVJKuaGwyHD3rF85kp7Dx2P7UEvH0ao05SkkaVitDoAHgc+NMc9XXiSlKsfIkSPtjqB86JXvklm47ThP/e0sujeLL3sDVWHlKSSLgckiUgfr9FXFJzBRyka33Xab3RGUjyzYeJipP+7gqh5NuLZXU7vjBL3yFJI7gPcdDwMkich6rJGAix+bjDH5lZZSKS/IysoCICZGL/0MZslHT3HP7HV0a1qTJ4Z30vlFfKDMQmKMOQT8RUTqA4eAWUBN4FLgdsdq+SKyGfjVGHNzZYVVyhODBw8G9D6SYJaelc/YD1YRWzWc10d3p2q4zrPnC+W+IdEYc0REPgNecQzYiIhUwxoFuJvjcU6lpFRKqTIUd64fPJnNzFt7U69GlN2RQoa7Q6SMKPF9JlYfymJvhlJKKXe98O3W3zrXE5sn2B0npOhgM0qpgDdn1X7e+GkX1/VuxujezeyOE3K0kCilAtqqPWn847ONnNe6Fo8N7Wh3nJCkgzaewco9abSpW42aMZF2R1FeMGbMGLsjKC/bn5bFuA9X0yg+mmnXdCdCZzq0hRaSUpw4nceYd1fQuXEcH9zUS4ecDgJaSIJLZm4Bt7y/ivzCIt6+IZG4mAi7I4Us/XQsRXxsJE9ddhbLdqUx6dMNGKP3YQa6lJQUUlJS7I6hvKCwyDDx41/ZcTyTadd2p1WdanZHCmnaIjmDy7o1Zm9qFlOSttO8Vgx3DmhjdyTlgSuuuALQ+0iCwZNfbSZpyzH+ObwTfdvUtjtOyNNCUoa7B7Rhb2oWL32XTNNaMQzv2sjuSEqFtHcX7/5tzvXr+zS3O45CC0mZRITnRnTm4Mls7p+znkY1o/UadaVs8u2mIzz59WYu6VSPhwd3sDuOctA+knKoGl6FN0Z3p1F8NLd+sIo9KaftjqRUyFm7/yR3f/wrXRrXZMqobjq3iB/RQlJO8bGRvDemBwA3vLeC46dybU6kVOjYl5rFzdNXUrd6FG/fkEh0pI6h5U+0kLihee1Y3h3Tg2MZudw0fSWncwvsjqTcMGHCBCZMmGB3DOWmtNN5jJm+goIiw3s39qC2TlDld7SQuKlb03j+c203Nh/OYPyM1eQV6IzCgWLUqFGMGjXK7hjKDadzC7hx+koOnsjmresT9TJfP6WFpAIual+PZy/rzM/bU3ho7nq9xyRA7N+/n/3799sdQ5VTfmEREz5aw4YDJ/n31d3o2UIvcvFXetVWBY3s0YSjGTm89F0ydWtE8dCg9nZHUmW47rrrAL2PJBAUFRnun7OORcnHee7yzlzcqb7dkdQZaCHxwB0XteboqRxe/2kntWIjufWClnZHUirgGWN4ev4WPl97iPsvacdVPXWqXH+nhcQDIsITw87iRFY+T8/fQvWocP2hV8pDbyzaxTuLdzPm3Obc1r+V3XFUOWgh8VCVMOGVkV3JzClg0mcbqBYVzpCzG9odS6mA9OGyvTz3zVaGdmnIY0M66nzrAUI7270gMjyM10d3J7FZPH+ftZYftx2zO5JSAefTNQd49PONDOxQl5dHdiFMbzgMGFpIvCQ6sgrvjOlB23rVmTBjNSt2p9kdSZVw7733cu+999odQ7nwzYbD3DdnHee1rsXUa87ReUUCjITCpauJiYlm1apVPtlXamYuI99YytGMXGbc0ouuTWr6ZL9KBaoftx1j7AerOLtxTT64qSexVfWMuz8QkdXGmMTyrKtl38tqVavKjFt6kRAbyXXvLGfDgXS7IymHbdu2sW3bNrtjKCfLdqUy/sPVtK1XnXfH9NAiEqD8qpCIyAIRMSLyVInl8SLytoikiMhpEUkSkc525SxLg7hoZo7tTVx0BKPfWc6mQ1pM/MG4ceMYN26c3TGUw/Jdqdz43kqaJMTwwU09iYvWGQ4Dld8UEhG5GujiYrkAXwKXAncCI4AI4EcRaezTkG5oVDOambf2JjayCqPfXs7WIxl2R1LKbyzflcqN01fSsGYU/721F7V0/KyA5heFRERqAq8A97h4ehjQF7jOGDPTGLPAsSwMeMB3Kd3XJCGGmWN7UzW8Cte+tZzko6fsjqSU7VbsTuPG6StpEBfFzLG9qVs9yu5IykN+UUiAF4BNxpiZLp4bBhwyxvxYvMAYkw7MA4b7KF+FNasVy8yxvakSJlzz1jJtmaiQtmJ3GmPeW6FFJMjYXkhEpC9wPXBbKat0Aja6WL4JaCoifj8caIvaVjEJDwvjqjeXsfGg9pmo0LN8Vypj3ltB/bgoZt6qRSSY2FpIRCQCeAOYbIwp7XKaBOCEi+XFN2rEl/LaY0VklYisOn78uOdhPdSqTjVmj+tDbGQ4V7+1jF/3uXpLqjI98sgjPPLII3YLhRqsAAAWbUlEQVTHCEmLko9zg6Ml8vGtvalbQ4tIMLG7RfIgEA08fYZ1BHB1s8sZb3s1xrxpjEk0xiTWqVPHg4je07RWDLPG9SYhNpLRby/XmxZ9bODAgQwcONDuGCHn201HuOX9VbSsXY1Z4/poEQlCthUSEWkK/AN4FKgqIjUdne44fV8Fq+XhaiKC4pZIQP1p3zg+hllj+1A/Loob3l3BLztS7I4UMtauXcvatWvtjhFSvlh7kNs+WkPHhjWYeWtvnd0wSNnZImkJRAEzsIpB8QPgPse/O2P1hXRysX1HYJ8xJrPyo3pX/bgoPh7bh6YJMdz43koWbDxsd6SQMHHiRCZOnGh3jJDx8Yp9TJy1lh7N45lxSy/iYvQ+kWBlZyFZC1zo4gFWcbkQ2IF1D0kjEelXvKGI1ACGOp4LSHWqV2XWuN6c1agGt320hpkr9tkdSSmvMMbwxk87eejTDfRrW4fpN/akmt6xHtRs+981xpwEFpZc7hg2eq8xZqHj+y+BpcAMEbkfq6UyCauP5AUfxa0UNWMimXFLL277aA2TPt3Aiaw8JvRrpUNnq4BVVGRNSvXO4t0MObsBL43sQtXwKnbHUpXM7s72MhljioAhwHfANOAzoBC40BgT8BNwx0SG89b1iQzv2pAXFmzj6a+3UFQU/ANpquCTV1DE32ev/W1Sqlev6qZFJET4XXvTGPOnP8eNMWnATY5H0ImoEsYrI7sSHxPJ24t3k5KZy/NXnK2/hCpgZOYWMGHGan7ensIDl7bTlnWI8btCEqrCwoT/G9qROtWr8uK32ziSkcMboxO1g9KLnnnmGbsjBKVjp3K45f1VbDqUwQtXnM3IxCZ2R1I+pvOR+KEv1h7k/jnraZIQzfQbe9IkIcbuSEq5tO3IKW6avpK003lMvaYbAzrUszuS8hKdjyTADe/aiA9u7klKZh6XTfuFdftP2h0pKCxZsoQlS5bYHSNo/JR8nBGvLSG/sIjZ4/poEQlhWkj8VO+WtZg74VyiIqpw1ZvL9F4TL3j44Yd5+OGH7Y4RFGYs28tN0625RL644zw6N46zO5KykRYSP9a6bjU+u+082jeozvgZa/hX0na9okvZqqCwiCe/2swjn2+kX9s6zBnfhwZx0XbHUjbTQuLn6lSvysxbe3P5OY14JSmZO2auISuvwO5YKgSdzMrjxukrf7u8983ruuuNhgrQq7YCQlREFV66sgsdG9Tgmflb2J2SxVvXd6dxvHbCK9/YeiSDsR+s5kh6Ds+P6MyoHk3tjqT8iLZIAoSIcMv5LXl3TA8OnMhi+NRfWKIDPiof+GbDYS6ftoSc/EI+Htdbi4j6E738NwDtPJ7J2A9WsTvlNPdd0o7xF7QiLExv/ipL8ci/Xbt2tTlJYCgoLOLl75KZtnAn3ZrW5PXR3amnQ8CHDHcu/9VCEqAycwt4aO56vlp/mIEd6vLSlV315kXlNccycrhz5q8s353G1T2b8PiwTjrSQojR+0hCQLWq4fz76m48PrQjC7cdZ8jUn3UK3zIkJSWRlJRkdwy/98uOFAa/+jPrD6Tz0pVdePZyHa5HnZm2SILA6r0nuP2jNaRl5fHwoPbccG5zHefIhf79+wOwcOFCW3P4q8Iiw9QfdjDl+2Ra1anGtGvPoW296nbHUjbRFkmI6d4snq/v6kvf1rV5fN5mbn5/FamZuXbHUgHk0MlsRr+9nFeSkrmsayO+uP08LSKq3LSQBIla1aryzg2JPD60I4t3pHDpv37m5+3H7Y6lAsC8dYe4dMoi1h04yQsjzualkV2I1ftDlBu0kAQREWHMeS344vbziIuO4Lp3VvDM/C3k5BfaHU35oVM5+dwzay13zvyVlnWqMf+u8xnZo4meFlVu00IShDo0qMG8O/pyba+mvLloF0P+vVgHflR/sHxXKoNf/ZnP1x7krotaM2d8H5rXjrU7lgpQ2tke5H5KPs6Dn6zneGYuE/q14s4BrUP2Cpxt27YB0K5dO5uT2Od0bgHPL9jKB0v30iQhmldGdiWxeYLdsZQf0vtISgjlQgKQnp3Pk19t5pPVB2hfvzqTr+zCWY10tNZQs3h7Cg/OXc+h9Gxu6NOcBy5tR0yk9oUo1/SqLfUHcdERTL6yC+/ckEja6TyGTV3Mk19t5nRuaA3+OG/ePObNm2d3DJ9Lz8pn0qfrGf3OcqqGhzF7XB8eH9ZJi4jyGm2RhJj07HxeWLCVj5bvo0FcFE8M68TFnerbHcsnQu0+EmMMn645yLPfbCHtdB63nt+Sv/+lLVERoXlqU7lHWySqVHHRETx9WWfmTjiXuOgIxn64mls/WMXBk9l2R1NetO3IKUa9uYx756yjSUIM8+7sy6TBHbSIqEqhbdsQ1b1ZPPPu7Ms7i3czJSmZiyYvZNwFLRnfv5We8ghgGTn5TP1hB+8u3k21qHCeu7wzIxOb6KCeqlLpJ0YIi6gSxvh+rRjapSHPfbOVV3/YwexVB3hwUDuGd2mkHz4BpKCwiJkr9vFK0nbSTucxMrExDw3qQEJspN3RVAjQQqJoVDOaf1/djRv6NOOJeZv5+6x1vL9kLw8P7kDPFnppqD8zxvDD1mM8M38LO4+fpleLBB75a0edQ135lHa2qz8oKjLMXXOAF7/dxrFTufRrW4f7L2kXFJcL79+/H4AmTZrYnMQ7Vu9NY/K3ySzdlUrL2rE8NKg9f+lYT+9MV16h95GUoIXEfdl5hXywdA+v/bSTk1n5/LVzA+65uC2t6lSzO1rIW7v/JC9/l8yi5OPUrhbJHRe25trezYiootfOKO/RQlKCFpKKy8jJ5+1Fu3h78W5y8gsZcnZDJvRvRYcGNeyO5rZZs2YBMGrUKJuTVMyGA+n86/tkkrYcIz4mgnH9WnF9n2Z6cYSqFFpIStBC4rmUzFzeWrSLGcv2cjqvkAHt63Lbha3p3ize7mjlFoj3kRhj+Hl7Cm8s2skvO1KpERXO2AtaMua8FlTTEXpVJXKnkOhPoiqX2tWqMmlwB27r35r3l+7h3V92M+K1JfRskcBN57VgYIe6hOupFa/JLyxi/obDvP7TLrYczqBu9apMGtSeq3s1pUaUTqms/IsWEuWWuJgI7hrQhpv7tmDmin2898sexs9YTaOa0Yzu3YyrejQhXi85rbAj6TnMXLGPmSv2cexULq3rVuOFK85meNeGITvYpvJ/WkhUhcRWDeeW81sy5tzmJG05xvtL9vD8gq1MSUpmaJeGXNm9MT1bJOgVROVQVGRYsjOVGcv28t2WoxQZQ7+2dXi2dzMubFdX7+dRfk8LifJIeJUwLj2rPpeeVZ9tR07x/tI9fPHrQT5ZfYCmCTFc0b0xI7o3plHNaLuj+p0dx07x6ZqDfLH2EAdPZpMQG8kt57fg2p7NaForxu54SpWbdrYrr8vKK2DBxiN8svoAS3amIgI9mycwuHMDLj2rPvVqRNmSKyUlBYDatWvbsn+AgyezWbDxCJ//epANB9OpEiac36Y2l3VrxCWd6utYWMpv6FVbJWghsc/+tCw+XXOQrzccIvloJmCN8zXorPoM7FAv6GflM8aQfDSTbzcd4X+bj7DxYAYAZzWqwWXdGjOsS0PqVK9qc0ql/kwLSQlaSPzDjmOn+GbDEeZvPMKWw9YHatOEGC5oW5t+bevSp1WtSr2kdfr06QCMGTOm0vYBkHY6j6U7U1m8I4XFO46zP80aWfmcpjW5uFN9Lu5Yj5Z6Y6fyc1pIStBC4n/2pp5m4bbjLEo+ztJdqWTlFRIeJnRqFEdis3gSm8XTvVk8db14Gqwy7iMxxnDgRDZr959k7f6TLNuVyqZDVpGsXjWcXi0TuKh9PQZ2qOvV96JUZdP7SJTfa1YrlhvOjeWGc5uTW1DI6j0n+HlHCqv3nGDGsr28s3g3AI3jo+nYoAbtG9SgQ/3qtG9Qg6YJMVSx4Uqm3IJC9qRksf3YKZKPZrL5UDpr958kJTMPgKrhYXRrWpP7Lm7Lua1rc3ajOL23RoUELSTKdlXDq3Bu69qc29rqBM8rKGLToXRW7z3Br/tPsvVwBklbjlLkaDxHhofROD6apgkxNImPoUlCNA3ioqkVG0lCtUgSYiOJj4l0a+ypnPxCTmblk56dz4msPA6nZ3PoZA6HTmZz6GQ2+9Ky2JOaRaEjRJhAi9qx9Gtbl65Na9KtSU3a1a+u412pkKSFRPmdyPAwujWNp1vT34dfyckvZPvRTLYcyWDHsUz2p2WxLy2LNXtPkJHjeu756IgqVI0Io2p4GFXDq7DhwEkALpy8kPzCIgqLDPmFRWTkFJBXUOTyNRJiI2kQF0XrutUYdFYD2tSrRpu61WlZJ1avsFLKQQuJCghREVXo3DjO5Twb6Vn5HD2VQ2pmHmmn80g7nUvq6TxO5xaQW1BEbn4RuQWF7IiogiB0bhRHeJgQXkWoEhZGjahwakRHEOd4xMdE0qBmFA3joomO1GKhVFm0kKiAFxcTQVxMBNQ783rPDPsZgJgYvdlPKW/SQqJChhYQpSqH9gyqkDFt2jSmTZtmdwylgo4WEhUyZs+ezezZs+2OoVTQ0UKilFLKI1pIlFJKeUQLiVJKKY/YVkhE5AoRmSsie0UkW0S2icizIlK9xHrxIvK2iKSIyGkRSRKRznblVkop9Ud2Xv57H7APeBg4AHQDHgcuFJFzjTFFYk2v9yXQArgTOAFMAn4Uka7GmAO2JFcByZuDNSqlfmdnIRlqjDnu9P1PIpIGvA/0B34AhgF9gYuMMT8CiMhSYDfwAHCXTxMrpZT6E9tObZUoIsVWOr42cnwdBhwqLiKO7dKBecDwyk2olFKqPPyts72f4+sWx9dOwEYX620CmoqIzg6klFI285tCIiKNgH8CScaY4lmoErD6RUpKc3yNd/Fc8euNFZFVIrLq+HFXjR+llFLe4BeFxNGy+AIoAG50fgpwNYVjmbMaGWPeNMYkGmMS69Sp452gSiml/sT2QRtFJArryqyWQL8SV2KlYbVKSipuibhqrSillPIhW1skIhIBzAV6AoONMRtKrLIJq5+kpI7APmNMZiVHVEopVQY7b0gMAz4CBgDDjTHLXKz2JdBIRPo5bVcDGOp4TimllM3sPLX1H+BK4GngtIj0dnrugOMU15fAUmCGiNzP7zckCvCCj/MqpZRyQYxx1Zftgx2L7AGalfL0E8aYxx3rJQCTgb8BUViF5R5jzDo39nUc2FvBqLWBlApuG6r0mLlPj1nF6HFzX3mPWTNjTLmuVLKtkAQKEVlljEm0O0cg0WPmPj1mFaPHzX2Vccz84vJfpZRSgUsLiVJKKY9oISnbm3YHCEB6zNynx6xi9Li5z+vHTPtIlFJKeURbJEoppTyihUQppZRHtJA4EZF7RGSeiBwWESMij7u5fV8RWeKYOviIiLwsItGVFNcviEiYiEwSkT0ikiMi60RkRDm3ne44ziUfUyo7t6+ISBMR+URE0kUkQ0Q+FZGm5dw2SkRedPw8ZovIUhG5oLIz283DY+bq58mISNfKzm0nEWksIv92/IxkOd5z83JuW+Hf4WJaSP7oVqAu8Lm7G4rI2cB3wDFgCPAI1kjG072Yzx89iTVF8lRgELAMmCMig8u5/XGgT4nHK96P6XsiEoM102d74AbgOqAN1lTRseV4iXewfiYfw/qZOgx8G8wfil44ZmD9zpX8mUr2elj/0hoYiTX6x89ubuvp7zAYY/TheABhjq/hWMPXP+7Gtp8B24EIp2XXO17nHLvfWyUdr7pALtZIBM7LvwfWl2P76VjD4dj+Xirp+NwNFAKtnZa1wJou4Z4ytu3i+Nm50WlZOLAN+NLu9+aPx8yxrgGesvt92HDcwpz+fYvjODQvx3Ye/Q4XP7RF4sQYU1SR7RyjGF8KzDbG5Ds9NRvII3inBb4EiARmlFg+A+gsIi18H8mvDAOWGWN2FC8wxuwGfqHsn4lhQD4wy2nbAuBj4BIRqer9uH7Bk2MWsir62YWXfoe1kHhHK6xxwP4wLbAxJgfYiTXsfTDqhPXXzI4Syzc5vpbnfdcVkRQRKRCRZBF5UESqeDWlfc40VXRZx6YTsNsYk+Vi20isUxnByJNjVmyCiOQ6+gp+EJHzvRcv6Hjjd9j+ia2CRPHkW6VNC+xqcq5gkACcNI62sJM0p+fPZC2wGuuHNgq4DHgW65z4LV7MaZczTRVd6jTR5di2+Plg5MkxA+sv6a+AQ1iDwt4P/CAifzHGLPRWyCDi6e8wEMSFREQGYnV+l+UnY0x/T3fn+FqhaYH9RQWOWYWnQgYwxpS8Omu+iGQCE0XkeWPM9vK8jp+r6PHx6NgGOE9+pq5z+vZnEfkCq4XzFNDXC9mCjVd+zoK2kABLgA7lWK/kqYOKOFP1juf3ZqK/c/eYpQHxIiIl/qKJd3reXTOBiUAi1sULgewEpf9MlDVNdBrg6pJXT45tIPDkmP2JMeaUiHwN3OxpsCDlld/hoC0kjnPLW320u51Y5xn/MC2wYz76lsAcH+XwSAWO2SagKlYfkfM51uLzqpsrEONMrbtAc6aposs6NpuAy0QkpkQ/SUesCzhKntMOFp4cs9KU9le38tLvsHa2e4ExJg9YAIwUEefifAXWf1KwTgu8AOtD7doSy0cDGx1X27jrGqxf+pUeZvMHXwK9RaRl8QLHTWLnUfbPxJdABNYsosXbhgOjgP8ZY3K9HdZPeHLM/sQxNfdfgeVeyhdsvPM7bPf1z/70wDqdcgXWjT0G6/LdKxyPGKf13gEKSmzbFcgGPsWah/5mrGbhHLvfVyUfs+eAHOAeoD/wGlAEDC2x3vfADqfvmwGLgNuAi4GhwLuObV+z+3156djEYv2VtwHr0tVhwDpgF1CtxLEoAB4rsf3HWKdzbnH8TH3iONZBeV+Sp8cMuA94C+uPkf5YNzRuwPqgPN/u9+aDY1f8WfWa4/NrguP7fk7rFADvlNiuXL/DZ9y33W/enx5YN8iZUh7NS67nYvsLsKYCzgGOAlOcC1AwPoAqWHfx78U6vbceuMLFeguBPU7fJ2CNILDXcbyygTXAHTjdXBXoD6x+jrlABnDK8Z6bl1inOS5ugAWigZeBI45jtBzob/d78tdjhvXHyC9Y08jmA6lYrZiedr8nHx230j67FpZYZ3qJ7cr1O3ymhw4jr5RSyiPaR6KUUsojWkiUUkp5RAuJUkopj2ghUUop5REtJEoppTyihUQppZRHtJAopZTyiBYSpZRSHtFCopRSyiNaSJTyARF5W0TWlFi2WkSMiDR2WjZZRHYH0SyRKgRoIVHKN04A1Yu/EZGLgHZYYx/FO5bFADcBrxpjCu0IqVRFaCFRyjf+UEiAe7FGO87k90mErscaQO8d30ZTyjNBO7GVUn7mBFADQEQ6YA2d3w64nN8LyR3A28aYDFsSKlVBWkiU8o0TQLSj7+Ne4AtjzC4RScea6nQg0B5rEialAooWEqV8o3i+8dZYs89d6Pg+A6tFcjnwqTFmrw3ZlPKIFhKlfKO4kDwMrDHGLHV8nw50x2qJnGdHMKU8pYVEKd8oLiTXYs27XiwDuApYboxZ5vNUSnmBXrWllG8UF5J9WFPHFkvHulLrZZ8nUspLdKpdpZRSHtEWiVJKKY9oIVFKKeURLSRKKaU8ooVEKaWUR7SQKKWU8ogWEqWUUh7RQqKUUsojWkiUUkp55P8Bgvgy2qlgsAkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "npr.seed(0)\n",
    "\n",
    "# Generate random 1D data set\n",
    "d = 1\n",
    "n = 30\n",
    "X = npr.randn(n,d)\n",
    "y = npr.randn(n,1)\n",
    "def f(w, X, y):\n",
    "    return np.sum((X@w-y)**2,axis=0)\n",
    "\n",
    "wgrid = np.arange(-1,1,2**(-5))\n",
    "fw = f(wgrid[None], X, y)\n",
    "minind = np.argmin(fw);\n",
    "minw = wgrid[minind]\n",
    "plt.plot(wgrid, fw);\n",
    "plt.xlabel(\"$w$\");\n",
    "plt.ylabel(\"$f(w)$\");\n",
    "plt.plot((minw, minw),plt.ylim(),'--k');\n",
    "plt.title(\"No regularization\"); # $f(w)=\\sum_{i=1}^n (wx_i-y_i)^2$\n",
    "print(\"minimizer: w=%.2f\" % minw);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding, ask yourself: do you understand what this picture is showing? It's the parameter space, but in 1D, so we don't need a contour plot like we did in 2D."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L0 regularization\n",
    "\n",
    "$$f(w)=\\sum_{i=1}^n (wx_i-y_i)^2 + \\lambda \\|w\\|_0$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minimizer: w=0.00\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAElCAYAAADOTWQ3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xm4HFWd//H3JyEQFpEQwBEREgMMklGiExUIDIsijLKJ4MaILOKuA4zIAzM/xUFnxowP4IIgAxhnFAFlH8coCAEJAQ2rRCMmBhAwCknIggRyc7+/P6raNJXq3O5b3V3VfT+v56mn76061XW6bt/+1vmec6oVEZiZmQ3XqLIrYGZmvc2BxMzMCnEgMTOzQhxIzMysEAcSMzMrxIHEzMwKcSAx6zBJx0sKScf38jFaqMuMtC4Tyq6LdYcDiRUiaUL6oXFdk+VHSzpV0jxJqyUtTj94XtHpulp71P3NZ5RdF6sGBxLrtkuBc4G1wFeAW4BjgbslbV9mxXrctcCr08eynUlSlyfKroh1x0ZlV8BGDklvAT4A3AocHBFr0vXXA1cAXwLeX14Ne1dELAeWl10PgIj4A/CHsuth3eMWiXXTB9PH/1cLIgARcSVwH/AuSVsO9ST1/QGS3iHpLknPSppVV2aUpJMl3S1pVbrcKemoBs/5ekk3peWWSbpa0k6SZkmKTNmGfQCt9A9IOlHSDZIelfS8pKclXS9paquvOa+PpFb3DSxn15U9UNK3JD2cPu/K9Hy9O1sPYFH66wcyzzdhQ+dA0haSvpAe43lJT6Xn+TU5r7dW9zGSPl93juZJeu9Q59a6yy0S66b9gFXAXTnbfgK8Dtgz/bkZ7wbeDFwPzAIGACQJ+B7wLuBXwLfT8m8HrpZ0SkR8pfYkkl4H3A5sAlwFPAbsC9wBLG361bXuAuB+kte7BNgJOBI4WNL+EZF3nnJfcwMz0jJZxwC7A3+uW/cZ4FXA3SQpqa2Bw4ErJL08Is5Py91PkpL8R+ABoL5v7JlGFZE0liSN+Yb0GFcDO5D8jQ6RdHBE3JGz6xXA64EfknxevRe4XNIzEfGjRsezLosIL16GvQATgACuG6LcFmm5BxtsPynd/rEmjnl8WnYA2Ddn+4fT7RcAo+vWb07yIfY8sH3d+tlp+bdnnue/0vWRWT8jXT8h59jrbaur7/GZshNz9n81sBK4ucXXnHuMnHKHkfRP/QLYdIi6bE4SLJYDm+X8zWc0OEbeOfhcuu5SQHXr9wMGgd8Co+rWz0rL3wlskSkfwI/Lfu97Wbc4tWXdUktZrWiwvbb+pS0857UR8bOc9R8HlgGnRsTa2sqIeBY4B9gYOAqSEUjA3sCciPhh5nnOZsNX/IVExKKcdb8m6UPaV9LGObs1es1DkvRa4HKS/osjIuK5IeryLElrbkvgjcM5Zp0PAKuBsyKNCOkxbgNuAHYGpuXsd1ZErMqUfwRYL/1n5XFqy7pF6WM7v7dg7noHkTYD/oYkPXVWkuV6kW3Tx93Sxz3SxzuzBSPiCUmPkaR82k7SzsBZwAHA9iQBrt541u+0Xu81N3mslwE3kvSLHh4RT2a2b0mS3jqC5PVulnmKlw/nuHXPPRG4LyL+mFNkVnrcKUA2SN6XU/4JYMfh1sfaz4HEuqU2oqhRi2PLTLlm/Cln3TiSoLUTSTqlkc3Tx5ekj09t4BhtDySSdgF+nh7/ZpJhu6tI0jxHkgS4TRrUp9VjjSXpy3glcExE3JvZvjFwG8kH+T0kqamlJCmwKSQf8nl1aVbtb5sXRAAWZ8r9RSSj0bIG8EChSnEgsa6IiFWSFgMTJY2uTzmldkkfF7TytDnraimy2RGxTxPPsTJ93LbB9u1y1g2mj6Nztg056ix1CrAV8L6I+F79BklvYl1LKWs4LbrLSAYx/EtEXJ2zvdYauDgiPpypyxnp9iJqf5OXNdj+skw56zGO6tZNt5F0uu+Zs+2twAvkj+hqWkSsBOYDfyNpiyZ2eSB93Du7IZ1tn5dCqY1OetFsfEmjaBwAsialjzdmnmNTklFKbSHpsyQjnb4bEV9spS6pvH6L2kVAXiBdT0SsIBky/GpJeQF7v/Tx/maez6rHgcS66dL08RxJY2or07kKrwO+n37oFPU1khTaN9K0zotImixpO4CIeIQkeO0l6e2ZomeT32q/J308LrP+UzSfBnssffzLB3U6bPnfyG8FtUzSMSSvYQ7JqLim65LufxTJKK+sZeljK7e1+W9gLPCvmWPsQ5LKW0gyes56kFNb1i6vV+N7L/0kIi6PiJskfZtkBM89kn7EurkETwJntKkuF5K0MN4P7C/pFpI8/MuB15KkcfZiXX/DJ0jmkVwrqX4eyQTgQSA7Ye5a4FHgJEk7AL8kCYRTSFpd+zG0i4ATgGskXUnSN7QvSaf0LGD/1l5yrm+T9Bc9AJyZM/BgVkTMImmJPAacIWkySYtuMnAIyWt9R/1OaZryF8B+ki4hCQIBXNigTwOSuxYcCnwkHT12O0kgejfJaK4TI2Kwwb5WdWWPP/bS2wvr5hRsaDm/rvxo4DTg1yTzOf5E8oG3QwvHPJ7m5kwcSzKUdll6rMeAHwMfBTbPlJ1K0un9bFr+mvS1/RJYnvPcOwP/S9JBvoJkCOsutDaP5M0ko8VWkkxIvLrV59jQ9ib+LmfXlZ1EEjSeSuvzM+DgDdR9t/RcLq97vgnptvXqn65/CUmLawFJGnNJep73yHk9s8jM32lmm5dyFqV/GDPLSPtY/gQ8FBFF51GY9S33kdiIl97PaXxm3SiSdMymJLcjMbMG3CKxEU/SNsDjJPe8ephkjsk+JBMb5wNviLrZ1Wb2Yg4kNuKlI7vOI+mzeDnJ5Lvfk/R7nBMRnbxxo1nPcyAxM7NCRsTw32222SYmTJhQdjXMzHrGPffc83RENLrjw4uMiEAyYcIE5s4d1r3uzMxGJEmPNlvWo7bMzKwQBxIzMyvEgcTMzApxIDEzs0IcSMzMrBAHEjMzK2REDP81MxtJBtYOcu5ND3PnwiXsPWk8px20KxuN7ly7wYHEzKzPnHvTw1w2exGr1wwyf/EKBJx+yG4dO55TW2ZmfebOhUtYvSb5nrDVawaZvXBJR4/nQGJm1mf2njSesWOSj/exY0YxbdL4IfYoxqktM7M+c9pBuyJg9sIlTJs0nlMP2rWjx3MgMTPrMxuNHsXph+zG6V06nlNbZmZWiAOJmZkV4kBiZmaFuI/EzKyHdXvyYR4HEjOzHtbtyYd5nNoyM+th3Z58mMeBxMysh3V78mEep7bMzHpYtycf5nEgMTPrYd2efJjHqS0zMyvEgcTMzApxIDEzs0LcR2Jm1iOqMPkwjwOJmVmPqMLkwzzlhzIzM2tKFSYf5nEgMTPrEVWYfJjHqS0zsx5RhcmHeRxIzMx6RBUmH+ZxasvMzApxIDEzs0Kc2jIzq6iqzhvJciAxM6uoqs4byapeaDMzM6C680ayHEjMzCqqqvNGspzaMjOrqKrOG8lyIDEzq6iqzhvJcmrLzMwKcSAxM7NCHEjMzKwQ95GYmVVAr0w+zONAYmZWAb0y+TBPb4Q7M7M+1yuTD/M4kJiZVUCvTD7M49SWmVkF9MrkwzylBhJJ04DPAVOAscAC4OsRcVldmbHAOcA/AFsB9wNnRMTt3a+xmVln9MrkwzylpbYkvRa4GRgDnAy8E/gFcKmkj9YVvTTd/lngUOAPwI8lTelujc3MLE+ZLZL3AKOBwyJiVbruJkl7AMcBF6Y/vw84MSK+BSDpNmAe8K/A4d2vtpmZ1Suzs31jYA3wXGb9M6yr1+FpmStrGyNiALgCOFjSJl2op5lZWw2sHWT6zPkcecFsps+cz8DawbKrVEiZgWRG+vhVSdtL2krSycCbgfPSbZOBRRHx58y+80gC0c5dqamZWRvV5ozc//tnuGz2Is676eGyq1RIaYEkIh4C9geOAJ4AlgEXAB+JiCvSYlun67OW1m3PJelDkuZKmvvUU0+1rd5mZkX18pyRPGV2tu8CXE3SujgMeAtwEXCRpGNrxYDI232o54+IiyNiakRM3XbbbdtUazOz4np5zkieMjvb/42k/+PQiFiTrvuppPHAVyR9j6TlsWPOvuPSx6U528zMKq2X54zkKTOQvAZ4oC6I1PycZKTWdiStlXdI2izTT7I78ALJvBMzs57Sy3NG8pTZ2b4YmCJp48z6NwGrSVobN5DMMzmmtlHSRsC7gZ9ExPNdqquZmTVQZovk68D3gRslfYNkGPDhwHuB8yLiBeB+SVcC50saAywCPgpMBI7Nf1ozs+ro5dvDN6u0QBIRP5D0NuAM4BKSW6QsBD4OfLOu6AnAF4EvkNwi5QHgkIi4t7s1NjNrXS/fHr5Zpd5rKyJ+BPxoiDLPAaeli5lZT8kb6tsvfSM1/dW+MjOrmH4b6pvHt5E3M+ugfhvqm8eBxMysg/ptqG8ep7bMzKwQBxIzMyvEqS0zszYZCXNG8jiQmJm1yUiYM5Kn/0OlmVmX9Nvt4ZvlQGJm1iYjYc5IHqe2zMzaZCTMGcnjQGJm1iYjYc5IHqe2zMysELdIzMyGaaQO981yIDEzG6aROtw3a+SFTjOzNhmpw32zHEjMzIZppA73zXJqy8xsmEbqcN8sBxIzs2EaqcN9s5zaMjOzQtwiMTNrgof6NuZAYmbWBA/1bczh1MysCR7q25gDiZlZEzzUtzGntszMmuChvo05kJiZNcFDfRtzasvMzApxi8TMLMNDfVvjQGJmluGhvq1xiDUzy/BQ39Y4kJiZZXiob2uc2jIzy/BQ39Y4kJiZZXiob2uc2jIzs0LcIjGzEc1DfYtzIDGzEc1DfYtz2DWzEc1DfYtrqUUiaU/gEGBPYHtgU+Bp4DfAbcB1EbGs3ZU0M+uUvSeNZ/7iFaxeM+ihvsPUVCCR9AHg08BkYAXwIPBb4Dlga+BNwPuBCyRdBXw+IhZ1pMZmZm3kob7FDRlIJD0AbAf8N3AccH9ERE65lwKHAscC8ySdEBFXtrm+ZmZt5aG+xTXTIvkWcFFErN5QoYhYDnwX+K6kPYC/akP9zMzaxiO0OmPIQBIR59d+lqS81kjOPg8ADxSsm5lZW3mEVme0GooXSPqUpC06Uhszsw7yCK3OaDWQXAt8Dvi9pOmSXtmBOpmZdYRvxtgZLQWSiPg0sANwJvD3wEJJV0h6QycqZ2bWTqcdtCsnTZvIlFduxUnTJnqEVpuoiS6PxjtLbwY+BbwNuAs4NyKubVPd2mbq1Kkxd+7csqthZtYzJN0TEVObKdtSi0TS9pImS9pH0mHAK4BbgCtIJin+YBiVfZuk2yWtkrRC0lxJB9ZtHyfpEklPS3pW0s2SXtPqccxsZBlYO8j0mfM58oLZTJ85n4G1g2VXqW+1eq+tx4FBYBWwElhOMkFxOXBN+tg0SR8Gvp4u55AEtinAZul2ATcAE4FPAstI0mq3SpoSEY+3WH8zGyE8Qqt7Wg0k9wG7AjOA8yPikeEeWNIE4Hzg9PohxsCP634+HNgHODAibk33mwMsAj5DklYzM1tP3ggtTzrsjFY72/+W5MN9Z+A3kq6U9MZhHvtEktbNRRsoczjwZC2IpHVYDtwIHDHM45rZCOARWt3T8pTOiLg1Ig4lSUGtBG6TdIekI9NUVLP2AeYD75G0UNKApAWSPl5XZjLwUM6+84AdPZ/FzBrxCK3uGfb3kUTEryWdDEwHzgKuAh4FdmnyKbZPl/9M918IHAN8XdJGEfEVkhtCPpKz79L0cRxJf816JH0I+BDAjjvu2GSVzKxf+B5a3dPqbeTvAbYAXgJsSXIbeYBaS+RVLTzdqPR5jo+Ia9J1t6R9J2dK+mr6vHnjk4ds+UTExcDFkAz/baFeZtaDfB+t8rTaIrmXZOTUhpZmLSFpvdyUWf8Tku88eTlJy2PrnH3HpY/+7hMzAzxKq0wtBZKIOLmNx55HMvckq9baGEzLvDWnzO7AYxGRm9Yys5HHo7TKU2a7rzYD/uDM+oOBxyNiMckckldI2q+2UdKWwGHpNjMzwKO0ytTMF1tdD5wdEfc184SSxgIfA/4cERsa2vt/wK3ANyVtA/wOOJqkBXJCWuYGYA7wHUmns25Cokg6+TvG+Vaz3uJvOixPM6mtR4G7JN0PXA78DHgwIgZqBSRtD7yRpKVwFPAEyTyRhiIiJB0J/DvweZJ+j/nAsRFxeVpmUNKhwJeBbwBjSQLLARHx+1ZeaKucbzXrLR6lVZ5mAskLwAHAe0luIf9SICStAJ4nCQBjSFoJPwdOAf4nIoa8sU1ErAA+ni6NyiwlCUobDEzt5nyrWXU5Y1AtzQSSU4CrIuKTklYCM4G9SEZVjSUZfTUfuD0iHu1YTbts70njmb94BavXDDrfalYxzhhUSzOBZCnrhtueAVwXEV/qXJWqwflWs+pyxqBamgkkdwBflrQtjScI9h3nW82qyxmDamkmkHwC+Ha6BHCzpAdJ7gRcW+ZFxJqO1bIinJc1qwZnDKplyEASEU8CB0n6K+BJ4EpgK5LZ57VO8jWSfgXcFxEndaqyZXNe1qz7Gl3AOWNQHU3PbI+IxZKuBc6LiF8DpHffnQK8Ll1e35FaVoTzsmbd5wu46mv1FinvzPy+iqQP5Y52VqqqnJc16z5fwFXfsG8jPxI5L2vWfb6Aqz4HkhZk87IDaweZPnO+O9/NOsgXcNXnQFKAc7dm7eWO9d7kQFKAc7dm7eWLs97kPEwBvm21WXvlXZxZ9blFUoBzt2bt5Y713uRAUkBe7taz382GzxdnvcmBpM2c4zVrjjvW+4cDSZu5A96sOb7o6h/OubSZO+DNmuOO9f7hFkmbOcdr1hx3rPcPB5I2cwe8WXN80dU/HEi6wLlgG+ncsd7fHEi6wB3wNtL5Yqq/Ob/SBe6At5HOHev9zS2SLsjLBbvfxEYSd6z3NweSLsjLBU+fOd9NfRsx3LHe3xxISuJ+E+tneS1ud6z3LweSkripb/3MnesjiwNJSdzUt37mFvfI4kBSEk9ctH7mFvfI4kBSIU4HWC/KuwByi3tkcSCpEKcDrBc1ugBy5/rI4bxJhXjiovUiTzY0t0gqxBMXrRe5P8QcSCrEExet6twfYnkcSCrO/SZWJe4PsTzOkVSc+02sStwfYnncIqm4bNrgkwfuzPSZ891nYqVwf4jlcSCpuGy/iftMrFvcH2LNciDpMe4zsW5xf4g1yzmRHpPXZzKwdpDpM+dz5AWzmT5zPgNrB0uupfUD94dYs9wi6TF5qQXfWsWKyktjuT/EmuVA0mPy5po43WVF5V2MuD/EmuVA0gd85WhF5V6MHLL+RYtZHgeSPuBbq1hRvhixIhxI+oBvrWKt8LBeazcHkj7lfhNrxMN6rd0qleuQNFNSSPpCZv04SZdIelrSs5JulvSasurZCzxM2BrxsF5rt8q0SCS9F9gjZ72AG4CJwCeBZcCZwK2SpkTE412taI/wMGGryaay9py4tftDrK0qEUgkbQWcB5wKXJ7ZfDiwD3BgRNyalp8DLAI+A3yqi1XtGR4mbDXZC4gT9prASdMmuj/E2qYSgQSYDsyLiO9JygskT9aCCEBELJd0I3AEDiRNyxuZ49Fd/S97ATFn0VKu+/g0X0RY25QeSCTtAxxHTlorNRl4KGf9POA4SVtExKpO1a+fON3V/zxD3cpQaiCRNAb4JvDliPhNg2JbA4/krF+aPo4D1gskkj4EfAhgxx13LFzXfuB0V//zDHUrQ9ktkjOATYEvbqCMgGiwvqGIuBi4GGDq1Kl5+xtOd/WyvL+TZ6hbGUoLJJJ2BP4Z+CCwiaRN6jZvknbAryRpeWyd8xTj0sdlHa1on3O6q3fl/Z2cxrIylNkieRUwFvhOzrZPp8vrSPpC3ppTZnfgMfePFNNMuuuOBU8T/lbGyslrffzgI3s5jWVdV2YguR84IGf9rSTB5VJgAckckhMk7RcRtwFI2hI4jPWHClsbZK9qR0tuoZSs2U70vAsDs04rLZBExDPArOz6ZP4hj0bErPT3G4A5wHcknc66CYkiGTZsbZZNd92x4On1rnxPdT9KV7kT3aqs7M72IUXEoKRDgS8D3yBJh80BDoiI35dauT6VvaqNmfP5zR9XvujK1/0oneNOdOs1lQskEbHeaKyIWAqcmC7WZXlXvkdfNMetlA5xJ7r1msoFEquevLx73gebWynt4U506zUOJDYsbqV0jjvRrdc4kNiwuJUyPM1M9nQnuvUaBxJrG7dSXiwvaDQTWN36sF7jQGJtU6SVUptR36vBpdmg4XubWT9yILGOaraVErz4QzcGA41SJQNLkaDh0VfWjxxIrKOabaXMznzoXn3fE6xYvWbIVgvQ0ZZMu4PGqe7/sD7kQGJdl9dKCR5+0YcuMGSrpXZb6GbSZLB+wMmu+9SBO/PVWxZ0NGi4/8P6kQOJdV3eh2k2uKwdDGbMeWSDrZbZC5f85ef6dc0GnOy6uxYu4VdpMHDQMGueA4lVQvZDd2DtIKNHaYOtlmmTxhMwZJqsUcDJrpuf3gamvoyDhtnQHEiskppptdT6F4ZKkzUKONl1u73sJX9pkThomDVPEf3/5YFTp06NuXPnll0N65KBtYOcd9PD6wWcodZ98sCd+dotC9YLGmYjkaR7ImJqU2UdSMzMLKuVQOLLLTMzK8SBxMzMCnEgMTOzQhxIzMysEAcSMzMrxIHEzMwKcSAxM7NCHEjMzKwQBxIzMyvEgcTMzApxIDEzs0IcSMzMrBAHEjMzK8SBxMzMCnEgMTOzQhxIzMysEH/VrlmJBtYOcu5ND3Nn+v3wp/lbGa0HOZCYlejcmx7mstmLWL1mkPmLVyDg9EN2K7taZi3xpY9Zie5cuITVawYBWL1mkNkLl5RcI7PWOZCYlWjvSeMZOyb5Nxw7ZhTTJo0vuUZmrXNqy6xEpx20KwJmL1zCtEnjOfWgXcuuklnLHEjMSrTR6FGcfshunF52RcwKcGrLzMwKcSAxM7NCHEjMzKwQBxIzMyvEgcTMzApxIDEzs0IUEWXXoeMkPQU8OszdtwGebmN1RgKfs9b5nA2Pz1vrmj1nO0XEts084YgIJEVImhsRU8uuRy/xOWudz9nw+Ly1rhPnzKktMzMrxIHEzMwKcSAZ2sVlV6AH+Zy1zudseHzeWtf2c+Y+EjMzK8QtEjMzK8SBxMzMCnEgqSPpNEk3SvqDpJB0dov77yPpTknPSVos6VxJm3aoupUgaZSkMyU9Imm1pAckvbPJfWek5zm7nN/peneLpFdK+oGk5ZJWSLpG0o5N7jtW0n+m78fnJM2R9HedrnPZCp6zvPdTSJrS6XqXSdIOkr6Wvkf+nL7mCU3uO+z/4RoHkhc7GdgOuK7VHSW9FrgJ+BNwKPAvwAnAjDbWr4rOAc4Gvg78PXAX8H1Jb2ty/6eAvTLLee2vZvdJ2gy4BdgN+ADwfmAX4FZJmzfxFJeSvCc/S/Ke+gPw437+UGzDOYPkfy77nnq47ZWtlp2BdwHLgJ+1uG/R/2GICC/pAoxKHzcCAji7hX2vBX4LjKlbd1z6PK8v+7V16HxtBzwPfD6z/qfAg03sPwN4vOzX0cHz84/AWmDnunUTgQHgtCH23SN975xQt24j4DfADWW/tiqes7RsAF8o+3WUcN5G1f38wfQ8TGhiv0L/w7XFLZI6ETE4nP0kjQEOAa6KiDV1m64CXgCOaEP1quhgYGPgO5n13wFeI2li96tUKYcDd0XEgtqKiFgEzGbo98ThwBrgyrp9B4ArgIMlbdL+6lZCkXM2Yg33s4s2/Q87kLTHJGAs8FD9yohYDSwEdi+jUl0wmeRqZkFm/bz0sZnXvZ2kpyUNSHpY0hmSRre1luWZTOY9kZrH0OdmMrAoIv6cs+/GJKmMflTknNV8VNLzaV/BLZL2bV/1+k47/of9ne1tsnX6uCxn29K67f1ma+CZSNvCdZbWbd+Q+4F7SN60Y4F3AP9OkhP/YBvrWZatafyeGFdg39r2flTknEFyJf2/wJPATsDpwC2SDoqIWe2qZB8p+j8M9HEgkfQWks7vodwWEfsXPVz6mDe7UznrKmkY50wUeM0RkR2d9X+SVgGnSPpSRPy2meepuOGen0LntscVeU+9v+7Xn0m6nqSF8wVgnzbUrd+05X3Wt4EEuBN4dRPlsqmD4dhQ9B7HumZi1bV6zpYC4yQpc0Uzrm57q74HnAJMJRm80MuW0fg9kXfVXW8pkDfktci57QVFztl6ImKlpB8CJxWtWJ9qy/9w3waSNLc8v0uHW0iSZ5xcv1LSWOBVwPe7VI9ChnHO5gGbkPQR1edYa3nVXw2jGhtq3fWaeWTeE6ndGfrczAPeIWmzTD/J7iQDOLI57X5R5Jw10uiq29r0P+zO9jaIiBeAmcC7JNUH56NJ/kg3lFKxzptJ8qF2bGb9PwAPpaNtWvU+kn/6XxSsWxXcAOwp6VW1FekksWkM/Z64ARgDHFO370bAu4GfRMTz7a5sRRQ5Z+uRtCXwduDuNtWv37Tnf7js8c9VWkjSKUeTTOwJkuG7R6fLZnXlLgUGMvtOAZ4DrgHeTNKUXgp8v+zX1eFz9h/AauA0YH/gQmAQOCxT7qfAgrrfdwJuBz4GvBU4DLgs3ffCsl9Xm87N5iRXeb8kGbp6OPAA8Dtgi8y5GAA+m9n/CpJ0zgfT99QP0nPdl/OSip4z4NPAf5FcjOxPMqHxlyQflPuW/dq6cO5qn1UXpp9fH01/36+uzABwaWa/pv6HN3jssl98lRaSCXLRYJmQLZez/98Bc9I/yh+B8+sDUD8uwGiSWfyPkqT3HgSOzik3C3ik7vetSe4g8Gh6vp4D7gU+Qd3kql5fSPo5rgZWACvT1zwhU2YCORNggU2Bc4HF6Tm6G9i/7NdU1XNGcjEym+RrZNcAS0haMW8s+zV16bw1+uyalSkzI7NfU//DG1p8G3kzMyvEfSRmZlaIA4mZmRXiQGJmZoU4kJiZWSEOJGZmVogDiZmZFeJAYmZmhTiQmJlZIQ4kZmZWiAOJWRdIukTSvZl190gKSTvUrfuypEV99C2RNgI4kJh1xzLgJbVfJB0I/DXJvY/Gpes2A04EvhoRa8uopNlwOJCYdceLAgnwTyR3O16ME9jlAAAA+ElEQVTFui8ROo7kBnqXdrdqZsX07RdbmVXMMmBLAEmvJrl1/l8DR7EukHwCuCQiVpRSQ7NhciAx645lwKZp38c/AddHxO8kLSf5qtO3ALuRfAmTWU9xIDHrjtr3je9M8u1zB6S/ryBpkRwFXBMRj5ZQN7NCHEjMuqMWSM4C7o2IOenvy4G/JWmJTCujYmZFOZCYdUctkBxL8r3rNSuA9wB3R8RdXa+VWRt41JZZd9QCyWMkXx1bs5xkpNa5Xa+RWZv4q3bNzKwQt0jMzKwQBxIzMyvEgcTMzApxIDEzs0IcSMzMrBAHEjMzK8SBxMzMCnEgMTOzQv4//l4rLcmF/PkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f0(w, X, y, λ=10):\n",
    "    return np.sum((X@w-y)**2,axis=0) + λ*(w!=0)\n",
    "fw0 = f0(wgrid[None], X, y).ravel()\n",
    "minind = np.argmin(fw0);\n",
    "minw = wgrid[minind]\n",
    "plt.plot(wgrid, fw0, '.',markersize=7);\n",
    "# plt.plot((minw, minw),plt.ylim(),'--k');\n",
    "# plt.plot(minw, fw0[minind], 'r*',markersize=10);\n",
    "plt.xlabel(\"$w$\");\n",
    "plt.ylabel(\"$f(w)$\");\n",
    "plt.title(\"L0 regularization\"); # $f(w)=\\sum_{i=1}^n (wx_i-y_i)^2$\n",
    "print(\"minimizer: w=%.2f\" % minw);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- L0-regularized minimum is often exactly at the ‘discontinuity’ at 0:\n",
    "  - Sets the feature to exactly 0 (does feature selection), but is **non-convex**.\n",
    "  \n",
    "- We can fiddle with the code to see a case where we don't get $w=0$:\n",
    "  - By making the un-regularized solution further from 0.\n",
    "  - By decreasing $\\lambda$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L2 regularization\n",
    "\n",
    "$$f(w)=\\sum_{i=1}^n (wx_i-y_i)^2 + \\lambda \\|w\\|^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minimizer: w=-0.06\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAElCAYAAADQhFSEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8FHX+x/HXJ70AKYTeQu8dkQ6niBSxC+oPFBuoZ8F6h2K5k1NPsXt4chZUVMCCigUFKSJFaoK00EIHIQmElkKS7++PWXRdEkjZ3dnyeT4e+1gyO7Pz3mF3P/ud+c53xBiDUkop5WkhdgdQSikVHLTgKKWU8gotOEoppbxCC45SSimv0IKjlFLKK7TgKKWU8gotOEr5CBEZJSJGREb58zrKkGWKI0uy3VmUd2jBUV4hIsmOL5cvSjFvbxF5UUTWiMhhEckRkXUi8oSIxHgjr6o4p//zKXZnUb5BC47yRZ8AdwOHgXeAyUAY8CTws4jE2hfN780EWjru7TYOK8teu4Mo7wizO4BSxXgReM8Y89vpCSISBnwKXAb8FXjOpmx+zRiTDWTbnQPAGLMf2G93DuU92sJRPscY85xzsXFMKwD+7fizT2mex/l4hYhcISLLROSEiCxwmidERG4TkV9E5LjjtkRErizhOTuJyBzHfIdF5DMRaSAiC0TEuMxb4jGKshy/EJGbReQrEdkpInkikiEiX4pIl7K+5uKO4ZzOfpbbk07zXiAi74rIZsfzHnNsr+GuOYB0x583ujxf8tm2gYhUEpEJjnXkicghx3ZuW8zrPZ09XET+4bSN1ovIdefatsq7tIWj/Mkpx31BGZcbDlwIfAksOL28iAjwMTAM2AC855h/CPCZiIw1xrxy+klEpCPwExAJzAB2Ab2Bn4Gssr+cUvsPkAL8AGQCDYDLgYtFpJ8xZlkxyxT7mkswxTGPq2uAVsBJp2kPA42AX7B2hSUClwLTRKSWMeZlx3wpwCvAvUAq4Hzs7khJQUQkCpgHnOdYx2dAXaz/o4EicrEx5udiFp0GdAK+wfpeuw74SESOGGO+K2l9ysuMMXrTm8dvQDJggC8q8ByvOZ7jr6Wcf5Rj/gKgdzGPj3E8/h8g1Gl6LNaXXR5Q22n6Ysf8Q1ye53+O6cZl+hTH9ORi1n3GY055R7nM27CY5VsCx4C5ZXzNxa6jmPmGAoXACiD6HFlisYpKNhBTzP/5lBLWUdw2eMIx7W1AnKb3BYqALUCI0/QFjvmXAJVc5jfA93a/9/X2x013qSm/ICL9gTuANKwvo7KYaYxZVMz0v2J1TLjPGFN4eqIx5gTwFBABXOlYfzLQA1hqjPnG5XmepOytrlIzxqQXM20jMB/oLSIRxSxW0ms+JxFpB3yEdXzlMmNMzjmynMBqHVYBupZnnU5uBHKBR4yjcjjWsRD4CmgC9CxmuUeMMcdd5t8BnLHbUdlHd6kpn+fYdz8DOA4MM8bklvEpVhbznDFAG6zdYo9Ye9f+pJrjvoXjvr3jfonrjMaYvSKyC2tXk9uJSBPgEeAvQG2sQuisKmcefD/jNZdyXTWAWVjHdy81xuxzebwK1m61y7Ber2s39VrlWa/TczcE1hiXY3gOCxzr7QC4FtM1xcy/F6hf3jzK/bTgKJ8mIs2BOUA4MMAYs7YcT3OwmGkJgGAdD3niLMue7oJd2XF/6CzrcHvBEZGmwHLH+udidWc+jrV76XKsQhhZQp6yrisK61hLPeAaY8xql8cjgIVYX/irsHaJZWHteuuAVQyKy1JaVRz3xRUbgAMu8/3OWL3vXBWgHaN8ihYc5bNEpBnWbqPKwCBjzNJyPlVxVxk86rhfbIzpVYrnOOa4r1bC49WLmVbkuA8t5rEzvjRLMBaIB643xnzs/ICInM8fLS9X5bmy4jtAN2C8MeazYh4/3bqYbIwZ45Llb47HK+L0/0mNEh6v4TKf8jNa/ZVPcuxGmof1ZTvUGPOTO5/fGHMM2AS0EZFKpVgk1XHfw/UBEalD8btuTvfGquMyfwglFwpXjR33s1yeIxqrV5ZbiMjjWD27PjTG/KssWRyKO65y+rhYcQX3DMaYo1hdqVuKSHGFva/jPqU0z6d8jxYc5XNEpBFWyyYJuNwYM89Dq3oNiAMmOXYnueZoLSLVAYwxO4BlQHcRGeIy65MUv7dgleP+Bpfp91D63W+7HPe/f6E7unM/TfGtqjITkWuwXsNS4JayZHEsfyVWrzZXhx33dYp5rCTvA1HAP13W0QtrF+I2rN6Cyg/pLjXlbZ2k5LG1fjDGfITVsqmLdc5LDxFxbVUcMX+c71ERb2C1WEYC/URkHtZxglpAO6zdR93543jIXY5MM0XE+TycZGAt4Hpi4kxgJ3CLiNQFfgU6Op53IX/8Yj+b/wI3AZ+LyHSsrse9sQ6uLwD6le0lF+s9rONZqcC4YjpQLDDGLMBq2ewC/iYirbFaiK2BgViv9QrnhYwxx0VkBdBXRN7CKhYGeKOEYy5gndx7CXC7o7fcT1gFazhW77WbjTFFJSyrfJwWHOVt9bC6vhbnCFZ33AaOv/tQ/KgCO4EKFxxHt9sRIvIdcCvWMYgYrIPWG4E7sYrE6flXiUhf4FmsL9d8rJbYSKwv42Muz3/S0Z37ZazC0AOrSHQHHqUUBccYs1pEBmJ1077Gsc4FWLu/Hi3XCz9TtOP+9rPMs8AYc0xELgAmAr2wes2lAIOxivQVxSx3A9YJoNfwx3GraZQwvI4xJkdE/oI1ztow4AGs7foN8A9jTGpxyyn/IE5d3ZVS5eA4BnQQWGeMqeh5KEoFLD2Go1QpOcbrquoyLQRrN1A01jAySqkSaAtHqVISkSRgD9aYZpuxztHphXUC6SbgPOez3ZVSf6YFR6lScvRkewlrUMxaWCc57sYacuUpY4wnB/BUyu9pwVFKKeUV2kvNSVJSkklOTrY7hlJK+ZVVq1ZlGGNKGoXjd1pwnCQnJ7NyZbnGPFRKqaAlIjtLM5/2UlNKKeUVWnCUUkp5hRYcpZRSXqEFRymllFdowVFKKeUVWnCUUkp5hRYcpZRSXqEFRymllFdowVFKKeUVWnCU8gFz585l7ty5dsdQyqN0aBulfMCECRMA6N+/v81JlPIcbeEopZTyCi04SimlvEILjlJKKa/QgqOUUsortNOAUj7gzTfftDuCUh6nBUcpH9C8eXO7I6ggtXbPEX7ceJBbezekclS4R9elu9SU8gGzZs1i1qxZdsdQQei/C7fx7uJ0QkQ8vi5t4SjlA1544QUAhg4danMSFUx2Zp5g9roD3N63MbGRni8H2sJRSqkg9c7P6YSGCKN6JHtlfVpwlFIqCB0+kc+MlXu4vEMdqleJ8so6teAopVQQmrpsJzmnCrmtTyOvrVMLjlJKBZncU4W8t3QH/ZpXo1mNyl5br3YaUMoHfPDBB3ZHUEFk5pq9ZBzPZ7QXWzegBUcpn1CvXj27I6ggUVRk+N+i7bSpU4Xujap6dd26S00pHzB9+nSmT59udwwVBOZtOsj2Qye4rXcjxAvn3jjTFo5SPuCNN94AYPjw4TYnUYFu8qLt1ImPZnDbWl5ft7ZwlFIqSKTsPsLy9Cxu6plMeKj3v/614CilVJD474JtVIkK49qu9W1ZvxYcpZQKAlsPHuf7DQe4sUcylbwwjE1xtOAopVQQmPzTNiLDQrw2jE1xtNOAUj7g008/tTuCCmD7s3OYuWYv13etT9VKkbbl0IKjlA9ISkqyO4IKYG8vSqfIwK29vXuipyvdpaaUD5gyZQpTpkyxO4YKQEdO5vPR8l1c2r429RJjbM2iBUcpH6AFR3nK+0t3cjK/kDF97W3dgBYcpZQKWCfzC3h3cToXtqhOi5pV7I6jBUcppQLVjBW7OXzyFHf0a2x3FEALjlJKBaRThUX8b1E65yUn0CU50e44gBYcpZQKSF+m7GPvkRxu7+sbrRvQbtFK+YRvv/3W7ggqgBQWGSbN30rLWlW4oEV1u+P8Tls4SvmAmJgYYmLs7bKqAse3v+5ne8YJ7r6gidcvQXA2WnCU8gGTJk1i0qRJdsdQAaCoyPCf+VtpUr0SA1vXtDvOn2jBUcoHzJgxgxkzZtgdQwWAHzcdZNOBY/z1L40JCfGd1g1owVFKqYBhjOH1eVuonxjD0Ha17Y5zBi04SikVIBZtySB1TzZ39mtMmA0XWDsX30uklFKqXF6ft5VacVFc2amu3VGKZWvBEZG6IvKaiCwVkZMiYkQkuZj5TAm3Di7zhYjIOBHZISK5IpIqIld56/UopZRdftmeyfIdWYzp04iIMN9sS9h9Hk4TYBiwClgEDDjLvFOAN12mbXb5+yngQeBRx3NeC3wiIpcYY/REB+WzFixYYHcE5eden7+VpEoRtl0+ujTsLjg/GWNqAIjIrZy94Ow1xiwr6UERqY5VbJ41xkx0TJ4vIk2AZwEtOEqpgLRq52EWbclg3KAWRIWH2h2nRLa2u4wxRW58uouBCGCqy/SpQFsRaejGdSnlVhMnTmTixInnnlGpYrzy4xYSYyMY2b2B3VHOyjd39BXvDhHJcxzrmScivV0ebw3kAVtdpq933LfyeEKlyunrr7/m66+/tjuG8kOrdh7mp82HGNOnETERdu+0Ojt/KThTgTuB/sBooCowT0T6Oc2TCBwxxhiXZbOcHj+DiIwWkZUisvLQoUPuTa2UUh7mL60b8JOCY4wZaYyZboxZZIyZCvQC9gETnGYTwLXYnJ5+tueebIzpYozpUq1atXLlO5FXwMGjueVaVimlysufWjfgJwXHlTHmGPANcJ7T5CwgQc4cqS7B6XG3KywyDHl1EU/OWn/umZVSyo38qXUDflpwHFxbNOuBSMD14g+nj91s8ESI0BDh0g51+PbXA6zfl+2JVaggEB0dTXR0tN0xlB/xt9YN+GnBEZEqwBDgF6fJs4F84P9cZh8BrDPGpHsqzy29GlI5KoyX527x1CpUgPvuu+/47rvv7I6h/Ii/tW7A/vNwEJGrHf/s7LgfJCKHgEPGmIUi8iDQHJiPddymAdb5NjVxKi7GmIMi8hIwTkSOAauB4cAFwGWefA1x0eHc1rsRL87ZzK97smlbN86Tq1NKBbnTrZtxg1r4TesGfKDgAJ+4/H36oiALgX5AGnCF4xYHHAUWA7cYY5a7LPsocBy4F6sgpQHDjDGzPJLcyU09k3lncTovzknj3Zu6enp1KsA89dRTADz22GM2J1H+4OW5m/2udQM+sEvNGCMl3Po5Hp9ljOlpjEkyxoQbY6oaYy4tpthgjCk0xkwwxjQwxkQaY9oZYz71xuuoHBXO6D6NmJ92iNW7DntjlSqA/Pjjj/z44492x1B+YHl6Fou2ZHB7X/85dnOa7QUnkNzYPZmqsRG8NMd1iDellKo4YwwTv0+jeuVIRnZLtjtOmWnBcaPYyDBu79uYRVsyWJ7ukV7YSqkgtmhLBst3ZHHXBU2IjvDdMdNKogXHzUZ0a0BSpUhenJNmdxSlVAAxxvDCD2nUiY9m+Hn17I5TLlpw3Cw6IpQ7+zVm2fYslmzNsDuO8hNVq1alatWqdsdQPmzOht9I3ZPNvRc2JTLM/1o3oAXHI64/vz41q0Tx/A9pnDm0m1Jn+uyzz/jss8/sjqF8VFGR4cU5m2mYFMuVnerYHafctOB4QFR4KPf2b8qaXUeYu/Gg3XGUUn7um1/3s+nAMcb2b0pYqP9+bftvch93Tee6NEyKZeL3aRQWaStHnd24ceMYN26c3TGUDyooLOKlOZtpXqMyQ9vVtjtOhWjB8ZCw0BDuv6gZab8d46vUvXbHUT5u6dKlLF261O4YygfNXLOX7RknuO+iZoSEnHXwe5+nBceDhrStRataVXhxzmbyC9x5cVOlVDDIPVXIy3O30LZOHBe3rmF3nArTguNBISHCQwObszsrh2krdtkdRynlZ6Yu28neIzn8fVALzrzyiv/RguNh/ZpVo2tyIq/+uJWT+QV2x1FK+Ymjuaf4z/yt9G6aRM8mSXbHcQstOB4mIjw8sDkZx/N4d/EOu+MoH1W3bl3q1q1rdwzlQyYv3M7hk6f428AWdkdxG/8a+c1PdUlO5IIW1Xlz4TZGnN+AuJhwuyMpHzN16lS7IygfcvBoLm//nM4l7WrRpk7gXO5EWzhe8tDFzTmWV8CkBVvtjqKU8nGvztvCqcIiHhzQ3O4obqUFx0ta1qrCFR3r8O6SHew5fNLuOMrHjB07lrFjx9odQ/mAHRknmLZ8N9d1rU9yUqzdcdxKC44XPeD4tfLiD3r5AvVnKSkppKSk2B1D+YCJP6QRHhrC3Rc2sTuK22nB8aI68dHc1DOZmSl7Wbc32+44Sikf8+uebL5eu59bezekeuUou+O4nRYcL7uzXxPiosP59+xNdkdRSvkQYwz/+nYDibER3Nankd1xPEILjpfFRYdz9wVNWbQlg582H7I7jlLKR8zbdJBl27MY278pVaICsyerFhwbjOhWn3qJ0Tzz3SYd2FMB0KxZM5o1a2Z3DGWTgsIinv52I42SYrmua32743iMFhwbRIaF8tDFLdi4/yhfrNGBPRVMnjyZyZMn2x1D2eTjFbvZdugEfx/UgnA/vvzAuQTuK/Nxl7StRbu6cbzwQxq5pwrtjqOUssmx3FO8PGczXRsmclEr/x+g82y04NgkJEQYN6gl+7KtM4pVcBs9ejSjR4+2O4aywX8XbiPzRD7jh7QMiAE6z0YLjo26N67KgFY1mDR/KweP5todR9lo8+bNbN6s52cFm31HcnhrUTqXd6hNu7rxdsfxOC04NntkcEvyC4uY+EOa3VGUUl428fs0DPDgxYE1hE1JtODYLDkpllE9kvlk1R49GVSpIPLrnmw+X7OXW3o1pG5CjN1xvEILjg+464KmJMRE8NTXGzBGu0krFeiMMfxj1nqSKkVwR7/GdsfxGi04PiAuOpz7LmrGL+lZfL/+N7vjKBt06NCBDh062B1DecmstftZufMwDw5oHrAneRZH9Bf1H7p06WJWrlxpy7oLCosY/Ooi8gqK+OG+PkSGhdqSQynlWTn5hVzwwgISYyP46q5ehIb4f880EVlljOlyrvm0heMjwkJDeHRIK3ZmnuT9JTvtjqOU8pD/LtzG/uxcnhjaOiCKTVlowfEhfZtV4y/Nq/Hqj1vIOJ5ndxzlRSNGjGDEiBF2x1AetvdIDv9duI1L2tWia8NEu+N4nRYcHzP+klbkFhTynI4mHVT27NnDnj177I6hPOyZbzcCMG5wS5uT2EMLjo9pXK0SN/dsyIyVe0jZfcTuOEopN1mensXXa/dze9/G1ImPtjuOLbTg+KC7L2xK9cqRPPHlOop0NGml/F5hkeGfX6+nVlwUt/cNnm7QrrTg+KBKkWGMG9yC1D3ZfLJqt91xlFIVNG3FLtbtPcrfB7UgOiJ4e6BqwfFRl3eoQ5cGCTw3O43snFN2x1Ee1r17d7p37253DOUBWSfyeW52Gt0aJXJp+9p2x7FVWFlmFpFuwECgG1AbiAYygDRgIfCFMeawu0MGIxHhyUtbM/T1n3lpzmaevLS13ZGUBz3zzDN2R1Ae8tzsTZzIK+Cfl7UJ+NGgz6VULRwRuVFEfgWWAGOBGGAL8AtwGDgfeAvYKyJTRKShh/IGlTZ14ri+a30+WLaTtAPH7I6jlCqj1bsOM23Fbm7qmUyzGpXtjmO7cxYcEUkFngW+BToDCcaYPsaYq4wxI4wxg40xLYFE4DagOrBeRIZ7MniweHBAcypHhfH4l+t0nLUAdtVVV3HVVVfZHUO5UWGR4fEv11GjSiT39tfLh0PpWjjvAg2NMX8zxqwxJXzrGWOyjTEfGmMGA90B7dPrBgmxETx0cXN+Sc9ipl6OOmBlZmaSmZlpdwzlRh8vtzoKPDqkFZUiy3T0ImCds+AYY142xuQCSCl3QBpjUo0x31c0nLJcd159OtaP51/fbCT7pHYgUMrXZR7P4/nv0+jeqCpD29WyO47PKGsvta0ico+IVPJIGlWskBBhwuVtOHwyn39/ryMQKOXrnpudxom8Av5xWeug7yjgrKwFZybwBLBbRJ4TkXoeyKSK0bp2HDf1bMjHy3exepd2BFTKV63YkcX0lbu5uVdD7SjgokwFxxjzIFAXGAcMAraJyDQROc8T4dSf3XdRM2pUjuLRmesoKCyyO45yowsvvJALL7zQ7hiqgvILinjk81+pEx/NvRc2tTuOzynziZ/GmBxjzH+NMW2xik40sEREFonIFW5PqH5XKTKMJ4a2YuP+o7y3VC9hEEgee+wxHnvsMbtjqAr636LtbDl4nH9e1ppY7ShwhrKe+FkbSHC5zQOOAtcCnwLBO26DFwxsU5N+zavx4g9pDG5bk1pxwTkIoFK+ZkfGCV79cQuD29bkwpY17I7jk8rawtkDpAJfA5OAh4HhQBLwOfB2WZ5MROqKyGsislREToqIEZHkYuaLEpHnRWS/iOQ45u9TzHwhIjJORHaISK6IpIpIQJ3cICL889I2FBQZ/vHVBrvjKDcZNGgQgwYNsjuGKidjDI99uY7w0BCeGKqjgpSkrAVnDZADTAF6G2PaGGN6GGMGGWOGG2NGl/H5mgDDsEYrWHSW+d7GOqn0ceASYD/wvYi4XgT+KeBJ4HWs3X3LgE9EZHAZc/m0+lVjuLd/U2avP8DsdQfsjqPcICcnh5ycHLtjqHL6KnUfi7Zk8PDA5tSoEmV3HJ9V1k4DnYFLsQpFmohMF5GuFVj/T8aYGo6TRT8pbgYRaQ9cD9xnjPmfMeZHrCK1C/in03zVgQeBZ40xE40x840xY4D5WCMlBJTbejeiZa0qPP7lOh3cUykbHTmZz1Nfb6BDvXj+7/wGdsfxaeXpNDDfGHMJ0AE4BiwUkZ9F5PLSnhjq9Fyl6Wp1KXAKmO60XAEwDbhYRCIdky8GIoCpLstPBdoG2vhu4aEh/PuqtmQcz+PZ7/TcHKXs8ux3mzh88hRPX9GW0BA95+Zsyn15AmPMRqzdXO2BrcAMYLObcjlrDaQbY066TF+PVWCaOM2X58jiOh9AKw9ks1W7uvHc2rsRHy/fxbLtOiyKUt62eGsG01bs5tbeDWlVu4rdcXxeWXuprQIqAZWBKlhdogFOl/VG7ov2u0SsYzyuspweP31/pJix3lzn+xMRGQ2MBqhfv37Fktrgvv7NmL3uAOM+/5Xv7u1NVLh2EvRHl1xyid0RVBmdzC/g75+vpWFSLPfp4JylUtaO4quxvvzPdnM3AYobMNS17Vra+f7EGDMZmAzQpUsXvxuOOToilKevaMuIt3/htXlbeOjiFnZHUuXw4IMP2h1BldHz36exOyuHGWO66w+9UipTwTHG3OapIGeRBRTX9Ehwevz0fYKIiEsrx3W+gNOraRJXd67Lmwu3M6RtbW3aK+VhK3dkMWXJDm7o3oCuDYvdeaKK4Q+XmF4PNBSRGJfprYB8/jhmsx6IBBoXMx9AQJ+0Mn5IS+JjInjo01RO6bA3fqdfv37069fP7hiqFHJPFfLwZ2upHRfNwwN1j0JZlOYCbF+KSMfSPqHjJM37ReT2ikX73VdAOHCN0zrCsE44/cEYk+eYPBurAP2fy/IjgHXGmHQ35fFJ8TERTLi8Dev3HWXS/G12x1EqYL364xa2HzrBM1e21evclFFpttZOYJmIpAAfYZ2gudbRNRn4fcibrsBQ4EpgL3BzaQKIyNWOf3Z23A8SkUPAIWPMQmNMiohMB14WkXAgHbgDaIhTcTHGHBSRl4BxInIM63jTcOAC4LLSZPF3A9vU5PIOtXlt3hb6t6pO69pxdkdSKqCs25vNmz9t55rOdenTrJrdcfxOaQpOPvAX4DqsSxPEAUZEjmJ1Q07AaoEIsBwYC3xQynNs4MwTPic57hcC/Rz/vgn4FzABiMcaXmegMWa1y7KPAseBe4GaQBowzBgzq5RZ/N6Tl7Zm8bZMHpiRyld39SIizB/2mirl+/IKCnlgRipVYyMYPyTgzrLwitIUnLHADGPM3Y6Ww2ysS0jXAqKATGAT1qgBZR7C2BhzzjOljDE5wP2O29nmK8QqShPKmiNQxMdE8PQVbbnt/ZW8Pn8r91+k3TWVcoeX5mwh7bdjvDvqPOJiwu2O45dKU3Cy+KOn19+AL4wx//ZcJFVRF7WqwZUd6/Cf+VsZ0KoGberorjVfN2zYMLsjqLNYtTOLyT9t49rz6vGXFtXtjuO3SlNwfgYmikg1Sj7XRfmYJ4a25uetGdautbt7Ehmm5wn4sjvvvNPuCKoEJ/IKuH9GKrXjoxl/ie5Kq4jS7OC/CzgAvIdVbOY6Lrb2qojcJCIdHAfzlQ+Jiwnn2avakvbbMV6c44kRh5Q7nTx5kpMnXUdvUr7gme82sivrJBOvaa+90ironAXHGLPPGHMRUAerhTMd6/IAA4G3gFXAMRFZLSJluh6O8qwLWtTguq71mPzTdh1rzccNHjyYwYMD6ioaAeGnzYeYumwXN/dsSLdGVe2O4/dK3YXJGHMAmAm8ZIwZZoxphtVjrS/wEJACdPJISlVu44e0okFiDA/MSOVorl7GQKnSyj55ioc/XUuT6pV46OLmdscJCGW9Hs5VjlGiT/993BjzszHmNWPMzcaYUp8gqrwjNjKMl4Z34MDRXJ74cv25F1BKYYxh/JfrOHQ8jxeHtdex0txET9IIAh3rJ3D3BU2YuWYvs1L32R1HKZ/3+Wrrs3Jf/6a0qxtvd5yAoQUnSNz1lyZ0qBfPozN/ZX+2XspYqZLszDzB41+uo2vDRO7o1+TcC6hS04ITJMJCQ3hpeAcKigwPzEilqEh7t/uSUaNGMWrUKLtjBL1ThUXcMy2F0BDhpeEd9AqebqYFJ4g0TIrl8UtasWRbJm/+tN3uOMqJFhzf8MrcLaTuPsLTV7alTnz0uRdQZaIFJ8gMP68eQ9rWYuIPaaza6Ynr5anyyMjIICMjw+4YQW3Z9kz+s2Ar13SuyyXtatsdJyBpwQkyIsIzV7WlVlwU93y8huyT2lXaF1x99dVcffXV555ReUT2yVPcPz2FBokxPHlpa7vjBCwtOEGoSlQ4r1/fid+O5vK3z9by5wukKhVcjDE8+Gkqh47n8cq1HYnV0QQ8RgtOkOpQL56HBzYIRQibAAAYzUlEQVRn9voDTP1ll91xlLLN2z+nM2fDb/x9UEva19Mu0J6kBSeI3dqrEf2aV+OprzewYd9Ru+Mo5XWrdx3m2e82MaBVDW7umWx3nICnBSeIhYQIL1zTnvjocO76eDXH8wrOvZBSAeLIyXzu/mgNNeOieP7q9ohoF2hP04IT5KpWiuSVazuyI+OEHs+x0R133MEdd9xhd4ygUeQ4H+3gsVz+c30nvaCal+jRMUX3xlV5eGALnv1uE53qJ3BLr4Z2Rwo6w4cPtztCUPnfou38uOkgTw5tpcdtvEhbOAqAMX0aMaBVDZ75diMrdmTZHSfo7N69m927d9sdIyj8sj2T575PY1CbmtzYI9nuOEFFC44CrPNzJg5rT92EaP764WoOHcuzO1JQGTlyJCNHjrQ7RsA7kJ3LXz9aTYPEGP59dTs9buNlWnDU76pEhfPGiM4czT3F3R+vpqCwyO5ISrlNXkEht09dRU5+IZNv6EyVKD1u421acNSftKxVhaevaMuy7Vk8/32a3XGUcpsnv9pAyu4jvDCsPU2qV7Y7TlDSgqPOcGWnuozoVp83f9rOlyl77Y6jVIVNW76Lj5fv4s5+jRnYppbdcYKWFhxVrMcvaU3X5EQe/nQta/ccsTuOUuWWsvsIj3+5nt5Nk3hggF4q2k5acFSxIsJCmDSiE0mVIhn9/ioOHsu1O1JAe+CBB3jggQfsjhFwDh7N5fYPVlG9SiSvXttRr29jMy04qkRJlSKZfENnsnNOcfsHq8grKLQ7UsAaOnQoQ4cOtTtGQMk9Vcht76/kaO4pJo/sQkJshN2Rgp4WHHVWrWvH8cKw9qzedYTxM9fpSAQekpaWRlqadtJwF2MMD36Sytq92bxybUda1a5idySFFhxVCoPb1uKeC5rwyao9vP1zut1xAtKYMWMYM2aM3TECxis/buHrtfv5+8AWXNSqht1xlIMObaNKZWz/ZqT9dox/fbuR+okxDGhd0+5IShVrVuo+Xp67hWs612V0n0Z2x1FOtIWjSiUkRHh5eEfa1YnjnmlrSN2tPdeU70nZfYQHP0mla3IiE65ooyMJ+BgtOKrUoiNCeevG80iqFMkt761kd9ZJuyMp9btdmSe59b0VVK8SyRsjOhEZFmp3JOVCC44qk2qVI5ly03nkFxRy85QVZOecsjuSUmSdyOfGd5dTUGR4d1RXqlaKtDuSKoYWHFVmTapX5r8jO7Mj8wR3TF1FfoGOuVZR48ePZ/z48XbH8Es5+YXc8t4K9h3J4a0butCkeiW7I6kSaMFR5dKjcRLPXtmOJdsyefjTVIqKtLt0RfTv35/+/fvbHcPvFBYZ7pm2hpTdR3jl2g50SU60O5I6C+2lpsrtqs51OXA0l+e/TyMhNoLHL2mlB2nLKSUlBYAOHTrYnMR/GGN44qt1zNnwG08ObaVjpPkBLTiqQu7s15jM4/m8szidqrER3HVBU7sj+aWxY8cCsGDBAnuD+JHX521l6rJdjOnTiFE99Sq1/kALjqoQEWH8kJYcPpnPxB82kxAbwf+d38DuWCrATVmczgtzNnN5h9r8bWALu+OoUtKCoyosJER47up2ZOecYvwX60iIiWBwW929oTzj01V7eHLWBi5qVYPnr2lPiA7I6Te004Byi/DQEP5zfSc6109g7LQUFm4+ZHckFYC++3U/D3+aSq8mSbx2XUfCQ/UrzJ/o/5Zym+iIUN6+8TwaV6/E6PdXsmRrht2RVABZuPkQ90xbQ4d68Uy+oTNR4Xpip7/RgqPcKi4mnA9vPZ/kqrHc8t5KftmeaXckv/D000/z9NNP2x3DZ/2yPZMxH6ykafXKvHtTV2Ii9GiAP9KCo9wuMTaCD287nzoJ0dw0ZQUrd2TZHcnn9ejRgx49etgdwyct257JTVNWUCc+mvdv6UpcdLjdkVQ5acFRHpFUKZKPbj2fmlWiGPXuCtbsOmx3JJ+2ZMkSlixZYncMn7NkWwY3vbuC2vHRfDy6G0k6ZI1f04KjPKZ6lSg+uq0bVStFcMM7y1mtRadEjzzyCI888ojdMXzKkq0Z3DxlBXUTovn4tm5UrxxldyRVQVpwlEfVjLOKTmJsBCPe+oWl2/SYjjq3n7dkcNOUFdRPjOHj0d2oVllbNoFAC47yuDrx0Xwypjt14qMZ9e5y5qcdtDuS8mHz0w5yy3sraJgUy8e36W60QOIXBUdE+omIKeZ2xGW+BBF5S0QyROSEiMwVkbZ25VZ/qF4limmju9HE0WV69rr9dkdSPujLlL3c9t5KmlSvxIe3nq+XGQgwflFwnNwDdHe6/T68rlijRn4FDATuBq4CwoH5IlLX+1GVq6qVIvnotm60rRPHXz9aw8w1e+yOpHzIB0t3MHZ6Cp0aJPDx6G5abAKQv3Vm32iMWVbCY5cCvYALjDHzAURkKZAOPIxVrJTN4qLD+eCW87n1vZXcNz2VzOP53Npbrzv/8ssv2x3BNsYYXp+3lRfmbKZ/yxq8fn1HPakzQPlbC+dsLgX2nS42AMaYbGAWcJltqdQZYiPDePem8xjUpiYTvtnIP2dtCPrr6XTo0CEoL01QVGR46uuNvDBnM1d2qsN/R3TSYhPA/K3gfCgihSKSKSIfiUh9p8daA+uKWWY9UF9E9DKAPiQqPJTXr+/EqB7JvLM4nbs/XkPuqUK7Y9lm7ty5zJ071+4YXpV7qpC/frSadxanc1PPZCZe3Z4wHRstoPnLLrVs4AVgIXAU6Ag8AiwVkY7GmINAIrCjmGVPn+aeABx3fVBERgOjAerXr+/6sPKg0BDhiaGtqBMfzb++3cih43n8b2QX4mKC70zyCRMmAATNVT8PHcvjtvdXkrrnCI9d0oqbeybrxfuCgF/8nDDGrDHGPGiMmWWMWWiMeRmrc0AN/jg2I0Bx+2XO+i42xkw2xnQxxnSpVq2ae4OrcxIRbuvTiFev68iaXYe58o3FpGecsDuW8qAtvx3jikmLSTtwjDdHdOaWXg212AQJvyg4xTHGrAY2A+c5JmVhtXJcJTju9TR3H3Zp+9p8cMv5ZJ3I57LXf2bRFr28QSBavDWDK99YQl5BEdPHdGNA65p2R1Je5LcFx8G5VbMe6ziOq1bALmPMGbvTlG/p1qgqX93Vi1px0Yx6dwXvLk7HmODuTBAojDG8tWg7N7yznNpx0Xzx1560qxtvdyzlZX5bcESkC9AM+MUx6Sugjoj0dZqnCjDU8ZjyA/USY/jszh5c0KI6/5i1gb9/9it5BcHbmSAQnMwv4N5pKUz4ZiP9W1bn0zusUSdU8BF/+AUpIh9inU+zGjiC1WlgHHAS6GSMyRCREOBnoB7wENYutHFAO6C9MWb3udbTpUsXs3LlSs+8CFUmRUWGl+Zu5rV5W+lQL57Xr+9I3YQYu2N5TFpaGgDNmze3OYl77cw8wZgPVpH22zEeHNCcO/s11uM1AUhEVhljupxzPj8pOOOA64AGQAxwAPgOeMIYs99pvkRgInA5EAUsBe43xqSWZj1acHzPt7/u5+FP1xIaIrw4rD0XtqxhdyRVSvM2/cbYaSmICK9e15G+zbRTTqAKqILjLVpwfNOOjBPc+eFqNuw/yug+jXjo4uYBdy37WbNmATB06FCbk1RcXkEh//4ujXcWp9OyVhXeHNGZ+lUDt3WqtOCUixYc35V7qpAJ32xg6rJddG6QwKvXdQyo4wD9+vUDYMGCBbbmqKhth45z90dr2LD/KDd2b8C4wS115IAgUNqCE1g/E1XAigoPZcLlbXn1uo5s2n+UgS/9xKer9mgvNh9hjGH6il1c8urP7M/O4X83dOEfl7XRYqP+xF9GGlAKsM7X6VgvngdmpPLgJ6n8sP4AT1/ZVq+ZYqODx3J5/Iv1zF5/gB6Nq/LisA7UjNOrc6ozaQtH+Z16jqtAPjq4JQvSDnHxSz8xe90Bu2MFHWMMM9fsYcBLPzEv7SB/H9SCD245X4uNKpG2cJRfCg2xhsTp27wa989I4fapqxjSthaPD21FjSr6hedp+7NzeHTmOuZtOkin+vE8d3V7mlTX8XHV2WmnASfaacA/nSos4s2F23ht3lbCQ0N4cEAzRnZPJjTEf8732L3bOk2sXr16Nic5u4LCIj5avovnZ6dxqqiIhy5uwage/rWtlftpL7Vy0ILj33ZmnmD8F+tYtCWDtnXiePqKtrStG2d3rIDxy/ZMnvhqPZsOHKNnk6o8fUVbGlSNtTuW8gFacMpBC47/M8bw9dr9/PPrDWQcz+OqTnV5YEAzasX5dhfq6dOnAzB8+HCbk5xpf3YOT3+7iVmp+6gTH834IS0Z2KamjhigfqcFpxy04ASO7JxTvD5vC+8t2UlICNzaqxFj+jaicpRvXmvHF8/Dyc45xVuLtvPWonQKjeH2vo25o29joiO0q7P6s9IWHO00oAJSXHQ4jw5pxQ3dk3n++zRen7+Vj5fv4p4LmzL8vHp6fshZnMwvYMqSHby5cDvZOacY0q4Wfx/YgnqJOlqAqhgtOCqg1UuM4dXrOnJLr4Y8/e1GnvhqPf+Zv5XRfRpx/fn1iYnQj8BpOfmFTF+xi9fnbyPjeB4XtKjOAwOa0bq2HgdT7qGfNhUU2teLZ9robizdlsmr87Yw4ZuNvLFgG7f0bsjIbg18dlebN2Qez+P9pTv5YNlOsk7kc37DRN4c2YnODYq7nqFS5acFRwUNEaFHkyR6NEli5Y4sXp+/ledmpzFp/jau7FSHkd0a0LRGZbtjes32Q8d5++d0Pl21h7yCIvq3rM7oPo05LzlBOwQoj9BOA06000Dw+XVPNu8uTufrtfvJLyyiW6NEbuiezEWtanh1ROqMjAwAkpKSPLqe3FOFfLduP9OW7+aX9CwiQkO4slMdbu3dkCbVg6fYKvfSXmrloAUneGUez2P6yt18uGwXe4/kkBgbwZC2tbisQ2061U8gxI9PbCwqMqTuOcJnq/fwZco+juUW0KBqDMO61OOaLnWpXllHZlAVowWnHLTgqMIiw/xNB5mZspcfN/5G7qki6sRHM7R9bQa0rkH7uvEeOat+ypQpAIwaNcotz1dYZFiensX36w/w/foD7M/OJSIshMFtajLsvHp0a1jVr4uo8i1acMpBC45ydjyvgDkbDvBlyj4WbcmgsMgQHxNO76bV6NesGr2bJbmtdVDR83CMMezOymHp9gyWbstk0ZYMMk/kExkWQp9m1RjYuib9W9YgLiZ4O0coz9HzcJSqoEqRYVzRsS5XdKzLkZP5LNqSwYK0QyzcfIhZqfsASK4aQ4d68datfgKtalUhIszzx35O5BWw6cBRNuw7SsrubJZtz2TvkRwAkipF0LNJEgPb1KRvs2rERurHXPkGfScqVQrxMREMbV+boe1rU1Rk2HjgKIu2ZLBm12GWbs/kixSrAIWFCPWrxtAoKZaGSbE0TKpEvcRoqsZGUrVSBAkxEaUqSAWFRRzPK2B/di57D+ewLzuHvYdz2JV1kk0HjrEj8wSnd04kxIRzfsOqjO7TiO6Nq9K0eiXtZaZ8khYcpcooJERoXTvuTydE7s/OYc2uI6zbm016xgnSM06waEsGeQVFZyxfOTKMSlFhhIYIYSFCSIiwds8Rigx0mTCH43kF5J46c7mI0BDqJETTvEZlruhYh1a1qtCydhVqx0VpgVF+QQuOUm5QKy6aWm2jGdy21u/TiooM+4/msifrJFkn8sk8kU+W43Yir4BCYygsMhQUGTaFhxIiwoDWNakUGUalyDBiI8OoUSWSOvHR1EmIJik2Ug/0K7+mBUcpDwkJEatYxJ97pOrnL18EQEyMjlemApcWHKV8gBYaFQy8dyq1UqpEkyZNYtKkSXbHUMqjtOAo5QNmzJjBjBkz7I6hlEdpwVFKKeUVWnCUUkp5hRYcpZRSXqEFRymllFdot2ilfEB5B+1Uyp9oC0cppZRXaMFRSinlFVpwlFJKeYUWHKWUUl6hBUcppZRXaMFRSinlFVpwlFJKeYUWHKWUUl6hBUcppZRXiDHG7gw+Q0QOATvLuXgSkOHGOMFAt1n56HYrO91mZVeWbdbAGFPtXDNpwXETEVlpjOlidw5/otusfHS7lZ1us7LzxDbTXWpKKaW8QguOUkopr9CC4z6T7Q7gh3SblY9ut7LTbVZ2bt9megxHKaWUV2gLRymllFdowVFKKeUVWnDKQUTuF5FZIrJfRIyIPFnG5XuJyBIRyRGRAyLyoohEeyiuzxCREBEZJyI7RCRXRFJF5KpSLjvFsa1dby97Orc3iEg9EflURLJF5KiIfC4i9Uu5bJSIPO94P+aIyFIR6ePpzHar4DYr7r1kRKSDp3PbSUTqishrjvfIScdrTi7lsuX+/J6mBad8bgOqA1+UdUERaQfMAQ4ClwDjgZuAKW7M56ueAp4EXgcGAcuAT0RkcCmXPwR0d7m95P6Y3iUiMcA8oAVwIzASaArMF5HYUjzF21jvycex3lP7ge8D+cvTDdsMrM+c6/tps9vD+pYmwDDgMLCojMtW9PMLxhi9lfEGhDjuwwADPFmGZWcCW4Bwp2k3OJ6nk92vzYPbrDqQB/zDZfqPwNpSLD8F2GP36/DQtrkXKASaOE1rCBQA959j2faO985NTtPCgDTgK7tfmy9uM8e8Bphg9+uwYbuFOP37Vsd2SC7FchX6/J6+aQunHIwxReVZTkTCgYHADGPMKaeHZgD5wGVuiOerLgYigKku06cCbUWkofcj+YxLgWXGmK2nJxhj0oHFnPs9cSlwCpjutGwBMA24WEQi3R/XJ1RkmwWt8n534abPrxYc72oMRAHrnCcaY3KBbUArO0J5SWusX0hbXaavd9yX5rVXF5EMESkQkc0i8jcRCXVrSnu0xuU94bCec2+X1kC6MeZkMctGYO1CCUQV2Wan3SEieY5jGfNEpLf74gUcd3x+CXNnInVOiY77w8U8luX0eCBKBI4YRzvcSZbT42eTAqzCeoNHAVcAz2Dtt7/VjTntkEjJ74mECix7+vFAVJFtBtYv86+BfUAD4CFgnohcZIxZ4K6QAaSin19ACw4i0h/rIP65LDTG9Kvo6hz3xZ1tK8VM81nl2G5CBV63Mca1N9q3InIcGCsi/zbGbCnN8/iw8m6bCm1XP1eR99NIpz8XiciXWC2mCUAvN2QLNG55nwV9wQGWAC1LMZ/rLovyONuvgQT+aJ76g7JutywgQUTE5VdSgtPjZfUxMBbogtURw18dpuT3RHG/4p1lAcV1Ba7IdvUHFdlmZzDGHBORb4BbKhosQLnl8xv0Bcex73uTl1a3DWs/aGvniSISBTQCPvFSjgorx3ZbD0RiHcdy3g98et/vhnLEOFuL0Z+sx+U94dCKc2+X9cAVIhLjchynFVZHFNd97oGiItusJCX9ildu+vxqpwEvMsbkA7OBYSLiXOyvxvrP/MqWYN4xG+sL8P9cpo8A1jl6GJXV9VhfECsqmM1uXwHdRKTR6QmOk/F6cu73xFdAOHCN07JhwHDgB2NMnrvD+oiKbLMziEgVYAjwi5vyBRr3fH7t7hfujzesXThXY51AZbC6NV/tuMU4zfc2UOCybAcgB/gcuBCrCZ8FfGL36/LCdnsWyAXuB/oBbwBFwFCX+X4Etjr93QD4CbgTGAAMBd5xLPuG3a/LDdslFutX469YXXovBVKB7UAll+1QADzusvw0rN1ItzreU586tnMgn9dV7m0GPAj8D+sHSz+sE0d/xfpC7W33a/PCtjv9XfWG4/vrDsfffZ3mKQDedlmuVJ/fs67b7hfvjzeskxBNCbdk1/mKWb4PsNTxn/cb8LJzoQrUGxCKNbLCTqxdi2uBq4uZbwGww+nvRKxRHXY6tlkOsBq4C6cT2fz5hnUc5jPgKHDM8XqTXeZJppgTjYFo4EXggGP7/AL0s/s1+eo2w/rBshjr8smngEysVlFXu1+Tl7ZbSd9dC1zmmeKyXKk+v2e76eUJlFJKeYUew1FKKeUVWnCUUkp5hRYcpZRSXqEFRymllFdowVFKKeUVWnCUUkp5hRYcpZRSXqEFRymllFdowVFKKeUVWnCU8jEi8paIrHaZtkpEjIjUdZo2UUTSA+SqpyoIaMFRyvccBiqf/kNELgCaY41vleCYFgPcDLxqjCm0I6RSZaUFRynf86eCAzyANTr2cf644NUNWIMpvu3daEqVX9BfgE0pH3QYqAIgIi2xLsnQHLiSPwrOXcBbxpijtiRUqhy04Cjlew4D0Y5jMw8AXxpjtotINtZlfvsDLbAuGKaU39CCo5TvOey4b4J1RcW/OP4+itXCuRL43Biz04ZsSpWbFhylfM/pgvMIsNoYs9TxdzbQGatl09OOYEpVhBYcpXzP6YLzf8Bwp+lHgWuBX4wxy7yeSqkK0l5qSvme0wVnF9Zlk0/LxuqZ9qLXEynlBnqJaaWUUl6hLRyllFJeoQVHKaWUV2jBUUop5RVacJRSSnmFFhyllFJeoQVHKaWUV2jBUUop5RVacJRSSnnF/wNZSdqyLzPiNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f2(w, X, y, λ=100):\n",
    "    return np.sum((X@w-y)**2,axis=0) + λ*np.sum(w**2,axis=0)\n",
    "fw2 = f2(wgrid[None], X, y).ravel()\n",
    "minind = np.argmin(fw2);\n",
    "minw = wgrid[minind]\n",
    "plt.plot(wgrid, fw2);\n",
    "plt.plot((minw, minw),plt.ylim(),'--k');\n",
    "# plt.plot(minw, fw0[minind], 'r*',markersize=10);\n",
    "plt.xlabel(\"$w$\");\n",
    "plt.ylabel(\"$f(w)$\");\n",
    "plt.title(\"L2 regularization\"); # $f(w)=\\sum_{i=1}^n (wx_i-y_i)^2$\n",
    "print(\"minimizer: w=%.2f\" % minw);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the solution is closer to zero than without regularization, but still not zero.\n",
    "- Why? Because no incentive to move all the way to zero; the slope of the regularization term goes to zero!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L1 regularization\n",
    "\n",
    "$$f(w)=\\sum_{i=1}^n (wx_i-y_i)^2 + \\lambda \\|w\\|_1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minimizer: w=0.00\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAElCAYAAADQhFSEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8FHX+x/HXJz2B0HvvXQgQIGDDLoqIShekg108y513ej/u4NSznRXpREB6UVHQOxREgdC7EKT3HkICSUj5/v6YjcZlgZTNzu7m83w89rHJ7MzOeyeb/ex85zvfEWMMSimlVGELsDuAUkqpokELjlJKKY/QgqOUUsojtOAopZTyCC04SimlPEILjlJKKY/QgqOUlxCRASJiRGSAL68jD1liHVlq2Z1FeYYWHOURIlLL8eHyRS7mrSAir4jIfBE54FjuvCdyKvfJ8TePtTuL8g5BdgdQyoUmwOtAFvArkGpvHL+yEIgDjtsdBHgFeBM4ancQ5RlacJQ32gncCmwyxiSJyAGglL2R/IMxJhFItDsHgDHmON5R+JSHaJOa8jrGmJPGmBXGmKSCPE/O4xUi8pCIxInIRRFZnmOeABEZKiJrRCTZcVslIg9f5Tlbicj/HPMlOJr9aorIchExTvNe9RhFXo5fiMggEflKRA6KSJqInBGRL0UkOq+v2dUxnOzs17iNzDHv7SIyRUR2O543ybG9ejrnAPY7fu3v9Hy1rrUNRKS4iIx2rCNNRE47tvMNLl5vdvZgEflHjm20Q0R6X2/bKs/SPRxVFPQE7gC+BJYDGQAiIsBMoAfwC/CZY/77gfkiMsIY80H2k4hIS2AFEArMAQ4BNwM/A+cKMf8nwGbgv8BZoCbQFbhHRDoaY+JcLOPyNV9FrGMeZ92xmjcv5Zj2MlAHWIPVFFYG6ALMEpHKxpj3HfNtBj4AngO2ADmP3V31eJyIhAE/AG0c65gPVMP6G90rIvcYY352segsoBXwDdbnWm9ghoicN8Ysudr6lIcZY/Smt0K/AbUAA3yRj2UPAOfzsdwAxzozgJtdPD7c8fgnQGCO6cWwPuzSgCo5pq90zH+/0/NMcEw3TtNjHdNruVj3FY/lyDvAad7aLpZvDCQBS/P4ml2uw8V8DwCZwDog/DpZimEVlUQgwsXfPPYq63C1Df7PMW0SIDmm38rvx/QCckxf7ph/FVDcaX4DfGf3e19vv9+0SU0VBQuNMT+5mP4UkAA8b4zJzJ5ojLkIjAJCgIfB6nEFdABWG2O+cXqekVx7D6JAjDH7XUzbCSwDbhaREBeLXe01X5eINAdmYB1fedAYk3KdLBex9g5LAG3zs84c+mN1EvmrcVQOxzp+BL4C6gE3uljur8aYZKf5DwBXNDsq+2iTmioK1jtPEJEIoBlWs9hfrda1PyjvuG/kuG/huF/lPKMx5qiIHMJqanI7EakH/BW4DaiCVQhzKsuVB9+veM25XFdFYBHW8d0uxphjTo+XwGpWexDr9UY4PUXl/Kw3x3PXxuosctLFLMsd640CnIvpJhfzHwVq5DePcj8tOKooOOViWmlAsI6H/N81li3muI903J++xjrcXnBEpD6w1rH+pVjdmpOxmpe6YhXC0Kvkyeu6wrCOtVQHuhtjNjo9HgL8iPWBvwGrSewcVtNbFFYxcJUlt0o47l0VG4ATTvP9xli975xloB2jvIoWHFUUuLrK4AXH/UpjzE25eI7sHnPlr/J4BRfTshz3gS4eu+JD8ypGYHUJ72OMmZnzARFpx+97Xs7yc2XFyUAM8KoxZr6Lx7P3LsYbY4Y7Zfmz4/GCyP6bVLzK4xWd5lM+Rqu/KpKM1eV6F9BMRIrnYpEtjvsOzg+ISFVcN91k98aq6jR/AFcvFM7qOu4XOT1HOFavLLcQkb9j9ez63Bjzr7xkcXB1XCX7uJirgnsFY8wFrK7UjUXEVWG/1XG/OTfPp7yPFhxVlH0ElATGOJqT/kBEmopIBQBjzAGsM/Tbi8j9TrOOxHVrwQbH/WNO058l981vhxz3v32gO7pzv47rvao8E5HuWK9hNTA4L1kcyz+M1avNWYLjvqqLx65mKhAG/NNpHTdhNSHuxeotqHyQNqkpT2t1jbG1/muMmQHWSYE5ppcDQnJOM8YMcEOWT7H2WPoBHUXkB6zjBJWB5ljNR+35/XjI01jn4SwUkZzn4dQCtgLOJyYuBA4Cg0WkGrANaOl43h/5/Rv7tYwFBgILRGQ2Vtfjm7EOri8HOubtJbv0GdbxrC3AKy46UCw3xizH2rM5BPxZRJpi7SE2Be7Feq0P5VzIGJMsIuuAW0VkIlaxMMCnVznmAvBvoDPwuKO33AqsgtUTq/faIGNM1lWWVV5OC47ytOpYXV9dOY/VHZerzJNz2oCCBnF0u+0rIkuAIVjHICKwDlrvBJ7EKhLZ828QkVuxxv96CLiM1TW5H9aHcZLT818SkTuB97EKQwesItEe+Bu5KDjGmI0ici9WN+3ujnUux2r++lu+XviVwh33j19jnuXGGmboduAd4CasXnObgfuwivRDLpZ7DOsE0O78ftxqFlcZXscYkyIit2GNs9YDeAFru34D/MMYs8XVcso3SI6u7kqpfHAcAzoFbDfGFPQ8FKX8lh7DUSqXHON1lXWaFoDVDBSONYyMUuoqdA9HqVwSkXLAEawxzXZjnaNzE9YJpLuANjnPdldK/ZEWHKVyydGT7T9Yg2JWxjrJ8TDWkCujjDGFOYCnUj5PC45SSimP0F5qOZQrV87UqlXL7hhKKeVTNmzYcMYYc7VROH6jBSeHWrVqsX59vsY8VEqpIktEDuZmPu2lppRSyiO04CillPIILThKKaU8QguOUkopj9CCo5RSyiO04CillPIILThKKaU8QguOUkopj9CCo5RSyiO04CjlQ5YuXcrSpUvtjqFUvujQNkr5kNGjRwNw55132pxEqbzTPRyllFIeoQVHKaWUR2jBUUop5RFacJRSSnmEdhpQyoeMGzfO7ghK5Zvu4SjlQxo2bEjDhg3tjqH8yK8nk/j3t7tITc8s9HVpwVHKhyxatIhFixbZHUP5iYzMLF6Yu4XZ6w6TnJZR6OvTJjWlfMi7774LwAMPPGBzEuUPxq3Yx9YjiXzSpxXliocW+vp0D0cppYqgXScu8P7S3XRuXpn7m1f2yDq14CilVBGTnpnFC3O2UDI8mH8+2Mxj69UmNaWUKmLGLNvLjmMXGNu3NWWKhXhsvbqHo5RSRciOY4l89MOvdI2qwr3NKnl03bqHo5QPmTZtmt0RlA+7nGE1pZUuFsLILk09vn4tOEr5kOrVq9sdQfmwD7//lV0nkpj4WDSlIjzXlJZNm9SU8iGzZ89m9uzZdsdQPmjToQTGLN9Dt9bVuLNJRVsy6B6OUj7k008/BaBnz542J1G+JDU9kxfmbqFSiTD+/kAT23JowVFKKT/39nfx7Dt9kemD21EiLNi2HNqkppRSfixu31kmr9zPY+1rclP9crZm0YKjlFJ+KjktgxfnbqFGmQj+0qmR3XG0SU0ppfzVv77ZydHzKcwd3p6IEPs/7u1PoJTKtXnz5tkdQfmIZfGnmLn2EMNvqUN0rTJ2xwG04CjlU8qVs7cNXvmGhIuXeXneVhpWjOT5uxrYHec3WnCU8iGxsbEADBgwwNYcynsZY3j1i+2cv3SZ2IFtCAsOtDvSb7TTgFI+JDY29reio5QrX205xjfbjjPizgY0rVLS7jh/oAVHKaX8xPHEFF77YjutapRi+C117I5zBS04SinlB7KyDC/N3Up6puG9HlEEBXrfx7v3JVJKKZVn0+IO8vOeM/zt/sbUKlfM7jguacFRSikft+dUMm8s2cmtDcrzaLsadse5Ku2lppQPWbx4sd0RlJdJz8zi+dmbCQ8O5O1uzRERuyNdlRYcpXxIRESE3RGUl/nw+1/ZdjSRsX1bUaFEmN1xrkmb1JTyIWPGjGHMmDF2x1BeYsPBc3yybA+PtKrGvc0q2x3nurTgKOVD5syZw5w5c+yOobxAcloGz8/eQpVS4YzsYt81bvJCm9SUUsoHjVr0C4cTLjF7WHsibbzGTV7oHo5SSvmY73acYPb6wzx+a13a1vaOgTlzQwuOUkr5kFNJqbyyYBtNq5Tg+Tu9Z2DO3NCCo5RSPiIry/Di3K1cTMvgg15RhAT51ke4HsNRyocsX77c7gjKRp+tPsCK3acZ9WBT6lWItDtOnvlWeVRKqSIq/kQSbyzZxe2NKtA3pqbdcfLF1oIjItVE5CMRWS0il0TEiEgtF/OZq9yinOYLEJFXROSAiKSKyBYRecRTr0epwvbOO+/wzjvv2B1DeVhqeibPzdpEibAg3vLy0QSuxe49nHpADyAB+Ok688YC7Z1uu53mGQWMBD4GOgFxwFwRuc9tiZWy0ddff83XX39tdwzlYW9/F8+uE0m83a0F5YqH2h0n3+w+hrPCGFMRQESGAHdfY96jxpi4qz0oIhWAF4E3jTHZXwGXiUg94E1AB6FSSvmcn349zaSf9/NY+5rc1qiC3XEKxNY9HGNMlhuf7h4gBJjuNH06cIOI1HbjupRSqtCdTU7jhTlbqFehOH+9r7HdcQrM7ia1vHhCRNIcx3p+EJGbnR5vCqQBe5ym73Dc+8bYD0opBRhj+PP8rZy/lM6HvVoSFhxod6QC85WCMx14ErgTGAaUBX4QkY455ikDnDfGGKdlz+V4/AoiMkxE1ovI+tOnT7s3tVJuFh4eTnh4uN0xlAdMjzvI0p2n+EunRjSpUsLuOG5h9zGcXDHG9Mvx608i8iWwHRgN3OSYLoBzscmefq3nHg+MB4iOjna1/HWdupBKUGAAZYqF5GdxpXJtyZIldkdQHhB/IonR3+ykY8PyDLyxlt1x3MZX9nD+wBiTBHwDtMkx+RxQWq7sL1g6x+Nul56ZRY9xq3li+gYuZ7jzkJRSqihKTc/k2ZmbiAwL4u1uLXy2C7QrPllwHJz3aHYAoUBdp/myj938UhghggMDeP6uBqzZf47/+2o7V7boKeU+o0aNYtSoUXbHUIXozSW7iD+ZxDvdW1A+0ne7QLvikwVHREoA9wNrckz+FrgMPOo0e19guzFmf2HleTCqKk/dVpeZaw8Tu+pAYa1GKb7//nu+//57u2OoQvLDrpPErjrAoBtr07Ghb3eBdsX2Yzgi0s3xY2vHfScROQ2cNsb8KCIvAg2BZcAxoCbW+TaVyFFcjDGnROQ/wCsikgRsBHoCtwMPFvbreOGuhvx6MplRX/9C3fLFuaVB+cJepVLKj5y8kMqLc7fSuHIJXr63od1xCoXtBQeY6/R79vVzfwQ6AvHAQ45bSeACsBIYbIxZ67Ts34Bk4DmsghQP9DDGLCqU5DkEBAj/6RnFI5+u4qkZG1n45I3Uq1C8sFerlPIDmVmG52dvJuVyJh/19o8u0K7Y3qRmjJGr3Do6Hl9kjLnRGFPOGBNsjClrjOniothgjMk0xow2xtQ0xoQaY5obY+Z56rUUCw1iYv9oQoMCGPLZOs5fuuypVSulfNjYH/eyau9ZRnZp4tdfVG0vOP6mWukIxvVrzbHzqTz5+UbSM7XnmnKfsmXLUrZsWbtjKDfaeCiB9/63m87NK9MjurrdcQqVFpxC0LpmGd54+AZW7T3L37/coT3XlNvMnz+f+fPn2x1DuUliSjrPztxE5ZJhvP7wDX7VBdoVbziG45ceaV2NfWeS+WTZXuqWL8aQm+vYHUkp5UWMMfxt4TaOJ6Yy9/H2lAgLtjtSodOCU4heuKsh+05f5F+Ld1K7XDHuaFzR7kjKx73yyisAvPHGGzYnUQU1e91hvt56nJfuaUirGqWvv4Af0Ca1QhQQILzbowXNqpTk2Zmb2Hn8gt2RlI9bvXo1q1evtjuGKqDdJ5MYuWgHN9Yry+O3Op+r7r+04BSyiJAgJjwWTfGwIAbHruNUUqrdkZRSNkq5nMlTn2+keGgQ/+kZRWCAfx+3yUkLjgdUKhnGpP5tSLiUztDP1pNyOdPuSEopm/xj0Q5+PZXMez2iqBAZZnccj9KC4yHNqpbkg15RbD2ayPOzN5OVpT3XlCpqvtpyjFnrDvNkx7pFcjQSLTgedHfTSvztvsZ8u+ME//52l91xlA+qVq0a1apVszuGyocDZy7y1wXbaF2zNH+6q4HdcWyhvdQ8bPBNtTl49hLjVuyjZtli9GlXw+5IyodMn+58BXXlC9IyMnlm5iYCA4QPe7ckKLBoftfXguNhIsL/PdCEwwmXeO3L7VQrHV4kd62VKkreWLyLbUcTGdevNVVLFd0rthbNMmuzoMAAPurdkvoVivPU5xvZdUK7S6vcGTFiBCNGjLA7hsqDb7cf/+2SA/c0rWR3HFtpwbFJZFgwkwe0ITwkkEFT1nHygnaXVte3efNmNm/ebHcMlUuHzl7ipXlbaVGtJH/p1MjuOLbTgmOjKqXCmTygDYkp6Qycso7ktAy7Iyml3CQtI5OnZ25EgI/7tCIkSD9udQvYrFnVknz8aCviTybxzIyNZOjo0kr5hTeX7GLrkUTe7t6C6mUi7I7jFbTgeIHbGlZg1IPNWBZ/mr9/paNLK+Xrvt1+gikr9biNM+2l5iX6tKvB4YRLfLp8L9VLR/BEx6IzvpLKvQYNiub5G77EOm6zRY/buKAFx4u8dHdDjiak8O9vd1GlVBgPRlW1O5LyMuPHj7c7grqG1PRMnpyxQY/bXIUWHC8SECC83b05p5JSeXHuFsoXD6VDvXJ2x1JK5dLob35h+9ELTHwsWo/buKDl18uEBgUyrl80dcoVZ/i0DXpJA/UHw4YNY9iwYXbHUC58ufko0+MOMfzWOtzZRK995YoWHC9UMjyYKQPbUCw0iIFT1nHsfIrdkZSX2L17N7t377Y7hnKy51QSryzYRptapXnx7oZ2x/FaWnC8VJVS4cQOasPFtAwGTFlLYkq63ZGUUi5cupzBk59vJDw4kI96tyK4iI6Tlhu6ZbxYo0olGPdYa/afucjQqetJTdfr6CjlTYwxvPrFdn49lcwHvVpSqWTRur5NXmnB8XId6pbj3R5RrN1/jhGzNpOp19FRymvMXHuYBRuP8twd9bmpvnbwuR4tOD6gS4sqvNa5Cd/uOMFIPTG0SIuKiiIqKsruGArYeuQ8I7/awa0NyvPs7fXtjuMTtFu0jxh8U21OJaUy7sd9VIgM5Zk79A1eFL3//vt2R1BAwsXLPDF9I+UjQ3m/ZxQBAWJ3JJ+gBceH/PmeRpy+kMa7/9tN+chQerXVi7cp5WlZWYbn52zmdFIacx9vT+liIXZH8hlacHxIQIDw727NOXvxMn9duI2yxUO5S/v7Fyl9+/YF9Mqfdvp42R6Wx59mVNdmtKheyu44PkWP4fiY4MAAxjzaihuqleLpGRtZs++s3ZGUBx05coQjR47YHaPIWrH7NP9ZupuHWlalr14ePs+04PigYqFBTBnQhmqlwxkydT2/HNPRCJQqbEcSLvHcrE00qBDJvx5qhoget8krLTg+qkyxEKYObkfx0CD6T1nLobOX7I6klN9KTc/kiekbycg0jO3XmogQPRqRH1pwfFjVUuFMG9yW9Mws+k5aw6kkvUy1UoVh5Fc72HY0kXd7tKB2uWJ2x/FZWnB8XL0KkUwZ0IYzyWn0n7xOh8Dxc+3bt6d9+/Z2xyhSZq09xKx1h3nqtrrcrRdTKxDJy0mEIhID3AvEAFWAcOAMEA/8CHxhjEkohJweER0dbdavX293jHxZsfs0gz9bR1T1Ukwd1I7wkEC7Iynl87YcPk/3satpV6cMsQPbEqjn27gkIhuMMdHXmy9Xezgi0l9EtgGrgBFABPArsAZIANoBE4GjIhIrIrXznVzlyy0NyvOfnlGsP5jAE59v4HJGlt2RlPJp5y5e5snPrZM7P+jVUouNG1z3yJeIbAEqAFOBx4DNxsVukYiUBDoDjwI7RGSgMWa2m/Oqa+jcvApJqRm8smAbf5qzWf9J/NAjjzwCwPz5821O4t8yMrN4ZuZGTienMe/x9pTRkzvdIjddLaYAY40x1zwibYxJBD4HPheRFoA2dtqgd9saXEhJ540lu4gMC+Z17b7pV86e1fOuPOGt7+JZuecsb3VrTvNqenKnu1y34Bhjfhu8SUTE1d6Ni2W2AFsKmE3l0/Bb65KYks6Y5XspGR7MXzo1sjuSUj5j0ZZjjF+xj34xNekRXd3uOH4lr53J94jIB8BkY0xyYQRS7vHSPQ25kJrO2B/3EhkWxFO31bM7klJeb+fxC7w8byvRNUvzWucmdsfxO3ntFr0Q+D/gsIi8JSJa/r2UiPDPLs3oGlWFt7+LJ3blfrsjKeXVzl+6zPBpG4gMC2LMo60ICdKzRtwtT3s4xpgXReQ1oD/wFDBCRBYA7xpj1hVGQJV/AQHCO91bcOlyJiMX/UJESBA92uh3BF92xx132B3BL2VmGZ6btZnjiSnMGtaeCiX0yp2FIU/n4VyxsMgdwLPAfUAc8J4xZqGbsnmcL5+Hcy1pGZkM+Ww9K/ec4YNeLXmgRRW7IynlVd76dhdjlu9ldNdm9I2paXccn5Pb83DytIcjIlWA0k63H4ALQC9gHqBnHHqZ0KBAxveLpv/ktTw/ezPhwYHcqZc1UAqAr7ceY8zyvfRuW51HdQToQpXXTgNHgCwgGUgCErGKTSKwwHGvvFB4SCCTBkTz6MQ1PDljI5P7t9FrsPugTp06AbBkyRKbk/iHX45d4KW5W2ldszT/6KKnEBS2vB4V2wSkALHAzcaYZsaYDsaYTsaYnsaYYW5PqNwmMiyYzwa2pU65YgyZuk6vpeODUlJSSElJsTuGXzh38TLDpq2nRHgQn/bVTgKekKctbIxpDXQB6gHxIjJbRNrmd+UiUk1EPhKR1SJySUSMiNRyMV+YiLwtIsdFJMUx/y0u5gsQkVdE5ICIpIrIFhF5JL/5/FHpYiFMH9KOqqXCGRS7jg0HfXboO6XyLSMzi6dnbORUUhrj+kVTIVI7CXhCnku6MWaZMaYzEIXVrPajiPwsIl0l7/uj9YAeWOOx/XSN+SYBQ4G/Yw2fcxz4TkSinOYbBYwEPgY6YXVkmCsi9+Uxl18rVzyUGUNjKB8ZyoDJa9l65LzdkZTyqH8t3smqvWd5/aEbiNLLRHtMvvchjTE7sYpAC2APMAfYncenWWGMqWiMuQ+Y62oGxzA5fYDnjTETjDHfYxWpQ8A/c8xXAXgReNMY846jMA4HlgFv5jGX36tYIowZQ2MoGRFMv0lr9aqhqsiYs/4wU1YeYECHWnRrXc3uOEVKngqOiGwQkXgROSYiyUAGsBNrUM8goE5ens8Yk5shjbsA6cBvA4EaYzKAWcA9IhLqmHwPEAJMd1p+OnCDjmB9pSqlwpk5NIaIkED6TlpD/IkkuyOp6+jcuTOdO3e2O4bP2nDwHK8u3M5N9crx6v2N7Y5T5OS1l9pGrOava93crSmw3xjjfA3lHVgFpp7j56ZAGtbelvN8AE0APd3eSfUyEcwYGkPPcat5dGIcM4fGUL9ipN2x1FW8+OKLdkfwWcfOpzB82kYqlwrj4z4tCQrUTgKelteRBoYWVpBrKIPrQnYux+PZ9+ddDC7qPN8fiMgwYBhAjRpFsw9+7XLFmDkshl7j4+g9YQ2zhsVQr0Jxu2Mp5TYplzMZNm09qemZzBzajlIRerkBO/hCiRfA1XAIzh0UcjvfHxhjxhtjoo0x0eXLl89nRN9Xt3xxZg5tBxj6TIhj32kdm9UbdezYkY4dO9odw6cYY3hp3hZ2HLvAB72idA/eRtctOCLypYi0zO0TOrow/0lEHi9YtN+cw/XeSekcj2ffl3bRU855PnUV9SpEMmNoDJlZht4T4jhw5qLdkZQqsDHL9/L11uO8fE8j7misI2zYKTd7OAeBOBFZIyLPiUgrEflDU5yIVHF0i56E1WV5ENbxHnfYAdQWkQin6U2Ay/x+zGYHEArUdTEfwC9uyuPXGlSM5POh7UjPNPQar0VH+bZvt5/g7e/i6RpVhcdvzVOfJlUIclNwLgO3AWuxLk2wDkgVkXOOEzFTgcNYQ9s0BUYAzY0xa92U8SsgGOiePcFR8HoC/zXGpDkmf+vI+qjT8n2B7cYY7TCQS40qleDzIe24nJlFr/Fx7Neio3zQjmOJPD97My2ql+LNR5rrsDVeIDedBkYAc4wxz4hIEtYHe3ugMhAGnAV2YZ1TczCvAUSkm+PH1o77TiJyGjhtjPnRGLNZRGYD74tIMFZPsyeA2uQoLsaYUyLyH+AVR86NWEXpduDBvOYq6hpXLsGMoe3oM2ENvcavZtaw9tQuV8zuWErlyqmkVIZ+tp5SEcFM6NeasGAdU9gb5KbgnOP34yB/Br4wxvzbjRmcT/gc47j/Eejo+Hkg8C9gNFAK6/LV9xpjnJvt/oY1sOhzQCUgHuhhjFnkxrxFRqNKJZg5NIY+E+LoOW41s4bFUKe89l6zU48ePeyO4PVS0zMZPm0DCZfSmfu4XtvGm1z3ejiOC6zVB/4NTAXa+evF1vz1ejgFFX8iiT4T4ggMEGYOi6GuFh3lpYwx/GnOFhZuOsrYvq24t1lluyMVCbm9Hk5ujuE8DZwAPsPqdrxURH4SkQ9FZKCIRDmaupSfalgpkpnDYsgyhp7j4vj1pI5IYJdLly5x6ZLzOdAq26c/7mXhpqO8cFcDLTZe6LoFxxhzzBhzF1AV65yW2Vg90e4FJgIbgCQR2ejopab8UIOKkcwaFoMI9Bofx87jOvaaHe677z7uu0/HonVlybbjvPVtPF1aVOHp2+vZHUe5kOsTP40xJ4CFwH+MMT2MMQ2AksCtwEvAZqBVoaRUXqFehUhmD4shODCA3hPi2H5Ur7envMPWI+d5fs5mWtYoxVvdtEeat8rr9XAecYwSnf17sjHmZ2PMR8aYQcaYXJ8gqnxTnfLFmTO8PcVCgugzIY7Nh/XSBspexxNTGPLZesoWC2V8v2jtkebFfGFoG+VlapSNYPbwGEpFhNB34hrWH9BBHJQ9LqZlMDh2PZcuZzJ5QBvKR4ZefyFlGy04Kl+qlbaKToXIUPpNWsvKPWfsjqSKmMwsw3OzNrHrxAU+7tOShpV0jDRvpwVH5VvlkuFodZ8PAAAYKElEQVTMHt6eGmUiGBi7jh92nbQ7kt8bMGAAAwYMsDuGV3hj8U6W7jzFyC5N6diwgt1xVC5c9zycokTPw8mfhIuX6T/FumroB71acn9z7Y6qCte01Qd47csdDOhQi5Fdmtodp8hz53k4Sl1T6WIhTB/SjqjqpXhm5kbmbzhidyS/debMGc6cKdrNl8t2neL/vtrBnY0r8FrnJtdfQHkNLTjKLUqEBTN1cFs61C3HC3O3MHX1Absj+aVu3brRrVu368/op3YcS+TpGRtpUqUEH/RqSWCAdn/2JVpwlNtEhAQxsX80dzauyN+/3MEny/agTbbKXY4npjAodh0lwoOZ1L8NxULzdMFi5QW04Ci3CgsO5NO+rXioZVXe/i6eN5fs0qKjCiw5LYNBseu5mGZ1f66oA3L6JP2KoNwuODCAd7u3IDIsiHEr9nEhNZ3RXW/Q5g+VLxmZWTz1+UZ2n0xiUv9oGlcuYXcklU9acFShCAgQ/tGlKSXCgvl42R6SUjN4r0cUIUG6U61yzxjDq19s58fdp3nj4Ru0+7OP04KjCo2I8OI9DSkRHsTri3eRmJLO2L6tte29AJ544gm7I3jUJ8v2MGvdYZ6+rR6929awO44qID0PJwc9D6fwzFl/mL/M38oN1UoxZUAbyhQLsTuS8nILNx3h+dlbeKhlVd7r0UIH5PRieh6O8io9oqsztm9rdh6/QPexqzh2PsXuSD7p8OHDHD582O4YhW7VnjO8PG8r7euU5d+P6OjP/kILjvKYu5tWYuqgtpy6kMYjn65izym9kFte9evXj379+tkdo1DFn0hi+PQN1C5XjLH9WutxPz+if0nlUTF1yjJzWAzpmVl0G7uajYcS7I6kvMix8yn0n7yWiJBApgxsS8lwvZiwP9GCozyuWdWSzHu8AyXDg+kzIY7vd+qgnwoSU9IZMGUtF9MyiB3Ylqqlwu2OpNxMC46yRa1yxZj3eAfqV4hk2LQNzFnn/8cl1NWlpmcybOp69p+5yLh+rfVcGz+lBUfZpnxkKDOHxdChbllenr+Vj3/4VUclKIKysgwvzNnCmv3neKd7CzrUK2d3JFVI9IQIZavioUFM6t+Gl+dt4Z3/7ubEhVT+0aWZjkpwFS+88ILdEdzKGMPob3byzbbj/O2+xjwYVdXuSKoQacFRtgsJCuC9HlFULBnGuB/3cSIxjY96tyQ8RK9N7+yBBx6wO4JbjV+xj8kr9zPwxloMubm23XFUIdMmNeUVAgKEVzo15p8PNuWHXSfpNSGOM8lpdsfyOvHx8cTHx9sdwy3mbTjCG0t20bl5ZV67v4mea1MEaMFRXuWx9rUY27c18Scu8PCYVew7nWx3JK8yfPhwhg8fbneMAvth10n+PH8rN9Urx7s9WhCgTahFghYc5XXublqJmUNjSE7L4JFPV7Hh4Dm7Iyk32ngogSc/30jjypGM7dea0CBtOi0qtOAor9SyRmkWPGGdq9N7whoWbTlmdyTlBntOJTEodh0VS4QxZUBbiutArkWKFhzltWqVK8aCJ2+kRbWSPDNzk15B1McdO5/CY5PWEhQQwNRBbSkfGWp3JOVhWnCUVytTLITpQ9rxYFQV3v4unj/P30p6ZpbdsVQenbt4mX6T1pCUmkHswDbULFvM7kjKBro/q7xeaFAg7/eMombZYnz4/a8cPZ/CmEdbF8lxtl599VW7I+RZcloGA6es5UhCClMHtaVZ1ZJ2R1I20evh5KDXw/F+8zcc4S8LtlKjTAST+rehVjn9puzN0jIyGRS7jrh95xjbtzV3NalodyRVCPR6OMovPdK6GtMGt+Psxct0HbOSuH1n7Y7kUZs3b2bz5s12x8iVzCzDiFmbWbnnLG890lyLjdKCo3xPTJ2yfPnUjZQtFkK/SWuYs77oDPw5YsQIRowYYXeM6zLG8LeF21iy/QSvdW7CI62r2R1JeQEtOMon1Sxr9WCLqVOWl+dt5fXFO8nM0uZhb2CM4fXFO5m17jBP31aPwTfpkDXKogVH+ayS4cFMGdCGfjE1Gb9iH8OmricpNd3uWEXeJ8v2MOGn/TzWviYv3N3A7jjKi2jBUT4tKDCAUV2bMaprM5bvPs1DY1Zx4MxFu2MVWbEr9/POf3fzcMuqjHygqY6Ppv5AC47yC/1iajJtcFvOJKfx4CcrWbnnjN2Ripz5G44wctEv3N2kIm91a67jo6kraMFRfqND3XJ89dRNVCoRxmOT1xK7cr/fjUzw+uuv8/rrr9sd4wrfbj/By/O3cmO9snzYuyVBgfrRoq6k5+HkoOfh+IfktAxGzNrM0p0n6d66GqO6NiMsWAeILCzL408xdOp6mlUtyfTB7Sim46MVOXoejiqyiocGMb5fa569oz5zNxyh5/g4jiem2B3LLVatWsWqVavsjvGb1XvPMnzaBupXiCR2YFstNuqadA8nB93D8T/fbj/BC3M2Ex4SxKd9W9GmVhm7IxVIx44dAVi+fLmtOcC6zEDfiWuoWiqcWcNiKFtcB+MsqnQPRyng3maV+OKpG4kMC6L3+DimxR30u+M6dth+NJH+k9dSPjKU6UPaabFRuaIFR/m9+hUj+eKpG7mlQXle+2I7L87dSmp6pt2xfNavJ5N4bPJaIkOD+HxIOyqWCLM7kvIRWnBUkVAyPJiJj0Xz3B31WbDpCA+PWcWhs5fsjuVz9p1Ops/ENQQGCJ8PjaFa6Qi7Iykf4hMFR0Q6iohxcTvvNF9pEZkoImdE5KKILBWRG+zKrbxLQIDw/F0NmNy/DUcSLvHAxz+zLP6U3bF8xsGzF+kzYQ1ZWYYZQ9pRW0fqVnnka11KngXW5fg9I/sHsU5p/gqoDTwDJACvAMtEJMoYc8STQZX3uq1RBb5+5maGT9/AoNh1PHt7fZ69oz6BPnCi4vvvv2/Leo8kXKLPhDWkZmQyc2gM9StG2pJD+TZfKzg7jTFxV3msC3ATcLsxZhmAiKwG9gMvYxUrpQCoUTaCBU904NUvtvPB97+y8VAC7/eM8vqD31FRUR5f5/HEFPpMWENSajozhsbQuHIJj2dQ/sEnmtRyqQtwLLvYABhjEoFFwIO2pVJeKzwkkHe6N+eNh29gzf5z3P/hz6w/cM7uWNe0dOlSli5d6rH1nbqQSp8Jazh38TJTB7fTq3WqAvG1gvO5iGSKyFkRmSEiNXI81hTY7mKZHUANESnumYjKl4gIvdvWYMETHQgNDqDX+Dgm/rTPa7tOjx49mtGjR3tkXaeSUuk1IY6TF1KJHdiGqOqlPLJe5b98peAkAu8CQ4DbgVHAncBqEangmKcM1nEbZ9lfWUu7emIRGSYi60Vk/enTp92bWvmMZlVL8tXTN3F7owqM/mYnw6dtIPFS0b3UwemkNHqPj+NEYiqxA9sS7eMnzCrv4BMFxxizyRjzojFmkTHmR2PM+8C9QEV+PzYjgKuvpdc8EmyMGW+MiTbGRJcvX969wZVPKRkezLh+rXn1/sb8sOsU9334ExsPufoO499OJ6XRZ0Icx86nMmVAG9rW1mKj3MMnCo4rxpiNwG6gjWPSOay9HGfZezZF75ND5ZmIMOTmOsx7ogMi0GPsasav2EtWEbma6JnkNB6dGMeRhBSmDGxDuzpl7Y6k/IjPFhyHnHs1O7CO4zhrAhwyxiR7LJXyeVHVS/HNszdzZ+OKvL54F0Omrufcxct2xypUZ5LTeHTCGg6du8SkAdHEaLFRbuazBUdEooEGwBrHpK+AqiJya455SgAPOB5TKk9Khgfzad9W/PPBpvz86xk6fbCCVTZf2G3cuHGMGzfO7c+bfczm4LmLTOrfhg51y7l9HUr5xGjRIvI51vk0G4HzQEuskzovAa2MMWdEJAD4GagOvMTvJ342B1oYYw5fbz06WrS6mu1HE3l21ib2n7nIkx3rMuLOBgT7yUXGTiVZXZ+PJqQwaUC0FhuVZ/42WvR2rPNspgDfASOABUA7Y8wZAGNMFtAZ+B8wBlgIZAK35abYKHUtzaqW5OtnbqJndHU+WbaX7mNX2zIW26JFi1i0aJHbnu/khVR6jY/j2HnrmI0WG1WYfGIPx1N0D0flxjdbj/OXBVsxBkZ1bUrXqKpYIysVPndeD+dEYiq9J8Rx6kIqsYPa+vy1gpR9/G0PRymvcX/zyix57mYaV47k+dlbeHbWZp87Z+fo+RR6jV/N6aQ0pg7WYqM8QwuOUvlQrXQEs4a156V7GrJk23E6fbCC1XvP2h0rVw6fu0TPcas5e/EyUwe3pXVNLTbKM7TgKJVPgQHCU7fVY/4THQgLDqTPxDjeWLyTtAzvvbjbvtPJdB+7muS0DGYMiaFVDZcDcChVKLTgKFVALaqX4utnb6JXmxqMW7GPBz9eyS/HLtgd6wq7TybRY1wc6ZlZzBwaww3VdCBO5VnaaSAH7TSgCuqHXSf58/xtnL90mRF3NmD4LXUIcmP36cOHrQ6X1atXz9Nyvxy7QN9JawgKEGYMbUe9Cno9G+U+2mlAKRvc3qgi/x1xC3c3qcTb38XTY9xq9p+56Lbnr169ep6LzaZDCfQav5rQoABmD2+vxUbZRguOUm5WulgIH/dpyQe9othzKplOH6xgysr9bhmPbfbs2cyePTvX86/ee5a+E9dQulgIc4a318tCK1tpk1oO2qSm3O1EYiqvLNjKsvjTtK1Vhre6NadWAT7083IezrJdp3h8+gZqlIlg+pB2VCwRlu/1KnUt2qSmlBeoVDKMyQPa8Ha35uw8cYF73bi3cy2Ltx1n2LT11K9YnNnD22uxUV5BC45ShUxE6B5dnf89fysxdcryj0W/0HP8avaeLpwBzOdtOMLTMzbSolopZgyNoUyxkEJZj1J5pQVHKQ+pVDKMKY69nfgTSXT64CfGLN9DemaW29Yx+ef9vDh3Cx3qlmPq4LaUCAt223MrVVBacJTyoOy9naV/upU7GlXgrW/j6frJSrYfTSzQ8xpjeO9/u/nn17/QqVklJg2IJiIkyE2plXIP7TSQg3YaUJ62ZNtxXvtyBwmXLjP05jo8d0d9wkMCrzr/mTPW9XjKlft9VOesLMM/v/6F2FUH6BFdjdcfusGt5/4odT257TSgX4GUslGnGyrToW45/rX4F8b+uJfF244zumszbmlQ3uX8OQsNQHpmFi/P28rCTUcZenNt/npfY4+NXK1UXunXIKVsVjIimLe6tWDWsBiCAoXHJq/luVmbOJOcdsW8sbGxxMbGApByOZPh0zawcNNRXrqnoRYb5fW0SS0HbVJTdkvLyGTMsr18unwv4SGB/KVTI3pGVycgwCok2efhfLH4vwz+bD2bDiUwqmszHm1X08bUqqjT83CU8kGhQYE8f1cDFj93Mw0rRfLKgm08MnYVO4793qngckYWPcatZtuRRD7p00qLjfIZegxHKS9Ur0JxZg+LYeGmo7y+eCcPfPQzj7WvRXJaBr+eTKb6+VRiB+kloZVv0YKjlJcSER5uVY07GlXknf/G89nqAxw/mkhwYACzhsXQrKpeXkD5Fm1SU8rLlYwIZlTXZnz51I1UKBFG0yoltNgon6R7OEr5iObVSrFr7Y92x1Aq37TgKOVDIiIi7I6gVL5pk5pSPmTMmDGMGTPG7hhK5YsWHKV8yJw5c5gzZ47dMZTKFy04SimlPEILjlJKKY/QgqOUUsojtOAopZTyCO0WrZQPWb58ud0RlMo33cNRSinlEVpwlFJKeYQWHKWUUh6hBUcppZRHaMFRSinlEVpwlFJKeYQWHKWUUh6hBUcppZRHaMFRSinlEWKMsTuD1xCR08DBfC5eDjjjxjhFgW6z/NHtlne6zfIuL9uspjGm/PVm0oLjJiKy3hgTbXcOX6LbLH90u+WdbrO8K4xtpk1qSimlPEILjlJKKY/QguM+4+0O4IN0m+WPbre8022Wd27fZnoMRymllEfoHo5SSimP0IKjlFLKI7Tg5IOI/ElEFonIcRExIjIyj8vfJCKrRCRFRE6IyHsiEl5Icb2GiASIyCsickBEUkVki4g8kstlYx3b2vn2fmHn9gQRqS4i80QkUUQuiMgCEamRy2XDRORtx/sxRURWi8gthZ3ZbgXcZq7eS0ZEogo7t51EpJqIfOR4j1xyvOZauVw23/+/2bTg5M9QoALwRV4XFJHmwP+AU0Bn4FVgIBDrxnzeahQwEvgY6ATEAXNF5L5cLn8aaO90+4/7Y3qWiEQAPwCNgP5AP6A+sExEiuXiKSZhvSf/jvWeOg58588fnm7YZmD9zzm/n3a7Pax3qQf0ABKAn/K4bEH/f8EYo7c83oAAx30QYICReVh2IfArEJxj2mOO52ll92srxG1WAUgD/uE0/Xtgay6WjwWO2P06CmnbPAdkAvVyTKsNZAB/us6yLRzvnYE5pgUB8cBXdr82b9xmjnkNMNru12HDdgvI8fMQx3aolYvlCvT/m33TPZx8MMZk5Wc5EQkG7gXmGGPSczw0B7gMPOiGeN7qHiAEmO40fTpwg4jU9nwkr9EFiDPG7MmeYIzZD6zk+u+JLkA6MDvHshnALOAeEQl1f1yvUJBtVmTl97MLN/3/asHxrLpAGLA950RjTCqwF2hiRygPaYr1DWmP0/QdjvvcvPYKInJGRDJEZLeI/FlEAt2a0h5NcXpPOOzg+tulKbDfGHPJxbIhWE0o/qgg2yzbEyKS5jiW8YOI3Oy+eH7HHf+/BLkzkbquMo77BBePncvxuD8qA5w3jv3wHM7lePxaNgMbsN7gYcBDwBtY7fZD3JjTDmW4+nuidAGWzX7cHxVkm4H1zfxr4BhQE3gJ+EFE7jLGLHdXSD9S0P9fQAsOInIn1kH86/nRGNOxoKtz3Ls621ZcTPNa+dhuQgFetzHGuTfaYhFJBkaIyL+NMb/m5nm8WH63TYG2q48ryPupX45ffxKRL7H2mEYDN7khm79xy/usyBccYBXQOBfzOTdZ5Me1vg2U5vfdU1+Q1+12DigtIuL0Lal0jsfzaiYwAojG6ojhqxK4+nvC1bf4nM4BrroCF2S7+oKCbLMrGGOSROQbYHBBg/kpt/z/FvmC42j73uWh1e3FagdtmnOiiIQBdYC5HspRYPnYbjuAUKzjWDnbgbPbfn/JR4xr7TH6kh04vSccmnD97bIDeEhEIpyO4zTB6oji3ObuLwqyza7mat/ilZv+f7XTgAcZYy4D3wI9RCRnse+G9cf8ypZgnvEt1gfgo07T+wLbHT2M8qoP1gfEugJms9tXQIyI1Mme4DgZ70au/574CggGuudYNgjoCfzXGJPm7rBeoiDb7AoiUgK4H1jjpnz+xj3/v3b3C/fFG1YTTjesE6gMVrfmbo5bRI75JgEZTstGASnAAuAOrF34c8Bcu1+XB7bbm0Aq8CegI/ApkAU84DTf98CeHL/XBFYATwJ3Aw8Akx3Lfmr363LDdimG9a1xG1aX3i7AFmAfUNxpO2QAf3dafhZWM9IQx3tqnmM7+/N5XfneZsCLwASsLywdsU4c3Yb1gXqz3a/NA9su+7PqU8fn1xOO32/NMU8GMMlpuVz9/15z3Xa/eF+8YZ2EaK5yq+U8n4vlbwFWO/54J4H3cxYqf70BgVgjKxzEalrcCnRzMd9y4ECO38tgjepw0LHNUoCNwNPkOJHNl29Yx2HmAxeAJMfrreU0Ty1cnGgMhAPvAScc22cN0NHu1+St2wzrC8tKrMsnpwNnsfaK2tr9mjy03a722bXcaZ5Yp+Vy9f97rZtenkAppZRH6DEcpZRSHqEFRymllEdowVFKKeURWnCUUkp5hBYcpZRSHqEFRymllEdowVFKKeURWnCUUkp5hBYcpZRSHqEFRykvIyITRWSj07QNImJEpFqOae+IyH4/ueqpKgK04CjlfRKAyOxfROR2oCHW+FalHdMigEHAh8aYTDtCKpVXWnCU8j5/KDjAC1ijYyfz+wWvHsMaTHGSZ6MplX9F/gJsSnmhBKAEgIg0xrokQ0PgYX4vOE8DE40xF2xJqFQ+aMFRyvskAOGOYzMvAF8aY/aJSCLWZX7vBBphXTBMKZ+hBUcp75PguK+HdUXF2xy/X8Daw3kYWGCMOWhDNqXyTQuOUt4nu+D8FdhojFnt+D0RaI21Z3OjHcGUKggtOEp5n+yC8yjQM8f0C0AvYI0xJs7jqZQqIO2lppT3yS44h7Aum5wtEatn2nseT6SUG+glppVSSnmE7uEopZTyCC04SimlPEILjlJKKY/QgqOUUsojtOAopZTyCC04SimlPEILjlJKKY/QgqOUUsoj/h8S29R6o6cuWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "λ = 100\n",
    "def f1(w, X, y, λ=100):\n",
    "    return np.sum((X@w-y)**2,axis=0) + λ*np.sum(np.abs(w),axis=0)\n",
    "fw1 = f1(wgrid[None], X, y).ravel()\n",
    "minind = np.argmin(fw1);\n",
    "minw = wgrid[minind]\n",
    "plt.plot(wgrid, fw1);\n",
    "plt.plot((minw, minw),plt.ylim(),'--k');\n",
    "# plt.plot(minw, fw0[minind], 'r*',markersize=10);\n",
    "plt.xlabel(\"$w$\");\n",
    "plt.ylabel(\"$f(w)$\");\n",
    "plt.title(\"L1 regularization\"); # $f(w)=\\sum_{i=1}^n (wx_i-y_i)^2$\n",
    "print(\"minimizer: w=%.2f\" % minw);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Notice the solution is _exactly_ zero.\n",
    "- Again, whether this happens depends on $\\lambda$ and the original solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization strength intuition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: what is the effect of changing the regularization strength $\\lambda$?\n",
    "\n",
    "Let's take the oversimplified case of $n=1,d=1$ with no intercept and just one point at $(x_i,y_i)=(1,1)$. We can get zero training error by setting $w=1$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XeUFFX6xvHvSxYEJKsoyYSoKIqKohJEMaKuOeu6+jNHFnVNGDGLOaCuWXfXRM6IKCtKEhQFRQmCIBlkiDNzf3/cmnUceobp6Z6+3dPP55w+c6iqrnqri+6nb9Wt2+acQ0REpKwqhS5AREQym4JEREQSoiAREZGEKEhERCQhChIREUmIgkRERBKiIBERkYQoSEREJCEKEpEsYmadzcyZWe/QtRRlZk9Gte0VuhaJT1YHiZm1iP7jlvQYW8Z1p+0bNhWyff+lTPYDNgAzU7ExMzvPzF40s8lmtjH6/3pyKradKqnaxyrJXmGGmgW8V8y8uSmsQySb7Qt845zLS9H27gOaA0uB34CdU7TdVErJPipIvJnOud6hixDJVmbWCqgLTE3hZv8G/OCcmx+1nO9K4bZTJSX7mNWntuJh3sioaXh8kXnVzewbM9tkZvtHB+yTaPZdhU+VFXrO/079mNkRZjbGzNaY2ZxCy5xqZmPNbLWZrTezqWZ2WYzaCq/rMDMbZ2ZrzWyRmT1kZpWj5S4ws+nRun4ys7+WsL+l2nYxzy3N/lc1s57R67bezFaa2TAzO7zIuo6Intu3mG11jeY/WZraSrNv8RzrQtOrmdm10fMWRvMXmdk7ZrZbkXUk7XgVWVcXMxtvZjlmttTMXjGzhqV9XUrz2iTKzGqb2YNmNjda/5fRMd8vWuTrZG1ra5xzo5xz85O9XjP7W3RMzjSzEwq9t5eYP81UM9nbLE557WOsDWXtA2gBOODjUi7fFFiObyI2LjT9yWg9N0f/7gy8Fk0bC/QueBR6Tudo/khgEzAIeAh4Lpr/SDR/LvAS0BeYEU17okhdBesaCqwD3gceB76Ppj8E3AisAP4JPAMsi+Z1irGfpd52Ma9TifsPGNA/mj8DeBjoB6wBcoHTi6zv+6jeajG29Va0nralPIal2rfSHutC07ePah8DvBC95h9H01YALcvjeBVa13BgI/Af4AF8kBe8vrViLN+7rK9NtOzYaHrnON5v9YFvgHxgSLSfI4H1wL+i9XUI9FnQO9r+yUlY19PRuv4DrATeBB4Ffoim98n0fdxi3SF2KF0e/BEkMyn0YVfk0aHIc06NnjMo+nf36I0xFqhUaLli37BF5jvgnCLzukfTPwZqFJpeNZrmgAOLWdfxhabXAhbhP6wWAM0LzTsgWn5AItsu4bUt6QPrQv748KtSaPqeQA6wCqhdaPpN0fJnFFlP3WjfJpbyeMf7upbqWEfzqgM7xthmJ3yYvFxOx6vwui4oMu+paPo9WzsuZXhtxhJ/kAwBNgMnFZn+UrSuPKBmCc/vHc8jzs+C3iQvSMZF65oM7FBoekN8Z4KvSlFL0vczmfu4xbqTvcJMevBHkJT0uD7G816N5t0J/Ir/1rFzkWVivmFjzN/iQxAYgP/Aahxj3t7R8x6Nsa7RMZZ/OZp3R4x5s4F5iWy7hNe22P3Hf2t3xGhF4L8FO+D8QtMa4r9tDyuy7BXRspeX8njHvW+lOdal2O50YG45Ha+CdX0PWJF5DfCh9PPWjksZ/s81A1pTwgd/kXWcEK3j+RjzOhTsw1bWsbX36p8ecR6j3iQvSFbhA3OXGPMWANNC7Gcy97HoQxfbvf7OuXi6xF0LHAHcHf37LOfcL2Xc9qQY0w7Gn+a50syKzqsa/W0d43nTYkxbvJV5Bydp2/HYD1jpnJseY95Y4LpomTcBnHPLzOxj4DQza+b+OOd7Cf60yLul3G5Z9q3Ux9rMDgB6AR2BxoXWCf70ZVHJOF4Fxrvo06KAc265mc0E2plZbefc78U8F+J8bVz8592vwLc4+sSYtzL6W+L1EefcFoWlGzNriW8pD3fO/VRkXi1gB2BiSevIhP0sSkFSBs65tWY2GtgFfyriowRWtyTGtPr4Y3NXCc+rFWPamhjTcrcyr+j/gbJuOx51gB+Lmbe40DKF9QPOAC4C7jGztvjTPW8451aXcrtx71tpj7WZHQaMxn+rH45vPeTgvwFehO+CWVQyjleBpcVM/y36WwcoKUjK7bibT6bOwPRiAqjgtUllj63ysm/0d1SMeW3xHZxifUnIaAqSMjCzbsCl+IuxO+D7avcq4+pcjGlrgPXOuZ3KuM5EpGLba4AmxcxrUmiZwkYDPwMXmdm9+NYIwCtxbjeufYvjWN8KVAMOdc59UWQdZ8ZRY1k1KmZ6ca9nUeV53BsANfGndWIpOBtQYovE4ry51YXp0l8QJLFCsV30tyLs558oSOJkZg2A1/FvvAOBN4CeZjbMOTem0KIFN1VVLsNmvgKOMbOdEzhlVlbJ2nZJ+/810MXM9nbOfVtkXqdCy/yPc86Z2SvA/cCxwLnAj865cXHUFNe+xXGswbdYlscIkSbRvPJ2qJlZ4dNbUf2tgTlbOa0F5ft/bnP0d4uwM7MdgIJuzVtrkZTUWoqld5zLJ0NJ3ZhL28U5E/bzT3QfSfz6ATsCVzrn5gDn408ZvG5m9QottyL627QM23g6+vuKmdUtOtPMWppZizKsN5XbLmn/34j+9im4ZyJa9+7AZcBqfPfgol7Fn97ph/+W+2op6igs3n0r7bEGmA/UN7M9C62vGr7rblXK355RfYXdBWyD7yK9NXG9NmbWzMxal+aeiOjU4yLgwOiUZME6auGDujrwq3OuuNNzBeuxeB6l2Oe4RPfXODPrXMJi+wLznXPLY8xrB6xyzs0taTuh97Ms1CLxWpfQnFzlnOsL/kYj4BTgXefcOwDOublmdhX+wvBLwOnR82bh3zxnmVkOsDBa/sGtFeOcG2JmffCnS2ab2XD8aYFG+A+MDsA5lMPwLUncdkn7/wZwGr4nz1QzGwrUA87Ef/Cd65zb4lSMc26xmQ3CnwrJxX8Ilcu+xXmswQfGUcB4M/tXVF83fIhM449THuVlBNDPzE7EX386BH9d4jv8/RolKsNxfwPfeuyC7yCxNY/h76UYZ2bv4jsfnMwfnRBSfn0kOsaHRf8saC1ca3+MRfWyc+7zQk8p+OKdSwxmVgffE3RAjHmV8b3fvkyw7LiUYR/LJtndwDLpQem6/86Nlt0VWIv/5rldjHW9Ey1/caFphwKfRc/7U1c9ttI9OFrmWGAw/ma0TfgP40/x91U0LM26+KPLX+cY88YWrqks297K61vS/lcFbsbf8LYB32VyODFukCyyzr9E6+qfwHEvcd/Kcqyj6WfgPxDX4TsNvIa/RvGn1zmZx6vwuoCuwHj8Rf5l+BZbo+KWT/D/3Nji6ixmvZWAe/BdqDfhr3c9GW3PAfeVx3t8KzW9Rsnv/YsKLWvRazKHQvc+FVnfYSUc14Iu1H3TdR8TeVi0MZGMYGb3AHfgb2rb4ptftolOs3wC3O3S4KJrRWVmbfBfeq5yzj0Xup50o2skkjGic+qX4k+5DA5cjmSXw/BdqeO9LpcVdI1E0l50j0Zn4Dj8mFZXuNQNNS6Cc+4l/HUxiUFBIpmgG74H0hLgQeDFsOWISGG6RiIiIgnJihZJw4YNXYsWLUKXISKSUSZPnrzMOVfcqAn/kxVB0qJFCyZNijU2ooiIFMfM5pVmOfXaEhGRhChIREQkIQoSERFJiIJEREQSoiAREZGEKEhERCQhChIREUmIgkREpCL67TsYdTekYPQSBYmISEWSuwnGPggvHgFTXoc1C8t9k1lxZ7uISFZYOBn6Xw1LvoN9TodjHoRaDct9swoSEZFMt2kdfHI/THgOtt0ezv4X7HFMyjavIBERyWRzxsGAa2DlXDjgYjjqbqhRN6UlKEhERDLRhtUw8k6Y/BrUawkXDoKWhwcpRUEiIpJpZg2FQTfA2t/g0Gug8z+gWs1g5ShIREQyRc4yGHozfPs+NN4Lznobmh4QuioFiYhI2nMOvnkfhvaCjb9Dl9ug4/VQpVroygAFiYhIelu9AAbdCD8Oh6bt4aRnoPGeoav6EwWJiEg6ys+HKa/BiDvB5UH3PnDw/0GlyqEr24KCREQk3Sz/CQZcC/M+h5ad4MQnoX7L0FUVS0EiIpIu8nL9TYWf3A+Vq0OPp6Hd+WAWurISKUhERNLB4m9hwNXw61TY43g4/jGos0PoqkpFQSIiElLuRhj3KHz+OGxTD05/DdqcnPatkMIUJCIiofwy0bdCls6EtmfBMX2gZv3QVcVNQSIikmqbcmDMfTDheajTFM59H3Y7KnRVZaYgERFJpZ/H+h5Zq+bBgX+DI++CGnVCV5UQBYmISCqsXwUjboepb0L9XeCiIdCiY+iqkkJBIiJS3mYO9nen5yz1Q5t0vgWqbhO6qqRRkIiIlJe1S/z4WDM+gib7wDnvwY7tQleVdAoSEZFkcw6m/wuG3eIvrHe9AzpeB5Wrhq6sXChIRESSadUv/rdCZo+EnQ7ygyw22iN0VeVKQSIikgz5+TDpFRjV27dIjn3Y98pKw0EWk01BIiKSqGWz/e+mz/8vtOriB1ms1zx0VSmjIBERKau8XPjiafikD1StASc9B/udk1HDmySDgkREpCwWTffDmyyaBnueCMc9BrWbhK4qCAWJiEg8Nm+AcQ/D532hZgM44w1oc1LoqoJSkIiIlNb8L30rZNkPsO850P3+jBxkMdkqhS6gNMzsNDP7wMzmmdl6M5tlZn3MrHbo2kQkC2xcC0N6wavdYfN6OO8DOOV5hUgkU1okPYH5wD+ABUA7oDfQxcwOdc7lB6xNRCqy2aNh4PWw+hc46DI48g6oru+whWVKkJzonFta6N+fmtkK4HWgMzAmSFUiUnGtW+EHWfz6bWiwG/x1GDTrELqqtJQRQVIkRApMjP42TWUtIpIFvusPg3vCuuVw+E1wRC/fvVdiyoggKUan6O/3QasQkYrj999gSE/4fgBs39ZfC9mhbeiq0l5GBomZNQXuAUY55yYVs8xlwGUAzZo1S2F1IpJxnIOv34Hh//AX04+8Cw69psIOsphsGRckZrYt0B/IBS4ubjnn3EvASwDt27d3qalORDLOynkw6Hr4aQw0OwR6PA0NdwtdVUbJqCAxsxrAAKAV0Mk5tyBwSSKSqfLzYWI/GHW3H9LkuEeh/SVQKSPuikgrGRMkZlYV+AA4COjmnPsmcEkikqmWzvKDLP7yJezaDU54ArbTKfCyyoggMbNKwNvAkcDxzrkJgUsSkUyUtxnGPwmfPgTVasEpL0LbM7NukMVky4ggAZ4FTgfuB3LMrHBn7gU6xSUiW/Xr1354k8XfQJuT4bhHYNvGoauqEDIlSI6N/t4WPQq7G3+Xu4jIljav9y2Q8U9BrYZw5lt+tF5JmowIEudci9A1iEgGmveFb4Usnw3tzoOj74Nt6oWuqsLJiCAREYnLxt99b6yJ/fxF9PM/hl26hK6qwlKQiEjF8uNIP8jimoXQ4Uroeru/sC7lRkEiIhXDuhUw7FaY/h403AMuGQE7HxS6qqygIBGRzOYcfPcxDPk7rF/pB1g8oidUqR66sqyhIBGRzPX7Yhh8E8wcBDvsB+d/BNvvE7qqrKMgEZHM4xxMfQuG3wZ5G+Goe6DDVVBZH2kh6FUXkcyyYg4MvA7mfArNO/pBFhvsErqqrKYgEZHMkJ8HX74IY+4FqwzHPw4HXKxBFtOAgkRE0t+Smf7GwgUTYbej/SCLdXcKXZVEFCQikr5yN8H4vjDuEai2LfylH+xzugZZTDMKEhFJTwun+KHef/sW9j4VjnkItm0UuiqJQUEiIull0zoY2we+eAa2bQJnvQutjwtdlZRAQSIi6WPu574VsuJn2P9COPpeqFE3dFWyFQoSEQlvwxoYdRdMehXqtYALBkCrTqGrklJSkIhIWD8Mh0E3wO+L4JCrocttUK1m6KokDgoSEQkjZzkMuwW++Tc02hPOeAN2ah+6KikDBYmIpJZz8O0HMLSXP6XV6RY4/CaoUi10ZVJGChIRSZ01v/pBFmcNgR33h5OegSZ7ha5KEqQgEZHy5xxMeR1G3AF5m+Ho+6HDFVCpcujKJAkUJCJSvlb8DAOuhbmfQYvDocdTUL9V6KokiRQkIlJm85bn0O+zn/l46q/kbMylVvUqnNxuRy49vBXN69WACc/DmPugclU48Ul/b4iGN6lwFCQiUiafzFrClW9NYXNePrn5DoC1G3N576tfmD75C95q/CZ1lk+H3Y+FEx6HOjsGrljKi4JEROI2b3kOV741hfWb8/40vSq5XFXpY66s1J+1y2qy9JjnaNThHLVCKjgN5C8icev32c9szsv/07R9bTaDqv2D66t8yOD8DnTf/ChP/bavQiQLKEhEJG4fT/31f6ezarCR26q8xYfV7qK2rePiTX/nhs1XsTS/Nh9NXRi4UkkFndoSkbjlbMwF4JBKM3iwSj+aV1rCW7lH8mDu2azlj+FNcjblhipRUkhBIiJxa1J9E9fmvc45VT5hTn4Tztx4B1+6PbdYrlY1fcRkAx1lEYnPrKGMqNaTWpuW80LuCfTNPZUNVN9isSqVjFPaNQ1QoKSagkRESmftUhh2M3z7ATUa7MmZS25kUm6LYhevWrkSfzu8Zerqk2B0sV1ESuYcTP83PHsQfDcAutxGtSvGcdV5p7NN1cpUqfTnXllVKhnbVK3Mc+ftT/MGtQIVLamkFomIFG/1Ahh0I/w4HHY6EHo8A41bA9Blj8YMu/5wXv5sDh9NXUjOplxqVavCKe2a8rfDWypEsog550LXUO7at2/vJk2aFLoMkcyRnw+T/wkj7wKXB0feCQddpkEWs4yZTXbObfVHYtQiEZE/W/6TH2Rx3ufQspMfI6u+rnVI8RQkIuLl5cKEZ+GTB6BydX8aq915ujNdtkpBIiKw+BvofzUs+hpanwDHPQp1dghdlWQIBYlINsvdCOMegc+fgG3qwemvQZuT1QqRuChIRLLVL1/5VsiyWbDv2dD9AahZP3RVkoEUJCLZZlMOjL4XvnwB6jSFc9+H3Y4KXZVkMAWJSDb56RMYeC2smg8HXgrd7oLqtUNXJRlOQSKSDdavhBG3w9S3oP4ucPFQaH5o6KqkglCQiFR03w+EwTdBzjI47AbodDNU3SZ0VVKBKEhEKqq1S2DI3+G7j2H7feCcf8OO+4WuSiogBYlIReMcTHsPht0Cm9dB1zug43VQuWroyqSCUpCIVCSrfoFB18PsUbDzwf7u9Ea7h65KKjgFiUhFkJ8Pk16BUb19i+TYh32vrEr6pQgpfwoSkUy37EcYcA3M/wJadfGDLNZrHroqySIKEpFMlbcZ/vs0jH3Q98I6+Xl/h7qGN5EUU5CIZKJF0/zwJounw549/CCLtZuErkqylIJEJJNs3gDjHobP+0LNBnDGG9DmpNBVSZZTkIhkivkTfCtk+Y+w37lw9H0aZFHSgoJEJN1tXAuj74GvXoK6O8N5H8KuR4auSuR/FCQi6Wz2KBh4Paxe4H8z/cg7ofq2oasS+RMFiUg6WrcCht8G096BhrvDX4dBsw6hqxKJKWPuVjKznc3sfTNbbWZrzOxDM2sWui6RpPuuPzx7MEz/FxzeE/7vM4WIpLWMaJGYWU1gDLARuBBwwH3AJ2bW1jmXE7I+kaT4fTEM6elH692+LZz3AezQNnRVIluVEUECXAq0AvZwzs0GMLPpwI/A/wGPB6xNJDHOwdfvwPBbfffebr3hkGugcqa8PSXblfrUlpmNN7Pzzax6eRZUjB7AhIIQAXDOzQHGA+pEL5lr5Tx48xTofyU0bgNXjPe/GaIQkQwSzzWSzcDrwK9m9riZtS6nmmLZC/g2xvQZQJsU1iGSHPl5MOEFeO4QWDDR35l+0RBouFvoykTiVuogcc51BvbEh8kFwAwzG2tmZ5pZef/QQX1gZYzpK4B6sZ5gZpeZ2SQzm7R06dJyLU4kLktnwT+PhWE3Q/ND4MoJcJBG6pXMFdf/XOfcLOfcjUBT4CKgMvAOsMDMHjSzVskv8Y/Nx5hW7Oh0zrmXnHPtnXPtGzVqVI5liZRS3mYY9wi8cBgs+wFOeRHOfR+22zl0ZSIJKdNXIOfcRufcm8B1wGdAI6AX8IOZ/cfMtk9ijeBbI7HGgqhH7JaKSHr59Wt4qQuMuQ9aHw9XfQX7nqWReqVCiDtIzGwbM/urmX0FTMSHyHXAjsAVwKHA20mt0l8L2SvG9DbAd0nelkjybF4PI++Cfl0hZwmc+Tac/hps2zh0ZSJJU+quIWa2D76r7blALaA/cLNz7pNCi/Uzs8XAf5JaJQwAHjWzVs65n6N6WgAdgVuSvC2R5Jg73v/g1IqfoN35cPS9sE3MS3oiGS2ePobTgF+BvsBLzrlFxSw3G/gi0cKK6AdcDfQ3s9vx10vuBX4BXkzytkQSs2ENjL4bJr4M2zWH8z+GXbqErkqk3MQTJKcDHzvn8kpayDn3PZDUd41zLsfMugJPAG/iL7KPBq53zq1N5rZEEvLjSD/I4pqF0OFK6Ho7VKsVuiqRclXqIHHOfVCehZRi+/OBU0PWIFKsdStg2K0w/T1o1BouGQk7Hxi6KpGU0O2zIolwDmZ8BEP+DhtWwRG94IieUCXEABAiYShIRMpqzSIYfBPMGgw77AcX9Ift9w5dlUjKKUhE4uUcTH0Tht8OeRvhqHugw1UaH0uylv7ni8RjxRwYeC3MGQfND4MeT0GDXUJXJRKUgkSkNPLz4MsXYcy9YJXhhCdg/4s0PpYIChKRrVvyPfS/GhZOgt26+xCp2zR0VSJpQ0EiUpzcTfD5E36gxeq14S8vwz6naXwskSIUJCKxLJwM/a+BJTNg71Ph2IehVsPQVYmkJQWJSGGb1sHYB+CLZ2HbJnDWu9D6uNBViaQ1BYlIgTmf+R5ZK36GAy7y3Xpr1A1dlUjaU5CIbFjth3qf/E+o1xIuHAgtjwhdlUjGUJBIdvthuB9kce1iOORq6HIbVKsZuiqRjKIgkeyUswyG3QLf/Acat4Ez34KdDghdlUhGUpBIdnEOvv0AhvbyvxvS+VY47EaoUi10ZSIZS0Ei2WP1Qhh8I/wwDJoeAD2egSZtQlclkvEUJFLx5efDlNdh5J2Qtxm6PwAHXw6VKoeuTKRCUJBIxbb8Jxh4Hcz9DFoc7gdZrN8qdFUiFYqCRCqmvFz48nkYcz9UrgonPgX7X6DhTUTKgYJEKp7fZvhBFn+dArsfCyc8DnV2DF2VSIWlIJGKI3cjfPaYf9TYDk57Ffb6i1ohIuVMQSIVw4JJvhWy9HvY5ww45kGo1SB0VSJZQUEimW1Tjr8OMuE5f/rqnH/D7t1DVyWSVRQkkrl+/tQPsrhyLrS/BLr1hhp1Ahclkn0UJJJ51q+CkXfAlDd8V96LBkOLw0JXJZK1FCSSWWYOhkE3Qs4S6HidH+Kk6jahqxLJagoSyQxrl/rxsWZ8CI33grPfhab7h65KRFCQSLpzDqb/G4bd7C+sd7ndt0Q0yKJI2lCQSPpavQAG3QA/joCdDvSDLDZuHboqESlCQSLpJz8fJr8KI3uDy/P3hBx0mQZZFElTChJJL8tm+y6988ZDq85w4pNQr0XgokSkJAoSSQ95ufDFMzC2D1Su7k9jtTtPw5uIZAAFiYS3+BvofxUsmgatT4DjHoU6O4SuSkRKSUEi4eRuhHGPwOdPwDb14PTXoc1JaoWIZBgFiYTxy1d+kMVls2Dfs/2vFtasH7oqESkDBYmk1sa1MOY++PIFqLsTnPsB7NYtdFUikgAFiaTOT2P8z96umg8HXgrd7oLqtUNXJSIJUpBI+Vu/EobfDl+/BQ12hYuHQvNDQ1clIkmiIJHy9f1AGHwT5CyDw26ETjdD1RqhqxKRJFKQSPn4/TcY+nf4rj9sv4//wakd9wtdlYiUAwWJJJdzMO09GHYLbF4PR94Jh14LlauGrkxEyomCRJJn1XwYeD38NBp2Ptjfnd5o99BViUg5U5BI4vLzYeLLMKq3//exj8CBf4NKlYKWJSKpoSCRxCz70d9Y+MsE2KUrnNAX6jUPXZWIpJCCRMombzP89ykY+5D/qduTn/d3qGt4E5GsoyCR+C2a5lshi6f7sbGOfQRqNwldlYgEoiCR0tu8AT59CMY/CTUbwBlvQpseoasSkcAUJFI6876AAVfD8tmw33nQ/T4/Yq+IZD0FiZRs4+8w6m6Y2A/qNoPzPoRdjwxdlYikEQWJFG/2KH9fyOoFcPDl0PUOqL5t6KpEJM0oSGRL61bA8H/AtHeh4e7w1+HQ7ODQVYlImlKQyJ/N+BiG9PQj9h7eE474uwZZFJESKUjE+32xH6V35iDYYV9/LWSHtqGrEpEMoCDJds7B12/7U1mbN0C33nDINVBZ/zVEpHTS/tPCzHYHrgK6AK2A34GJwB3OuWkha8t4K+f6Xyz8eSw0OxR6PA0Ndw1dlYhkmLQPEuBofIi8DkwBtgN6AV+aWUfn3OSQxWWk/Dz4qh+MvhusEhz/GBzwVw2yKCJlkglB8h7wrHPOFUwwszHAXOA64IJAdWWmpbP88CYLvoJdj4ITnoDtdg5dlYhksLQPEufcshjTVpvZD0DTACVlprzN8HlfGPcwVKsFp7wEbc/QIIsikrC0D5JYzKw+sDfwz9C1ZIRfp/pWyG/fwl6n+EEWt20UuioRqSAyMkiApwED+ha3gJldBlwG0KxZsxSVlWY2r4exfeC/T0OtxnDm27DnCaGrEpEKJuVXV82sm5m5UjzGFvP8W4FzgKudc7OL245z7iXnXHvnXPtGjbLw2/fc8fB8Rz9S737nwlVfKkREpFyEaJH8F9izFMutKzrBzC4HHgBud869muzCKoQNa/xP3k56BbZrDhf0h1adAxclIhVZyoPEObcOmBnv88zsfOA54DHn3P1JL6wi+GEEDLoB1iyEDldB19v8hXURkXKUEddIzOwU/IX1l51zPUPXk3ZylsPwW2H6v6BRa7hkJOx8YOiqRCRLpH2QmNkRwLvAdOA1M+tQaPZG59zUMJWlAedgxocwpBdsWAWdbobDb4Iq1UNXJiIGP5GxAAAHZklEQVRZJO2DBOgKVAfaAeOLzJsHtEh1QWlhzSIYfCPMGgI7toMe/WH7vUNXJSJZKO2DxDnXG+gduIz04RxMeQNG3AF5G+Goe6HDlRpkUUSC0adPJlkxBwZeC3PGQfPDoMdT0GCX0FWJSJZTkGSC/Dz48gUYfS9UqgIn9IX9L9QgiyKSFhQk6e6372DANbBwEuzW3Q+yWFdDjIlI+lCQpKvcTfD54zDuUahRB059BfY+VYMsikjaUZCko4WT/SCLS76DvU+DYx+CWg1DVyUiEpOCJJ1sWgef3A8TnoNtt4ez34M9jg1dlYhIiRQk6WLOZ/5ayMo5cMDFcNTdUKNu6KpERLZKQRLahtUw8k6Y/BrUawkXDoSWR4SuSkSk1BQkIc0a6gdZXPsbHHoNdP4HVKsZuioRkbgoSELIWQZDb4Zv34fGbfwPTu10QOiqRETKREGSSs7BN+/D0F6w8XffAjnsBqhSLXRlIiJlpiBJldUL/SCLPwyDpu3hpGegcWl+30tEJL0pSMpbfj5MeQ1G3An5udD9ATj4cqhUOXRlIiJJoSApT8t/goHXwdzPfE+sE5+C+i1DVyUiklQKkvKQl+tvKvzkfqhczQfI/hdoeBMRqZAUJMm2+FsYcDX8OhX2OA6Ofwzq7Bi6KhGRcqMgSZbcjfDZY/5RYzs47VXY6y9qhYhIhacgSYZfJvpWyNKZ0PZM6N4HajUIXZWISEooSBKxKQfGRIMs1tkRzvkP7H506KpERFJKQVJWP4+FAdfCqnnQ/hLo1tv/boiISJZRkMRr/SoYcTtMfRPq7wIXDYEWHUNXJSISjIIkHjMHw6AbIWcJdLwOOt8KVbcJXZWISFAKktJYu8SPjzXjI2iyN5z9LjTdP3RVIiJpQUFSEudg+r9h2M3+wnrX26Hj9VC5aujKRETShoKkOHmb4b1z4McRsNNBfpDFRnuErkpEJO0oSIpTuSo02A12ORIOulSDLIqIFENBUpJjHghdgYhI2qsUugAREclsChIREUmIgkRERBKiIBERkYQoSEREJCEKEhERSYiCREREEqIgERGRhJhzLnQN5c7MlgLzQteRBhoCy0IXIX+iY5KedFy85s65RltbKCuCRDwzm+Scax+6DvmDjkl60nGJj05tiYhIQhQkIiKSEAVJdnkpdAGyBR2T9KTjEgddIxERkYSoRSIiIglRkIiISEIUJFnIzHY3syfNbLqZrTWzRWY2wMz2DV1bNjCznc3sfTNbbWZrzOxDM2sWuq5sZmanmdkHZjbPzNab2Swz62NmtUPXlgl0jSQLmdnVwGXA68AUYDugF9AO6OicmxywvArNzGoC04CNwO2AA+4DagJtnXM5AcvLWmY2AZgP9AcW4N8LvYGZwKHOufxw1aU/BUkWMrOGwHJX6OCbWV1gLjDQOXdBqNoqOjO7Dngc2MM5Nzua1hL4EejlnHs8ZH3ZyswaOeeWFpl2Af7L1pHOuTFhKssMOrWVhZxzy1yRbxDOudXAD0DTMFVljR7AhIIQAXDOzQHGAycFqyrLFQ2RyMTor94TW6EgEQDMrD6wN/B96FoquL2Ab2NMnwG0SXEtUrJO0V+9J7ZCQSIFngYM6Bu6kAquPrAyxvQVQL0U1yLFMLOmwD3AKOfcpND1pDsFSQVgZt3MzJXiMbaY598KnANcXfiUi5SbWBcmLeVVSExmti3+onsucHHgcjJCldAFSFL8F9izFMutKzrBzC4HHgBud869muzCZAsr8a2SouoRu6UiKWRmNYABQCugk3NuQeCSMoKCpAJwzq3Dd1OMi5mdDzwHPOacuz/phUksM/DXSYpqA3yX4lqkEDOrCnwAHAR0c859E7ikjKFTW1nKzE4B/gm87JzrGbqeLDIA6GBmrQommFkLoGM0TwIws0rA28CRwEnOuQmBS8oouo8kC5nZEcAI/Dfgq4HCN1ttdM5NDVJYFjCzWvgbEtfzxw2J9wK18Tckrg1YXtYys+eBy4H7gUFFZi/QKa6SKUiykJn1Bu4qZvY851yL1FWTfaLhUJ4AjsJfZB8NXO+cmxuyrmxmZnOB5sXMvts51zt11WQeBYmIiCRE10hERCQhChIREUmIgkRERBKiIBERkYQoSEREJCEKEhERSYiCREREEqIgERGRhChIREQkIQoSkRQys1pmNtPMvopGmy2YfrSZ5ZvZVSHrEykLDZEikmJm1g6YADzhnLvFzBoD04GvnHM9wlYnEj8FiUgAZnYD8BhwNNAT2AfY1zm3LGhhImWgIBEJwMwMGAx0BaoBRznnRoetSqRsdI1EJADnv8G9CVQHpilEJJMpSEQCMLPtgb7AFGBfM7sucEkiZaYgEUmx6LTW68Am/I9b9QUeMrO2QQsTKSNdIxFJMTO7CXgY6Oqc+9TMquF7cVUH2jvn1gctUCROapGIpFDU9fcBoI9z7lMA59wm4GygBfB4uOpEykYtEhERSYhaJCIikhAFiYiIJERBIiIiCVGQiIhIQhQkIiKSEAWJiIgkREEiIiIJUZCIiEhC/h/Cp6rZMne1fAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(1,1,'.',markersize=20)\n",
    "plt.plot((-3,3),(-3,3));\n",
    "plt.xlabel(\"x\");\n",
    "plt.ylabel(\"y\");\n",
    "plt.title(\"Extreme toy example: $d=1,n=1$\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our regularized loss is \n",
    "\n",
    "$$f(w)=(w-1)^2+\\lambda w^2$$\n",
    "\n",
    "- The left-hand term (data term) wants $w=1$\n",
    "- The right-hand term (regularization term) wants $w=0$\n",
    "\n",
    "In this particular case, because I picked such a simple example, we can do the math by hand. Taking the derivative and setting it to zero yields $\\frac{df}{dw}=2(w-1)+2\\lambda w=0$, so the minimum occurs at $w=\\frac{1}{1+\\lambda}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEYCAYAAABoYED3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcXFWZ//HPt6p6SXcSukMaDYSQBJAhGEAnKgoMiKMgOuCCuwIuBBxnxmV0XIdxwd2fOg4uwODoICMqOgNuiGyiCGpQQYKyJmwJkJDOnl7r+f1xbyWVSvVWXd1V1f19v171qup7z6l6TiXpJ+fcc89RRGBmZjbRMrUOwMzMpgcnHDMzmxROOGZmNimccMzMbFI44ZiZ2aRwwjEzs0nhhGNmZpPCCcfMzCaFE46ZmU2KXK0DqCdz586NhQsX1joMM7OGcuutt66PiK6RyjnhFFm4cCErVqyodRhmZg1F0gOjKechNTMzmxROOGZmNimccMzMbFLUXcKRNF/Sf0i6WdJ2SSFp4SjrZiS9X9JqST2SbpP08omN2MzMRqPuEg5wEPBKoBv45Rjrfgz4MHA+8ELgFuB7kk6uZoBmZjZ29ThL7caIeBKApLcALxhNJUn7AO8GPhURn0sPXy/pIOBTwE8mIlgzMxuduuvhRES+wqonAs3At0qOfwtYKmnRuAIzM7NxqbuEMw6HAb3AvSXHV6bPSybqg7/569X88LY1E/X2ZmZTwlRKOHOAjRERJcc3FJ3fg6TlklZIWrFu3bqKPvjS3zzAj29fW1FdM7PpYiolHAGlyaZwfEgRcWFELIuIZV1dI67MUFY2k2EgX+6jzcysYColnA1Ap6TSBNNZdH5CNGXFQL7SS09mZtPDVEo4K4EW4MCS44VrN3dO1AfnMmJg0D0cM7PhTKWEcxXQB7yu5PjrgTsiYtVEfXAuk3EPx8xsBPV4Hw6STktf/nX6/EJJ64B1EfGLtMwA8M2IeDNARDwu6QvA+yVtAX4PvAo4ATh1IuPNZUXfgBOOmdlw6jLhAN8r+fkr6fMvgOPT19n0UeyDwFbg7cCTgbuAV0bEDycmzEQum2Fb3+BEfoSZWcOry4QTEcPOLBuqTEQMAuelj0mTy4hBD6mZmQ1rKl3DqRlPGjAzG5kTThU0ZTP0D7qHY2Y2HCecKshmxKBv/DQzG5YTThXksqLfQ2pmZsNywqmCJt+HY2Y2IiecKshmPaRmZjYSJ5wqaMp4SM3MbCROOFWQy2YY8Cw1M7NhOeFUQS4rb09gZjYCJ5wqyGWccMzMRuKEUwW5TIbBfLDnZqNmZlbghFMFTdlkWTf3cszMhuaEUwXZTPI1ej01M7OhOeFUQaGH0++bP83MhuSEUwW5TJJwBt3DMTMbkhNOFWSzydfoHo6Z2dCccKqgKe3h+BqOmdnQnHCqIJf2cLyempnZ0JxwqqBwDcebsJmZDc0Jpwpyvg/HzGxETjhVkPN9OGZmI3LCqYLCkJo3YTMzG5oTThUUhtS8J46Z2dCccKqgybPUzMxG5IRTBdmd9+F4SM3MbChOOFWway0193DMzIbihFMFhSG1/gH3cMzMhlJ3CUfS/pIul7RJ0mZJP5C0YJR1F0j6pqQHJW2XdLek8yS1T2TMzbnka+zzkJqZ2ZBytQ6gmKQ24DqgFzgDCOA84HpJh0fEtmHqtgPXAE3AvwIPAs8APgIcDLxqouLe2cNxwjEzG1JdJRzgLGAxcEhE3Asg6XbgHuBs4PPD1D2aJLGcGBFXp8eulzQHeLektojYPhFBN6cJp9dDamZmQ6q3IbVTgFsKyQYgIlYBNwGnjlC3OX3eXHJ8I0k7Va0gS7Xk3MMxMxtJvSWcw4A7yhxfCSwZoe41JD2hT0taImmmpBOAtwNfG244brwKQ2p97uGYmQ2p3hLOHKC7zPENQOdwFSOiBziGpE0rgS3AtcCPgH8Yqp6k5ZJWSFqxbt26ioLeOWnACcfMbEj1lnAgmShQasThMEmtwHeAfYA3AMcB7yGZLPDlIT8s4sKIWBYRy7q6uioK2JMGzMxGVm+TBrpJejmlOinf8yn2ZuB44KCIuC89dqOkTcCFkr4WEbdVLdIihRs/3cMxMxtavfVwVpJcxym1BLhzhLpLge6iZFPw2/T50HHGNiRJNGcz9HnxTjOzIdVbwrkSOErS4sIBSQtJpjxfOULdR4FOSQeVHH9W+vxIlWIsqzmXcQ/HzGwY9ZZwLgJWA1dIOlXSKcAVwEPABYVCkg6QNCDp3KK63yCZKPATSWdIeq6k9wCfA24lmVo9YZpzGV/DMTMbRl0lnHTq8gnA3cAlwKXAKuCEiNhaVFRAlqL4I2I1cBTwR5LVCX5CciPphcDzI2JCs0FTVu7hmJkNo94mDRARDwIvH6HMasrMXIuIO4FXTkxkw3MPx8xseHXVw2lkTdkMvU44ZmZDcsKpkuasJw2YmQ3HCadKPKRmZjY8J5wqcQ/HzGx4Y044kg6eiEAaXVPWPRwzs+FU0sP5S7oL57OrHk0D842fZmbDqyThXAa8CPiVpJskvbTKMTWk5pyXtjEzG86YE05EvA5YCHyKZIfNyyXdLemcdMXmaSm5hjNY6zDMzOpWRZMGImJtRHwQ2B84B+gDvgI8KOnfJM2tYowNIenheEjNzGwo45qlFhG9EXFRRDwVOBH4HXAu8ICkr1YjwEbhWWpmZsMb89I2khaQ7FnTmT4Xv36EZIuBpwLLgbdWLdI619qUoaffCcfMbCiVrKW2quh18Xpmg8BmYBPJApqbxhFXw2ltytLT72s4ZmZDqSThiGQb6P8DPgM8DGwqWc152mlpytI7kCefDzKZEXfENjObdiq5hnMUyR41pwI/Bd4GtFczqEY0oykLQK+v45iZlVXJtOjfRsTLSLZsvhx4J7Ba0n9KmrBtnOtda1PyVXpYzcysvIpnqUXEPRGxnOSenC8ALwX+JOlKScdUKb6G0Zr2cHp8L46ZWVnjXrwzIh6LiA8AC4D3AEcAv5B083jfu5EUhtR29DnhmJmVU8m06A8Ds4Z5zCSZWPDMqkXZAHYNqfkajplZOZXMUju35Oc8yRTo7vSxquj1tNHiITUzs2FVknCeD2wgTSoRMa3utxlKay5NOB5SMzMra8wJJyKunYhAGt2MZvdwzMyG4x0/q8TXcMzMhueEUyWFITXPUjMzK88Jp0o8pGZmNjwnnCrZOWnAQ2pmZmU54VRJi5e2MTMbVt0lHEn7S7pc0iZJmyX9IN2DZ7T1D5X0PUnrJe2QdJekt09kzAAtuQySE46Z2VAquQ9nwkhqA64DeoEzSLZBOA+4XtLhEbFthPrL0vo3AG8huSH1YJLVDyaUJFpz3hPHzGwoIyYcSV+v8L0jIt48xjpnAYuBQyLi3vTzbwfuAc4GPj9MnBngm8C1EfHSolPXjzGGinnXTzOzoY2mh3Nmhe8dwFgTzinALYVkAxARqyTdRLL/zpAJBzgeWAKcM8bPrJrWpiw73MMxMytrNAln0YRHscthJJu7lVoJvGKEuoUtEVol3QL8NcnyO5cB742IHVWLcggzvM20mdmQRkw4EfHAZASSmkP5RT83AJ0j1N03ff4OcD7wPmAZ8FFgf5L9eiZUS1PWQ2pmZkOoq0kDqShzTKOoV5hx962IKKxofYOkLPApSUsi4s493lhaDiwHWLBg1JPhykqu4biHY2ZWTr1Ni+4m6eWU6mTk7Q6eSJ9/XnL86vT5yHKVIuLCiFgWEcu6urpGHWg5HlIzMxtaxT0cSfOA5wH7AS1likREfGyMb7uS5DpOqSXAHr2TMnVhzx5SoXc04WNdM5qybNrRP9EfY2bWkCpKOJI+QnKNpLi+2PXLvvB6rAnnSuBzkhZHxP3pZy0Ejk4/bzg/Jbl/5yTgR0XHT0yfV4wxljFra8mxrXdgoj/GzKwhjXlITdLrgH8FfgmcRpJcvgm8FriIpCdxGXBCBfFcBKwGrpB0qqRTSGatPQRcUBTDAZIGJO3cfTQingA+CZwj6ROS/lbS+0h2KP1m8VTriTKzJcvWXg+pmZmVU0kP563Aw8BJETEgCWB1RFwGXCbpf4EfA98e6xtHxDZJJwBfAC4hSWbXAu+IiK1FRQVk2TNhfhTYAvw98G5gLfBZxt7Tqkh7s3s4ZmZDqSThLAW+HRHFv1mzhRcR8TNJPwPeA/xwrG8eEQ8CLx+hzGrKzFyLiCC5OXS4G0QnzMzWHDv6BxnMB9nMaCbWmZlNH5XMUmti14wwgB3AXiVl7gCOqDSoRjWzJcnfW93LMTPbQyUJZy0wr+jnB4HDS8rsB0y737rtacLxsJqZ2Z4qSTh/IBlWK7gOOFbSGyS1S3oRyZDYH6oRYCOZ6YRjZjakShLOj4DDJBXWWPsUyTYA3wA2k0xtFvChagTYSAoJZ4sTjpnZHsY8aSAivkGSXAo/PyTpGcA/AweSTGv+SkT8qTohNg4PqZmZDa0qa6lFxCrgH6rxXo2svSWZrOeEY2a2p3pbS62hzWppAvDNn2ZmZVS6tM184J0kC2LOJ5kqXSoi4sBxxNZwCj2crT1eT83MrNSYE46k44GfAK0kU58fo/wU6Gl35+POazh97uGYmZWqpIfzGZKVBU4H/icivONYqiWXoSkr3/hpZlbGeJa2+Va1g2l0kmhvybG1xwnHzKxUJZMGukm2fLYyvICnmVl5ld74eVy1A5kqZrbkPKRmZlZGJQnnA8Bekr4sqb3aATW62TNybPYsNTOzPVSy0sB6SScBvwFOl3Q3ydI2ZYrG88YbYKPZa0Yzj2zcUeswzMzqTiXTog8Drgc600NPG6JoDHF8Sutoa+LONeXyr5nZ9FbJkNrngb1Jtm4+AGiKiEyZR3b4t5maOmY0sXGHh9TMzEpVMi362cAPIuK8agczFXS0NbG9b5DegUFactMy55qZlVVJD6ePZEVoK2OvtmYANrmXY2a2m0oSzg3AM6scx5TRMSNZVm7TdiccM7NilSScfwGWSHqfpGm3XtpIOtqShOPrOGZmu6vkGs6HgDuAjwNnSfojQ0+LfvN4gmtEHTOSIbWN7uGYme2mkoRzZtHrRemjnACmX8JJezi+hmNmtrtKEs5QCcaAvQpDatv7ahyJmVl9qWSlgQcmIpCpYlZLjmxG7uGYmZXwFtNVJom9ZjT5Go6ZWQknnAnQ0dbEhm0eUjMzKzbikJqkr5NMAPhARDyW/jwaFc1Sk7Q/8AXg+STbVF8DvCMiHhzj+7wf+ARwU0QcM9Y4xqNrZgvrtvZO5keamdW90VzDOZMk4XwaeIzdZ6kNZ8yz1CS1AdcBvcAZ6XucB1wv6fCI2DbK91kMfBB4fCyfXy1zZ7Xw57Wba/HRZmZ1azQJpzAr7ZGSnyfCWcBi4JCIuBdA0u3APcDZJAuHjsZXgUuBQ6hsJt64dM1s4cYt7uGYmRUb8Zdx6ay0CZ6ldgpwSyHZpJ+3StJNwKmMIuFIei3wdOA1wA8mKtDhzJ3ZzJaeAXr6B2lt8gKeZmZQwaQBSadLOnyEMkslnV5BPIeRrGJQaiWwZBSxdZJc//mXiNhQwedXRdesFgDW+zqOmdlOlcxS+wbwkhHKnAL8VwXvPQfoLnN8A7s2fBvOZ4G7SWKsmbkzCwnHM9XMzAom6vpGlsp3/CxXb8RFQiUdC5wOPD0iRv3ZkpYDywEWLFgw2mrDKiScdb6OY2a200Tdh/MUyvdURtJN0ssp1TmK97sAuBh4WFKHpA6ShJpNf24pVykiLoyIZRGxrKurq4KQ9+QhNTOzPY2qh1Pm3puXSFpYpmgWWAAcC/y4gnhWklzHKbUEuHOEuoemj3PKnOsG3gl8sYKYxmzvmcmK0evdwzEz22m0Q2pnFr0O4Mj0UU4AvyH5BT9WVwKfk7Q4Iu4HSBPb0cD7Rqj73DLHvkiSBP8RuLfM+QnRksvS0dbEo5t7Jusjzczq3mgTTuHeGwH3k/wi//cy5QaB7tHeoFnGRcA/AFdI+hBJ8voY8BDJkFkShHQAcB/w0Yj4KEBE3FD6ZpI2Arly5ybavnvNYO0mJxwzs4JRJZzie28kfQS4fiLux4mIbZJOIJnafAlJgruWZGmbrUVFRdJzqdu14PbtaOXh7h21DsPMrG5Usj3BRwAkzQReCjwN2Itk188/AP9bkhzG+v4PAi8focxqRjFzLSKOrzSO8dq3Ywa/XVWzW4HMzOpORdOiJb0C+BrQwe6/+AP4oqSzI+LyKsTXsObtNYPNPQNs6x2gvWXSV9cxM6s7Y/5NKOn5wLeBPPDfwA3Ao8CTSS7cvxb4tqSNEXFN9UJtLPt2tAKwdtMODtpnVo2jMTOrvUr+630uyWrOx0bE70vOfVPS+cCNablpnHBmAPDIxh4nHDMzKrvo/jTgO2WSDQARsQL4LskCmtNWIeGs2eiJA2ZmUFnC6QXWjlBmTVpu2nrSrBZyGfHQhu21DsXMrC5UknB+CYy0g+bRJMNq01Yum2HBnDZWP1HpLUlmZlNLJQnnvcBSSZ+S1F58QlK7pM8AT2XklQGmvEVz21m13j0cMzOobNLAe4HbgfcAyyX9nmTr6SeRXLfZi6R3815pt1tlIiLGtOV0o1s4t51f3/cEEUHJd2FmNu1UknDOLHrdAZxQpsxx6aNYANMu4ezoH+Sxzb08ea/WWodjZlZTlSScRSMXMYDFc5MRx/vXb3XCMbNpr5Klbaq+htpUtTBNOKvXb+c5B9Y4GDOzGhv3miuSZpOupRYRm8cf0tQxb3YrLbmMZ6qZmVHhasuSspLeJ+leks3NVgPdku5Nj3vxMCCTEYvmtnPv4xWvZWpmNmVUspZaM3AVyaSAINmrZi0wD1gIfBw4SdILIqKveqE2pr968iyvGm1mRmU9nHcBx5NsIX1oRCyMiGdHxELgEOCHJFtMv6taQTayJfvOZs2mHrq3Tfvca2bTXCUJ57XAHcBLIuKe4hMRcR/wMmAl8Lrxh9f4lszbC4A/r/XlLTOb3ipJOAcBP42IfLmT6fGfAp6XBRw6L1kp+k4nHDOb5ipJOH3AzBHKtAP9Fbz3lLP3zBaeNLuFO9c44ZjZ9FZJwrkdOE1SV7mTkuYCpwG3jSewqWTJvNnu4ZjZtFdJwjkf6AJ+K+nNkhZLmiFpkaQ3Ar9Jz59fzUAb2dL99uLux7awtXeg1qGYmdVMJSsNfFfSkSSrQV9YpoiAz0TEd8cb3FSxbOEc8gF/fHAjxxw8t9bhmJnVREU3fkbEB4DnAF8H/gDcnz5/HTg6Iqb91gTFnragg4zgd6t9P46ZTV8VrwgQEbcAt1QxlilrVmsThzx5Nrc+0F3rUMzMaqaiHo6N3bIDOvnDg90MDJadTW5mNuU54UySZy6aw7a+QW5/ZFOtQzEzqwknnElyzEFzkeDGu9fVOhQzs5pwwpkkne3NHDG/g1844ZjZNFV3CUfS/pIul7RJ0mZJP5C0YBT1lkm6UNJfJG2X9KCkSyXVzQ6lxz2li9se2uiFPM1sWqqrhCOpDbgO+CvgDOANwMHA9ZLaR6j+auAw4EvAC0nuE3o6sELS/hMW9Bgcd0gX+cC9HDOblupto7SzgMXAIRFxL4Ck24F7gLOBzw9T99MRsdtvckk3AavS9z13QiIegyPnd/Dk2a38+E9recnT9qt1OGZmk6quejjAKcAthWQDEBGrgJuAU4erWJps0mMPAOuAuvjtnsmIk5fO4xd3rWNLj9c2NbPppd4SzmEke+2UWgksGeubSToU2Af48zjjqpoXHT6PvsE8P7/zsVqHYmY2qeot4cwByt2OvwHoHMsbScoBXyPp4Vw8/tCq42n7dzC/cwaX3/pwrUMxM5tU9ZZwAKLMMVXwPueTrPf2+ogYck0ZScslrZC0Yt26ib+Yn8mIVz9jf3593xOsWr9twj/PzKxe1FvC6Sbp5ZTqpHzPpyxJnwSWA2+KiKuHKxsRF0bEsohY1tVVdoufqnvlsv3JZcS3f/vgpHyemVk9qLeEs5LkOk6pJcCdo3kDSR8kmRL99oi4pIqxVc0+s1t5/pIn8b0VD9HTP1jrcMzMJkW9JZwrgaMkLS4ckLQQODo9NyxJ/wScB3wwIv5jgmKsijOes5Du7f1c5l6OmU0T9ZZwLgJWA1dIOlXSKcAVwEPABYVCkg6QNCDp3KJjrwa+CFwFXCfpqKLHmGe4TbRnLZrDMxZ2csGN99M74F6OmU19dZVwImIbcAJwN3AJcCnJjZsnRMTWoqICsuwe/0np8ZOAm0seX5nw4MdIEv94wsGs3dTjGWtmNi3U20oDRMSDwMtHKLOakplrEXEmcOZExTURjj14Lk9f0MEXr7mHU4/cj5ktdffHYWZWNXXVw5luJHHu3x3Gui29fPn6e0euYGbWwJxwauzI/Tt42dP34+JfruLex7fUOhwzswnjhFMH3v/CQ2lvyfKu795Gv7egNrMpygmnDnTNauHjL13K7Q9v8tCamU1ZTjh14uSl83jJkfvypWvv4Vf3rK91OGZmVeeEU0fOe+lSDt5nFm/7n9/zwBNeZ83MphYnnDoysyXHhaf/NQBv/MbvWL+1t8YRmZlVjxNOnTlg73YuOn0Zazbu4A0X/5ZN271Rm5lNDU44deiZi+ZwwRuWcd/jW3n1Rbfw+OaeWodkZjZuTjh16rindHHRGct44IltvOyrv+a+dVtHrmRmVseccOrYcU/p4rLlR7Gjb5CXfPkmrl75aK1DMjOrmBNOnTt8fgf/97ajWTS3neWX3Mp5P7rTe+iYWUNywmkA+89p43vnPJs3HHUA//mrVZz8779kxeoNtQ7LzGxMnHAaREsuy8de8lQuefMz6R3I84oLbuafv3sbj27yhAIzawxOOA3m2IO7uPqdf8PyYxfzw9vWcPznruf/XX0XG7f31To0M7NhKSJqHUPdWLZsWaxYsaLWYYzaQxu28+mr/sKPbl9LW3OW1z5zAW8+dhHz9ppR69DMbBqRdGtELBuxnBPOLo2WcAruenQLX/vFfVx52xoE/O2hT+I1z1rAsQfNJZPRiPXNzMbDCacCjZpwCh7asJ3/vnk13//9I2zY1sf8zhmceuS+nLx0HkvmzUZy8jGz6nPCqUCjJ5yC3oFBfn7nY3zndw/x6/ueYDAfLNy7jZOXzuN5h+7DEfM7yGV9+c7MqsMJpwJTJeEUe2JrLz9b+Rg/+dNabr4/ST6zW3McfdBc/uYpXTznwL1ZMKfNvR8zq5gTTgWmYsIptnF7H7+6dz033r2OG+9ez6PpGm1zZ7aw7IBOli3s5OkHdHLYvrNpyWVrHK2ZNYrRJpzcZARj9aGjrZkXH74vLz58XyKCex/fym9WbeDWB7pZ8cAGrkqXzsllxEH7zGTJvNks2Td9zJtNR1tzjVtgZo3MPZwiU72HM5LHN/dw6wPd3LFmE3eu2czKNZt5fMuuPXnmzmxhcVc7B3bN5MCudhZ3tbN47kzmd87wNSGzacw9HBuzfWa38sKl83jh0nk7j63b0suf127mzrWbue/xrdy/fhtX3bGW7qJ9enIZMa+jlfkdbezXOYP5nTPYr2MG8zvbmN85gyfv1UqTE5LZtOeEY8PqmtVC16wu/uYpXbsd37Ctj/vXbeX+ddtY9cQ2HunewcPd2/nlPet4bPPuO5VKMKetma5ZLTxpdiv7zGphn9kt7DNr1+uuma3MmdlMe3PWExjMpignHKvInPZm5rTPYdnCOXuc6x0YZO3GHh7u3sEjG7ezZmMPj2/pZd2W5PmuR7ewbmsvg/k9h3Obsxk62pqY095MZ1szne1NdLY1M6e9mY62Zua0N9HR1szs1hyzW5uY1drErNYcbU5UZnXPCceqriWXZeHcdhbObR+yTD4fbNjex+Obe3l8Sw/rtvTSvb2P7u39dG/rY8O2Prq393H3Y1vpTl+XyU87ZTNiVmuOWTsTUY5ZrU07X89uzdHekiSmtuYc7S1ZZjTnaG/OMqM5S3tzjraW5FxbU9YrNJhNgLpLOJL2B74APB8QcA3wjoh4cBR1W4GPAa8HOoA/Au+NiBsnLmKrRCYj5s5sYe7MFpYwe8Ty+Xywuac/TUT9bOnpZ0vPAFt6BtjcU/LzjuT1Qxu27zy/tXeAscyPaW3K0N6c2y0ZteaytDZlaG3K0pJLnlubsrQ0ZWgpnMslPydlk2MtZeoV6jRnMzRl5UkXNi3UVcKR1AZcB/QCZwABnAdcL+nwiNg2wltcDLwIeA9wP/A24GeSnh0Rf5y4yG2iZTKio6254qnZ+XzQMzDItt5BdvQNsq1vgO19A2zvS44VXm/vG0jK9A+yrXegqGxSb2vvAD39g/QMDNLbn09f5+kbyI+vfYKmbIbmbIbmXIambIamnNKElKGlcKzofHPR+V3HMjuPFernMiKbzdCUSRJbLiNyWSXPmQzZrGjKZHYdK1MmeV2mTEZkM/Jwpo1KXSUc4CxgMXBIRNwLIOl24B7gbODzQ1WUdATwWuBNEfFf6bFfACuBjwKnTGzoVs8yGSXDZc0T81c+nw/6BtME1J+ndyB57ukfpHegcLzodZqk+gby9A8mj76BPH1Fr/sHk/cslCk8b98xuEe93etH2etjE2lXgsrsnszS41mJTCZ5zqZJKvk5GQ7NFB3PZvYsn0mTW1Ju9zoZ7Up8mT3qQjaTIZtht8/Y7fOKPiMjyCpJoBkldTIZ0p+LjhVep3VKz0sUxTf0+d3fp7jsrvO7ytLwyb3eEs4pwC2FZAMQEask3QScyjAJJ63bD3ynqO6ApMuA90lqiYjeIWubjUMmI1ozyTBaPRjMR5KEBvMMDgb9+TwDaSLqH8ynz+nP+fxuxwcGg4F8MDCYpz8fDOZ3JbGBwXx6bleZgXwwkC86XlSmP58nnw8GI0nKg/mkXj6S14XnwXzQN5BnMCItHwzmkzoD+Tz5YGe54jqDhfdJX+fz7Cw/lQ2VkAo/F5Lq7skrTWZp8hW7ykvix/90zISvMFJvCecw4Ioyx1cCrxhF3VURsb1M3WbgoPS12ZSX/O+9fhLgZIuInUmqkKAGdktMu5JVcSLLB8lzPnmOws/pudiZKHd9RuH8rrKF9xj+fHGMu79X4b2Ly7JbjOXOl37ebmXz5T872PVzZhJ6TvWWcOYA3WWObwA6x1HLSk0CAAAJI0lEQVS3cN7MpgFp13Cd1Y96nBpTrjM8mr81qqSupOWSVkhasW7dutHEZ2ZmFai3hNNN+Z5IJ+V7L8U2DFO3cH4PEXFhRCyLiGVdXV3lipiZWRXUW8JZSXItptQS4M5R1F2UTq0urdsH3LtnFTMzmyz1lnCuBI6StLhwQNJC4Oj03Eh1myiaXCApB7wKuNoz1MzMaqveEs5FwGrgCkmnSjqFZNbaQ8AFhUKSDpA0IOncwrH0xs7vAF+U9BZJzwMuAxYB/zaJbTAzszLqKuGkKwmcANwNXAJcCqwCToiIrUVFBWTZM/43Av9FsjrBj4H9gZMi4vcTHLqZmY2g3qZFk66Z9vIRyqymzOyziNgBvCt9mJlZHamrHo6ZmU1d3mK6iKR1wAMVVp8LrK9iOI3AbZ4eplubp1t7YfxtPiAiRryvxAmnSiStGM2e3lOJ2zw9TLc2T7f2wuS12UNqZmY2KZxwzMxsUjjhVM+FtQ6gBtzm6WG6tXm6tRcmqc2+hmNmZpPCPRwzM5sUTjjjIGl/SZdL2iRps6QfSFpQ67iqQdJpkr4v6QFJOyTdJemTkmaVlOuU9J+S1kvaJukaSUtrFXc1SbpKUkg6r+T4lGuzpJMl3Shpa/p3eYWkE4rOT5k2Szpa0tWSHk/b+ntJbyop0yrps5LWpn//b5b0N7WKeSwkzZf0H2nM29O/wwvLlBtVGyVlJL1f0mpJPZJukzTszflDccKpULoq9XXAXwFnAG8ADgaul9Rey9iq5N3AIPAB4CTgq8BbgZ9LygAo2Vz9yvT8P5KsENFE8h3Mr0XQ1SLpNcARZY5PuTZLOptkzcJbgZeSLID7PaAtPT9l2izpcOAakvjPImnL74CLJb21qOjF6flzgRcDa4GfSTpyciOuyEHAK0m2dPnlMOVG28aPAR8GzgdeCNwCfE/SyWOOLNLtRf0Y2wN4O8kv5IOKji0CBoB31Tq+KrSvq8yx00k2uTsh/fnU9OfnFpXZi2TvoS/Vug3jaHsH8CjwmrR95xWdm1JtBhYCO4B3DFNmyrQZ+ATJdiUzS47fAtycvj4ibe8bi87ngLuAK2vdhlG0MVP0+i1pWxaWlBlVG4F9gF7gIyX1rwVuH2ts7uFU7hTglojYuc9ORKwCbiL5B9rQIqLc9qe/S5/3S59PAdZExPVF9TYBP6Sxv4PPACsj4ttlzk21Nr8JyANfG6bMVGpzM9BPkmSLbWTXiM8paZnvFE5GxADJ6vMnSmqZhDgrFhH5URQbbRtPJPnOvlVS/1vAUkmLxhKbE07lDgPuKHN8Jcmmb1PRcenzn9Pn4b6DBZJmTkpUVSTpGJKe3N8PUWSqtfkY4C/AqyXdl277ca+ktxWVmUpt/kb6/CVJ+0rqkHQW8DzgC+m5w4BVEbG9pO5Kkl++B01KpBNrtG08jKSHU7qB5cr0eUy/65xwKjeH8tteb2DXttZThqT9gI8C10TEivTwcN8BNNj3IKmJZN+lz0XEXUMUm1JtBvYlufb4WeBTwAuAnwPnS3p7WmbKtDki7gCOJ+mZPULSri8D50TEZWmxkdpbbiv7RjPaNs4BNkY6jjZMuVGpu+0JGky5m5j22Dah0aX/g72C5PrUG4tPMbW+g/cCM4CPD1NmqrU5A8wCzoyIH6THrktnNb1f0peYQm2WdDDwfZL/oZ9DMrR2KvA1ST0RcSlTqL3DGG0bq/pdOOFUrpvy2b2T8v9zaEiSWklmKC0GjouIh4tOb2Do7wAa6HtIp7N/kOQia0vJOH2LpA5gC1OozaknSHo4Py85fjXJrLR5TK02f4Lk2sWLI6I/PXatpL2Bf5f0bZL2lru9odDeDWXONZrRtnED0ClJJb2cir4LD6lVbiXJ+GapJcCdkxzLhEiHmL4PPBM4OSL+VFJkuO/gwdh9l9Z6txhoJbkY2l30gGSKeDewlKnVZtg1Fl+q8D/YPFOrzUuB24qSTcFvgb1JZmWtBBaltz4UW0Iyw630ekYjGm0bVwItwIFlysEYf9c54VTuSuAoSYsLB9JhiKPTcw0tvdfmUpKLqadGxC1lil0J7CfpuKJ6s4G/o/G+gz8Czy3zgCQJPZfkH+FUajPA/6bPJ5YcPxF4OCIeZWq1+VHgSEnNJcefBfSQ/I/9SpL7dF5ROCkpB7wKuDoieicp1ok02jZeRZKAXldS//XAHenM3NGr9ZzxRn0A7SS/gP5EMgZ8CnAbcD8lc/wb8UFyo2cA5wFHlTzmp2UywK+Bh4BXk/ySuoHkH+3+tW5Dlb6H0vtwplSbSXoy15EMrZ1DMmngwrTdZ061NgOnpW37Wfrv9gUkNzQG8PmicpeR9GrfQvKfrstJEtLTa92GMbTztKJ/x29Nfz5urG0kmUzSA7yLZMLFV0l6vn835rhq/cU08oNkDPT7wGaS8f3/o+QGq0Z9AKvTv6jlHh8uKjcH+Hr6y2c7yQ1hR9Q6/ip+D7slnKnYZmA2yUytx0j+N3s78Nqp2maSu+VvANal/27/SDINPltUZgbweZIeUQ/wG+D4Wsc+hjYO9W/3hrG2EcgCHyLZDbk3/ftxWiVxebVoMzObFL6GY2Zmk8IJx8zMJoUTjpmZTQonHDMzmxROOGZmNimccMzMbFI44ZjVMUnL0y2C10v6QmG3VbNG5L+8ZvVtFfA5kmVI3gE8v7bhmFXOCcesjkXEzyPiPSTLi0CytJBZQ3LCMWsMhcVTj6hpFGbj4IRj1hgKq/IeXtMozMbBCcesMfxr+rw43YHVrOE44ZjVOUkvAN5U+JFkEzGzhuOEY1bHJM0CLgI2kmwEBx5WswblhGNW3z5Dsu/S24GfpMc8ccAaUq7WAZhZeZKeC5wN/Cgi/ltSYR9593CsIXkDNrM6JKmdZGfFOcBhEbFGUpZkh8p+oCP8j9cajIfUzOrTJ4HFwD9FxBqAiBgEVpJsCb2wdqGZVcYJx6zOSDoGeBvww4i4pOT0H9NnD6tZw3HCMasjkmYAFwObSK7flLotffbEAWs4njRgVl8+BjwFOD0i1pY57x6ONSxPGjAzs0nhITUzM5sUTjhmZjYpnHDMzGxSOOGYmdmkcMIxM7NJ4YRjZmaTwgnHzMwmhROOmZlNCiccMzObFE44ZmY2Kf4/UCrQv9iLUOQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(0,100,1000)\n",
    "plt.plot(x, 1/(1+x))\n",
    "plt.xlabel('$\\lambda$', fontsize=20)\n",
    "plt.ylabel('optimal $w$', fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a _regularization path_ but in the very simple case of just one $w$. We recover the limiting cases:\n",
    "\n",
    "- $\\lambda=0$ gives us $w=1$\n",
    "- $\\lambda \\rightarrow \\infty$ gives us $w=0$\n",
    "\n",
    "All this pretty much generalizes to larger $n$ and $d$. So $\\lambda$ controls the relative importance of training error vs. regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularization paths\n",
    "\n",
    "Here's an example with $d>1,n>1$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = 10\n",
    "n = 30\n",
    "w_true = npr.randn(d)*3\n",
    "X = npr.randn(n,d)\n",
    "y = npr.randn(n)/5 + X@w_true\n",
    "λ_valsL2 = 10**np.linspace(-2,4,1000)\n",
    "λ_valsL1 = 10**np.linspace(-2,2,1000)\n",
    "\n",
    "WsL2 = np.zeros((len(λ_valsL2),d))\n",
    "WsL1 = np.zeros((len(λ_valsL1),d))\n",
    "for i,λ in enumerate(λ_valsL2):\n",
    "    ridge = Ridge(alpha=λ)\n",
    "    ridge.fit(X,y)\n",
    "    WsL2[i] = ridge.coef_\n",
    "for i,λ in enumerate(λ_valsL1):    \n",
    "    lasso = Lasso(alpha=λ)\n",
    "    lasso.fit(X,y)\n",
    "    WsL1[i] = lasso.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEpCAYAAAC0kdQLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXd8HsWB//+e3acXSY96t9y7DS6AbcB0CBA6ISEJIY3kIJfc5ZtLu+SSHMn9ruQScglJIAmh5DjA9BISAjamg7tx75Ylq7dHT3+e3fn9sY+kR/IjW7Yly7Ln/fK8Znb6rh7PZ6fsjJBSolAoFArF8aKNdgUUCoVCcWqgBEWhUCgUw4ISFIVCoVAMC0pQFAqFQjEsKEFRKBQKxbCgBEWhUCgUw4ISFIXiBCKEuF0IIYUQt4/lMo6iLg+m61Iz2nVRjDxKUBTDhhCiJt14PDuEuOcJIX4mhFgnhOgQQkSFEJuEED8QQnhORH0Vx0/G3/zB0a6LYvRRgqIYLZYBfw90AA8A9wM24IfAW0II7+hVbczzDDA9bY8238GqS/1oV0Qx8thGuwKK05afAQ9JKZt6PIQQNuBJ4FrgLuA/R6luYxopZRfQNdr1AJBSNgANo10PxYlB9VAUo4KU8j8zxSTtlwL+I315/lDyyZwvEEJcL4R4TwgRFkK8nhFHE0J8UQjxvhAilDbvCCFuGCTPeUKIv6XjdQghnhJCjBNCvC6EkAPiDjpHcDTzB0KIzwkhnhdC7BdCxIUQrUKI54QQC472nrPNofTU/TDmhxlxLxJC/FEIsSOdb3f6ed0ysB7A3vTlZwbkV3O4ZyCE8AkhfpwuIy6EaEk/59lZ7ren7nYhxI8yntFmIcQnjvRsFScO1UNRnGwk03bqKNPdAlwMPAe83pNeCCGA/wM+BmwBHkrHvwp4SgjxD1LKX/RkIoQ4E3gDcAJPALXAecBbQPvR386QuRdYD7wCtAHjgOuAy4UQF0gp38uSJus9D8KD6TgDuRmYAUQy/L4JTADexxqqygeuAR4TQpRJKe9Jx1sP/AL4GrAByJw76xysIkIIF7AcWJgu4ymgEutvdIUQ4nIp5VtZkj4GzANewmq7PgE8KoTolFK+PFh5ihOIlFIZZYbFADWABJ49jjx+mc7jriHGvz0dPwWclyX8S+nwewE9w9+L1ZjFgfIM/7fT8a8akM/v0v5ygP+Daf+aLGUfEpZR39sHxB2fJf10oBt49SjvOWsZWeJ9FDCAVYD7CHXxYolGF+DJ8jd/cJAysj2DH6T9/gCIDP+lgAnsBLQM/9fT8d8BfAPiS+Cvo/3bV8YyashLcdIghLgE+DtgO1ZjczQ8I6V8M4v/XVgT//8opTR6PKWUYeBuwAHckC6/BlgMvCulfGlAPj/k6HtNQ0ZKuTeL31ZgBXCeEMKRJdlg93xEhBBzgEex5jeulVJGj1CXMFbvLgc461jKzOAzQAz4rkwrQ7qMlcDzwCRgSZZ035VShgbE3wccMiyoGB3UkJfipCA9dv4EEAI+JqWMHWUWq7Pk6QFmYQ1bfdca/epHUdqelrbnpu13BkaUUtYLIWqxhoKGHSHEJOC7wIVAOZbQZVLAoZPbh9zzEMsqAV7AmkO9Rkp5cEB4Dtaw17VY9ztwGXfZsZSbkfd4YJ0cMIeW5vV0uWcAA8VyXZb49UD1sdZHMbwoQVGMOkKIqcDfADtwmZRy4zFk05zFLwAIrPmIHxwmbc8SZX/abjlMGcMuKEKIycAH6fJfxVruG8Ia/rkOS+icg9TnaMtyYc11VAE3SynXDgh3ACuxGvQ1WENW7VhDY2dgNfbZ6jJUctJ2NjEBaBwQrxdprV4bSAq1uOikQQmKYlQRQkzBGtbxAx+RUr57jFllOykumLbfllKeO4Q8utN20SDhxVn8zLStZwk7pFEchH8A8oBbpZT/lxkghDibvp7TQI7ldLwHgHOA70kpn8oS3tM7uF9K+aUBdflWOvx46PmblAwSXjIgnmIMoZRdMWqkh3mWYzWmH5VSvjGc+Uspu4FtwCwhhG8ISTak7cUDA4QQFWQfWulZzVQxIL7G4EIwkIlp+4UBebixVjUNC0KIf8FaGfW/UsqfHE1d0mSb1+iZl8omqIcgpQxiLTWeLoTIJtxL0/b6oeSnOLlQgqIYFYQQE7B6JoXAdVLK5SNU1C+BXODX6eGegfWYKYQoBpBS7gPeAxYJIa4aEPWHZO/Rr0nbtw3w/ypDHx6rTdu9DXZ6ufO/kb1XdNQIIW7Guod3gc8fTV3S6W/AWhU2kI60XZElbDAeBlzAvw4o41ysIb7dWKvtFGMMNeSlGAnmicH3dnpFSvkoVs+kEuubj8VCiIG9gk7Z973D8fAbrB7Hp4ELhBDLscbpy4A5WMM7i+ibj/hKuk7PCCEyv0OpATYCAz+8ewbYD3xeCFEJfAicmc53JX1v3Ifjt8BngaeFEI9jLc09D2vy+nXggqO75aw8hDWftAH4TpYFCq9LKV/H6pnUAt8SQszE6uHNBK7AutfrMxNJKUNCiFXAUiHE77HEQAK/GWTOA6yPV68GvpxebfYGliDdgrX663NSSnOQtIqTmdFet6zMqWPo+ybhcOaedNwjxds3xDJvZ2jfXHwSq0fUgfXtSS3wV6xlyt4BcRdgTY6H0/GfTt/bh0BXlrwnAS9iTaQHsZa+TubovkO5GGt1WTfWh41PHW0ehwsfwvP+YUbciVji0ZKuz5vA5Yep+7T0s+zKyK8mHXZI/dP+fqwe2C4gkb7np4G5We7ndQZ8/zOUMGVOvBHpP4pCoTgM6TmYZmCTlPJ4v8NQKE5J1ByKQpFBer+oggF+GtYwjRtrmxOFQpEF1UNRKDIQQhQCdVh7au3A+kblXKwPJLcBC2XG19oKhaIPJSgKRQbplWA/x5rTKMP6iO8A1rzI3VLKkdwgUqEY0yhBUSgUCsWwoOZQFAqFQjEsnFbfoRQWFsqamprRroZCoVCMKdasWdMqpRxsS6JeTitBqampYfXqY9qgVaFQKE5bhBD7hxJPDXkpFAqFYlhQgqJQKBSKYUEJikKhUCiGBSUoCoVCoRgWlKAoFAqFYlhQgqJQKBSKYeG0WjZ8rKRSIcKR3Qh6zpAQWPsFCssIkQ4b4EZgnTtxqLvf9ZDSaPSdYZEZpiGEhhC2dJ0UCoVidFCCMgRCoW2sWXvLaFdjCAiE0NPGNsAezK+/vybsaLoTTTjQNCea5rCutUON3ut2oesebDYvuu7DZvOlbS9CDOlkWIVCcQqgBGUIeL2TmDvn94BEIkHKtNu0jg7q8UeCNDPiZIRJM8Nt+YNEZvoDyHS+/crpCTMzyulJLwETUxpImUKmbaSJmXGd3R7oNkgZYWSyA8OMY/aaRNqOpes9dDTNlRYYLzabD5stF7s9gMOej90ewO4IWLY9gMMewOEoxOEoVEKkUIxBlKAMgd0JJz+qrwZAQyBEehBK9A4+WXbGdV8YCCF6J6uESOeRET8zz574vQNdmWWIfgNm6AJsQqALgU2ArgnsQmDTMvyEwJY2PfEz0/S4nULg0jWcmsCpabg0gUuzru3CGl6zTmVLZQiNZQwzjpEKYRhhUkYYIxUiZYRJpUIYRihtW9epVBeh0BYSiQ5Sqc6sz1sIHYejCKezDJezFKerFKezFJezDLe7Go+nBpvNP1J/boVCcYwoQRkChoTOpEFvf0H2nXNqyt4+Q+81ve6MvosEM0t6KXv7Ohn+g5dh9vaIrHoZ0ko7kgjA1Ss0GaKjC3y6jt/mwKcX4dVL8Nt0fLqG36bjdWn4dR2/TSfPrpNv0ymw2/Do1nyQaaZIpbpIJjtIJDtIJttJxFuIxxuJx5uIxRsJhbfT1r4Sw4j0q5Pdno/HU4PbPQ6PuwavdzI+3zTc7io1l6RQjBKn1fb1CxYskKfiXl6mlKSkJJUWmFTaGJK0nT08ZVp+KSlJSkncNImbkphpEjMlcaPvOm4ODDeJGpKwYRAyTLpTlh0yDOLm4X9TTk2Qb7cRsOnk223kO2wU2G2UOe2UOu2UOeyUuSzba9ORUmIYIaKxeqLR/UQj+4hE9hGJ7ica3U883tibt6578Xmn4PNPx+ebTm7OXLzeqWiaendSKI4VIcQaKeWCI8VT/8tOATQhcAiBY7QrkiZhmpa4pPrEpjNl0JZM0ZE0aE+m+kzCYHN3lJZkkmDq0L6WX9coddqpcjkY7/ZR455HjfccagqdzHI7cGoahhEhHN5Fd2grodBWQqHtNDW9QH39owDouocc/xxyc88kN3ceubkLsNtzTvRjUShOeZSgKIYdh6aRr2nk24/u5xU2DBrjSRriyV67xxyIJXi/K0zY6BMdAVS47EzxuJjuK2Sa91Kml32UWV4XDiGIxero6lpHV3AdXV1r2V97P1IagEZOzlzy85eQH1hCbu4ZaNrJIscKxdhFDXkpxgxSSlqTKfZFE+yLxtkbjbM3Emd7OMbOSJxk+resC5jgdjLL5+bMHA9n5niZ5XPjIEYwuJH2jndob3+HYHADYKLrHgKBxRQXXUZh4cXY7Xmje6MKxUnGUIe8lKAoTgmSpmRPNM7WUJTt4Rhbw1E+7I5SH08CYBMww+vmjBwPZ+d6WRzwUajF6Oh8l/b2t2ltfY14vBEhdPLyzqao6DKKi67A6TzimUIKxSmPEpQsKEE5/WiKJ1kXjLCuO8K6YJh1wQjd6WGz8W4HS/L8LA74WJTrxZvYRnPLK7S0vEIkshvQKCg4j9KS6ygquhRdd4/uzSgUo4QSlCwoQVEYUrIlFOWdzhBvd4R4ryvUuxhgutfFJQU5XFKQw1TbQdqanqeh8Vni8QZ03Udx8RVUlN9CTs6ZGdvgKBSnPkpQsqAERTEQQ0o2h6K82RHitbYgH3SFSEnIs+lckO/nkgI/Z9l2Emp5hubmlzGMMD7fDCorPklp6TXoume0b0GhGHGUoGRBCYriSARTBivbu3m1LchrbUFakykcQrA0389VBW7mGisJNjxMKLwdm81PWemNVFV9Bre7erSrrlCMGEpQsqAERXE0mFKyLhjh+ZZOXmzupD6exC4ESwM+LvV3MiPyf3S3vIiUBsXFH2HcuDvI8c8a7WorFMOOEpQsKEFRHCvZxMWja1yZ7+R83qCw9V6k2U1+YAnV4+4gP7BEzbMoThmUoGRBCYpiOJBS8n5XmCca23m+uZOQYVLptHGpez9nhu6jILmZHP8cxk/4GgX5S5WwKMY8SlCyoARFMdxEDJO/tHbxREM7b3R0YwJnecKcn3ic2cm/kJ8zhwnjv0Z+/rlKWBRjFiUoWVCCohhJGuIJHm9o55GDbdTHkxToKS6Qr3Ke8SyTcquZMP5rBAKLlbAoxhxKULKgBEVxIjCkZHlbkIcPtvFqWxCQnCk2c6H5AktzbUyd9E/k5p4x2tVUKIaMEpQsKEFRnGgOxBI8erCNRxvaaEqkKKSNi+TL3FhgsmDSV/B6J4x2FRWKI6IEJQtKUBSjRdKU/KW1iwfrmni7K4pOirPlu9ycG2eJ/xriwVxC7XFi4SSJaIp4NEUyblhHPKf/i2q6wOG24XDZcLh13D4H3jwnvnwnvjwXnlwHmqaG0xTDjzoPRaE4iRApk/ldUN3pYWNTihe1KO8XnMU7QSfVXVtY1HCACavG40h4cXhsONw27E69VyCEACNlkogZJKIpEjEDOeAgM92uESj1ECj1kl/mJb/cS0lNDt4852jcsuI0RAmKQjECxCNJ6nd00rC7i4ZdnbTUdmMalgC4fHY+UeHj88LD6qJuntf8PF5+Hu5rI3w0J8hdU+Yx1Z972PyllMQjKUIdMULtcUIdMTpbonQ0hGnc3cXOVU29cb15Tkpqciiu8VM6PpeSCTnY7PqI3r/i9EQNeSkUw4A0Ja11IfZvaqN2cxuNe4NIU6LZBCXjciiblEvpxDyKx/nx5Dj6rfSSUrKyYTP37d3EG/GJGMLOQk+EO2qmckVRPvZjGMZKxFK0HwzTtC9I874gTfuCdDVHAdBtGiXjc6iYkkfFlIASGMURUXMoWVCCohhODMOkflsHu9Y2s+/DNqLBBABF1X6qZ+ZTPaOA4hr/UTXWu5rf4/6db/NSfA5toogiW4rbKyv4VHkhJU77cdU3Fk7SuLuL+h0d1O/opOVAN0hLYMom5VI9o4Dqmfnkl3vV0mZFP5SgZEEJiuJ46RWRNc3s2dBCPJzC7tIZN6uAcbMKqJ5RgCfn+I4TllLS1PIqT+x8mRfjs9kozsSG5CNFedxeUcjiPN+wNPjxSJKDu7qo397Bga3ttB8MA+DNdVA9s4DqmQVUTgvg8h6fkCnGPkpQsqAERXEsGIZJ3bYOdg8QkfFzCpk0v5iqGfkjMmQkpUFT00u8tfsRXorP5k1xKSHcTPG4uL2igJtL8/Hbhq/cUEeM2i3t1G5uo25bB/FICiGgZHwO1TMLqJqRT/G4HLWS7DRECUoWlKAohoqRskRk19pm9q5vIR5J4XDp1MwtZNK8kRORbJhmkoaGp9i297esTExiue0mdhmleHWNm0oC3F5RyHTf8J4maRomTfu6qd1szQk111rDY06vjapp+VTPzKdqegG+gFpBdjqgBCULSlAUh2MwERk/t4iJ84upnp6PbtdGr35GnPqDj7Jv32/Ylgyw0vFJ3kzNJiEFC3I83Fiaz0eL8ih0DP/izWgowYGt7RzY3E7tlnYi6fmi/HIv1TOs+aKyyblqcv8URQlKFpSgKAZipEwObG1n95pm9m5s7Scik+YXUzXKIpINw4hysGEZ+/ffT2u8m/ecn+BNcTG743Z0AUsDfm4oCXBFYS6+YRwS60FKSVt9mNotbRzY0s7BXZ2YKYnNrlE+JWAJzMx88ko8anL/FEEJShaUoCgAjKQlIrvWNrN3QyuJaAqH28aEuYVMnHdyikg2TDNBY9Pz7N//WyKRvTQ5FrDe+xmWR6uoixu4NeukycsLc7mkIIcix8hMrifjBvU7Ojiwxeq9dDZFAPAFnJRPyaNicoDyyXnkFruVwIxRlKBkQQnK6UsslGT/5jb2fdhK7aY2EjEDp8fG+EwRsZ38IpINKQ1aWl7lQN1DdHa+D8JFW/7nWGW7lOVdOvXxJAJYmOvl8sJcLsr3M83rGrHGPdgapXZLO3XbOji4s4NodxIAT66D8sl5VEzOo2xyHvmlXoSa4B8TKEHJghKU0wcpJZ1NEfZubGXfxlYad3chJbhzHNTMKmDivGIqpwXGrIgMRndoG3V1D9PY+BymGcPrnUF3wSdZw1m82pFkU8j6uLHQbuPcgI/zAn7OC/iodo/M5HrP3+Hgzk7qd3RycGcn4c44AA6XTnFNDsU1OZSkjdom5uRECUoWlKCc2kSCCeq3d1C3rZ267R0EW2MAFFT6GD+nkJrZhRSP858Wb8XJZCeNjc/R0Pg03d2bEMJGQcEFmIGr2Szm8W4wxRsd3bQkUgBUOO3Mz/Uyz+9hfq6X2T43Ln34xVZKSbA1xsGdnb1f8LfVhTDT+5J585wUj/NTUOmjoNxHQYWX3GKPWqo8ypyygiKEqAJ+DlwKCOBV4B+klLVHSqsE5dQiGkrQuCeYFpEO2upDADjcNiqm5FE1PZ+aOYX4812jXNPRJRTaQUPj0zQ1Pk880YQQNgJ551BYeBkd3vP4IOJmVVeYNcEwdTFreMouBNO9Lqb73Ez3upjhczPd5xqReZhU0qD1QIimfUGa9gZpqe2mqzlCT9Nks2sEyrwUVHgJlHnJK/aQW+wmt9CNzaFWlZ0ITklBEUJ4gA1AHPge1sbePwY8wBwpZfhw6ZWgjF1MU9J+METjniCNe7po3NPVb2+qskm5VE4LUDk1n6JqH9oIvF2PdaQ0CQY30NLyCs0trxCN7gPA7R5HfmAxgfzFJDzz2RR1sjoYYVN3lC3haG8vBqDAbmOix0mN20GN28l4t5NxaXfApg/bvEwqYdDRGKG1LkTbwRDt9SFa68O929sAIMCX5yQ3LTA5BS58ARf+fCe+gAtvnvOUG9IcLU5VQfka8DNgqpRyV9pvPLAT+KaU8meHS68EZWyQiKVoqw/TVtdNa13IalTqQ6QSJgBuv53SCbm9pnicX72pHiVSSsKRXbS3v0VHx7t0dLyPYVg9PJerkpycOZbxzybmmMDuhJstoSjbwjH2RuPsiyZoiCf75enWNEqdNkqddsqcDkoddkqdNkqcdgrsNvLtNgJ2nYDNdszDafFIkq6WKJ3NEbqao3Q197lj4eQh8d05DvwBJ948J+4cBx6/A5fPjsfvwO234/Y7cPsduLw29RJyGE5VQXkNcEkplwzwXwkgpVx6uPRKUE4epCkJdcatxqApQmeT1TB0NIZ75z4AnB4bhZU+Cip9FI/LoXRCLjmFI7dC6XTFNFN0d2+ks2stweAGgsGNxGJ1veF2ez5ez0S83km43dW4XOXgKKdJFnHQ8LE/luRgPEljpkkkiZvZ2xePrhGw6RTYbeTZdXy6jtem4dN1fLqWcZ32s2l4dR2nJnBqGq607dQErrSdihuEOqyt/C27zx3ujBPtThALJRmsybO7dJxu6yyaHjvT7fRYZ9TYHDo2h4Y9bVvXh/ppujhlfqen6gFbM4HnsvhvBm4+wXVRHIZU0uj7T90eI9QRo7vXHSfYGu3tcQDYHBp5JR6Kx+UwbVEZhVV+Cit9+ALOYflP2XvyoWnZvddSgimtRqbXLx1upMPMtJ15bWT4916b6Wv6XWePO8h1uozeOph99c6sozRlX31Necj90fPIhLDcgr7n2DPBLaxwkbbBiVcswisWU65BUu8i6thNzHaAuK2OWKSWxo4XMbTufs/WLnWmywCzycEmc7Hhx0YeusghTC6d5BAUXoLCQ7dwEcRFl+Ggy9TpSmh0odGAICwFYSAMpDh67IBTgBOBU4AjR2DLAX2chk24seFGB3Qp0UyJMCxbMyTCMNEME2HKtB1DpNJ/v6QJcRPZIRESNKT16CQIpGV63SCk7IsjJLoALf2IdQGakGjCutaEQGRcCwGa1vPnkdbfJ10G6fwEEiFkb5weP4lVriT928BM/5hNTGlS7fRw63U3HcOTHTpjTVDygY4s/u1AIFsCIcQdwB0A1dXVx1RosjFM54t7sgdmvu4M8ubT/43oyPEPGzbY69Vg2R6uBzqUMtINmzSteQyZbrykKTEN2etnGv3tgcV6AJ8m0PS0yXeg2zR0Xev1AyAYs8y2NqtxyWz8+4lBhiD0NrAZ11Km/z/Jwz/n0UIAmrBWnGkCoYt+12gZDX26NRI94qBZtuhthdLhOmDT+sSk577TvYTe52jI3r9xfyHts632yI5bTsMtp6avrXQpLULS3krK0UbS0UrS0UbK0YFhC5Gyh4jZGjDtIQx7GIREx/rPGcis00D3AO8kdmK4ieEmiosYbuK4SGAniZ0kjn52osct7SRwkMRBCjsGGgY2DDRMdOLoGD1G6Ji6TkrXMTP90THT6Uw0JDo9EtFjTESfnxgbQ2VnR9Zy6wiXMdYEBbL/DAd9hZVS3g/cD9aQ1zEVKEEmzcEjDFa6yHaRbiiypTvkWvT954Z+dk9DIMFqQOlpSHvi9bzRgin7Gvk+QbDcZo+7RyTSb9iGYfYJxKAPxqqzZtOw2QSaTUN3aZZQ2C1jc2jY7NYwgNbTAB7VM8xoWDNsoQ24FqK3oe2XRsuSh3ZoniKjoe4rQwNdILR0mt7rjMZfP4woHCbsdFi6LKVBKhXBSEUxjQiGESWVjGAaUQwjgmkkkNJAmqle2zRTSJnq85MmIPr1Hnt/kBm/pf5PU/R3SwMSIUQyDMkopGKQjCNSMUjGIBWHVBRScUQyipGIEzEhiI2QaSdqOIhLG0lpIyVtlkAJG6bQMYSGKTRMTUPKtMgIkX6Xsdz0ClCmAaRli4zOhFVfSP+n7rt3eu7d6rFImXGPA0RaCJEuN30NSKHhEza46hj/mENkrAlKB1YvZSABsvdchoWgYfJmd8pqdDPe4A65Ngfxz3iLzgzv9c9MMzDOIGPQx8vAxt5m17G5NGx2Dbur/7ixw2XD4dYPGVv2+B04PbbTonFUHD1C6Njtfux2/8gVkoxC5wHorIWuWssONkCoqc9E2nqjS6ALD7v1fGplAa1mIeGUn4RZSEq6MXQXhm2QjysNA2GkMoyBMA0w07ZhWOIlpTXgpWkImw3d4cDm8mDz+HH5c3HnFuILFODLycPn9+LzuvF5Xfi9HnxeNy5n/xM9xxJjTVA2Y82jDGQGsGWkChVCYHNoiJ43W6y3VpF+sxUZb8A9/kD2OJro84Pet+V+8XreqiE9HNQ3LJTp1nuubYP4p6/7iYbD6kEoEVCMGRJhaN0JbbugdYfl7kyLR7i5f1zNBv4yDG8xuz0T2Oo+gwMRB6GgIBmzYUonUrOD0ZdEN+PoyQSkkmhGF3oqgZZKIlJJpJHCNCGpOxGefLRAMa7iYnILC8gvLKCoqICy4gLKSvLxusauEAwXY01Qngd+KoSYIKXcAyCEqAGWAN8eqUILxE6u1b9oXaS7sL1dUyky/DLCe39XGeGZY11Z0ww1nONLLzTLaDoIPcO2WTOC/fz0LPEG87OBzQm6E2yOAbYTdMcAOx1uc1l+p/l/xtOeRASat0LjRsvuEY9g32ozhAZ51RCogalXQF41cV8FayMpNrV20tjUTrIlguh0IoS1lFyYJs5oBE8shkx1YRgpSMXREnFEMkFS2Ag58jDySvAUlhAoLae0qoKammomVpUQ8CqhGCpjTVB+B3wFeE4I0fNh493AAeC+ESvV7oXS2fSOT/UMWva4+01iDxY+WJqjyFPKY0ifJdwaSwMzZXXRzXRX3Ryi30ig2cDhtZ61o8f4Mtye9LUP3AFw56XttHGlr+2n91fxY4ZQiyUcjR/2mbad1u8SwOGHwslQs8SyC6dA4RRkYDx7uw7yzpZ32L13N+HtzbjCXWjppswRi5HbHcIWbcYwEsRlEhIxhJSk0GlzFmDklpEzoYrymvFMmTaFGRMrKPQp0RgOxtR3KABCiGr6b73yGtbWK/uOlFZ9hzJMZBUew/JLxcFIpO04pBID7MzwRN+EaCICyQgkQtYQRyJs+fW7DkOiu6/RyYbNbQmLtxB8JeAvAV/BQwPFAAAgAElEQVTpAHcx+MuU+JwoUglo+hAOrIK6VVD3gTVc1UNutfXClmnyqkEITGmytXUrb215iz179hBvjpMTzUFDQ5gmeR0d+Lo60WNhopokLFPpJbTQZs8n6C8nr3oi42dM54w505hVGcClDgE7ak7V71BI79l142jX47RG00BzjE7ZUkK8G6Idh5pYp2VHOqyx9VCT9eYbbs4uQr5SyKuyGq8ek5vhVoJzbAQbLNGoW2WJSMN6a2UVQE4FVC6As+6AsjOgdJb1ApDBwdBB3t76JGu2rqG9tp2C7gKcphO3dFESSlBSvwNndydRp06rQ6dTmkgBLfZC2nLGUTh5BnPmn8HNMyqpzleHfJ1IxpygKE5zhABXjmUC44aWxjSslT7djRBqhlAjBA9C537rTbl+LWx5HszMrTuEJSqFk6FgkmUKJ0PBZMgpV/M9PaTi0LCxr+dRtxq6DlhhusMSjYVfsESk8izIrTg0CzPFuuZ1rNi/go1bN+JqdlEaLcUlXVRSQnlbG6U7d2MkorQXBWjEAJ+NkO5hn6cGWTGF2Qvm88kzxzOrPFftTDyKKEFRnPpoujXM5SsePI5pWILTdQA69kP7HmtMv20X7H8Xkhn7jto9UDARCqdaY/tF1vg++RNP7V6NlNBVlxaP1ZbdsN4augTIrYLKhXDOnVB1ljV0NcgS3FAixNsH3+b12tfZsGsDBR0FVIWrmGROwqZDTThEyZp1pGJhmsqL2J5rB+y0OvLY46nBNn4WFy2Zxx1zyinLdZ+4Z6A4LGNuDuV4UHMoimNCSuhu6Fu62rt8dYf1DUTPggehQd44KJqankjOEBx31o0cTm4SEUsw6lb1DV+FGq0wm8vqfVQttHoelQshp+yw2UWSEVbWreTlvS/z7oF3KQuWMSU0BV/ch6YLJni8VK3fgLFjKweLAzR7XEgkXY4AWz2TiFTO5orFs/no3HJqCr0n4AEoejglN4c8XpSgKIadRKS/wLTugJYdlp8R74vnLUoLzOT+gpNT0bd502gSaT901VXL9r5VfYHxlmhUnWUNX5XMAv3IZ6PEjThv1b/FX/b+hZV1K9GjOrMis6gIVkAKSoqLmWkYeJ59hvpUjLqSfGICUg4vG92T2O2fwrwzZ/LJc2pYNKFADWeNEqfspLxCcVLh8EDZHMtkYhrWHE3rTqth7hGbzc9Yiwd6sHuhcFJ6WezUvjkbf5nVqxkuselZzNDdAO17+3pa7buhdRd0H+yL6y+3hqumXmmJSOUCa9XckIuSrG5azXO7nuO12tcIJUNUySquil2FbJTous6MadOY2txCx5NPsNcpaCn0Ah468ifwjm0yXUWT+fTiCfz8rGqKc07hYcRTDCUoCsVIoOmQP8EyUy7v85cSwq1pgdlu9WZad0Dte/DhsgF52MBb3Df/4/T3fYvj8FofhWbuWyWlNdcTD1nLrXtWw3U3WiY54Pw5d8ASr/HnQ8nMviW7RyEemRwMHeT53c/z3K7nqAvV4bV7uSTnEkoaS2ira8PhdLDw3IVM7exkxwP387YNQkU+7C4Pewrm8oaYSG5xMXecP4Gb51fhVmfcjDnUkJdCcbLQs8VIx15rNVrvqrQma+lzj1AkwpadDc2WFh4/OH3WB5/+UqvHk1NmLZUO1FiLCjzZtsU7OqKpKK/Vvsazu57lg4YPkEjOKj2Ly/IvI7Y1xt49e/F6vZxzzjlMMk3W/voe9iRjJOw63rxC1pWcw9/i5VQX+vmHS6Zw9ZwybOqgq5MONeSlUIw1HF4oP8MyR6Ln49KB6PYTsqR5R8cOntj+BC/teYlQMkSFr4K/m/t3nJ9/Pls/2Mqmdzfhdru59NJLmV5VyXv/8WMeb67H0DRKKqp5r/pCXmz1Uepy85OrJ3PT/ErsSkjGPEpQFIqxyCh8XBo34ryy7xWe2P4E61vW49AcXFZzGddPup5ZebN4Y+UbPPX8U+i6zvnnn8+Zs2ay+ne/5uH1qzEEjAsUs3fJJ/jRDgNnSOc7V07iM4tr1JfrpxBKUBQKxWHZH9zPkzue5Nldz9IZ76TaX803FnyDaydeS44jhw0bNnDvn+4lHA4zf/58zpk/j81/fZGH7r8HwzSokhquKz/D3QfzaNka5+b5lfzTFVMp9qvJ9lMNJSgKheIQkmaSlQdW8vj2x3mv4T10oXNR9UXcPOVmzi47G01o1NfXs+zPy6ivr6eyspIbr7uWug/e5tFvfoVUIkF5Z5gZSy7i3qpLeWV7O3MqXfzutgWcUZU32renGCGUoCgUil7aom0s27GMZduX0RxtpsRTwl1n3MUNk2+g2GPtNJBIJFixYgXvvfceHo+HK6+4nOiurbxw93cw4nHKOkNMNW0c+PQ/8andNpJ7O/neVdP57JLx6Oo7klMaJSgKhYKtbVv509Y/8fLel0maSRaXL+Z753yP8yrPw6b1NRN79+7l+eefp6Ojg7lzZpMb6uTdX/8UI5Gg2ulj3LadBJYs5SczrmfFliiLJuTy7zfOZlyB+rL9dEAJikJxmpIyU6w4sII/bfkTa5vX4ra5uWHyDdw67VYm5E3oFzcWi/G3v/2NNWvWkJeby5kVJex79lGMRJLJc+dR9fZqnBs30v7JL3JjcjrJ5gT/dv1sPnFWldrt9zRCCYpCcZrRFe/iqZ1P8di2x2gIN1Dhq+AbC77BdZOuI9eZe0j8ffv28fTTT9Pd3U11IIfg2rfZFY8zfclS5kyaQfgHPwJd58+f+Wd+0ZHLnEovv/j4mYxX+22ddihBUShOE/Z07uGRrY/w4u4XiRkxFpYu5FtnfYsLKi9A1w5duptKpVixYgVvv/02brsNf90uOrZ2Mm3x+Zxz48exrd1Awze/jSyr4F8WfZ7VnW7uvGAi/3jpFPVNyWmKEhSF4hRGSsmapjU8uPlBVtatxKk7uWrCVdw67Vam5k8dNF1LSwtPLnuCpuYWHMF2bA21TFt8Lmdffwv5FZW0/e73HPzZz4hMn8MdUz6G7sjl0dvOZNHEghN4d4qTDSUoCsUpiGEavFb7Gg9ufpAPWz8k4Axw59w7uWXaLeS7Bt9yRUrJ22+sZPnrryNTKdyN+5l75jzO+sa3yC+vQEpJ809/SvsfHmDfGefy1aqrmTehmF/eeqb6rkShBGUopJJJIl2dfZOLAgQChOj1E0L02/Ki59qKl76mJy79wvrcGeGk885SlkIxGNFUlGd3PcvDmx+mLlRHlb+K7539Pa6ZdA1u2+EPomrcu5snH3+M1lgSWzjI3Joqzr3rTgKl5QBI06Tpxz+h49FHeXf2hdw97iN8cekkvnn5VLX/lgJQgjIkGnfv4PEffGu0q9GfLGLVI1TQJ0II0DQNTdMRmoamW3aPW9M0hOjz7xeupcO1AeFCQ7fZ0O12y9gs25ZxnenWHQ5s6Ti63Y7d6cLhcmF3ubG7XDhcbnS7XQnmcdAR6+DRbY/y2LbH6Ix3MqdwDl9f8HUuqroo6/xID1JKaj/cwDsvPsPOrgim20tVjpcb/+5L5JX0HZglDYOG732frmee4aUZF/Pg9Kv59c1z+cjswx+qpTi9UIIyBAKl5Vx6x9+nr6T1T0pAYlkS2XNqn0z7WZF63b27Ost0TNnnJ6UVt4eea5mtLAbJW/bW4JC8pWliGgbSNC23aaTtPn/TMJDS7HWbZp9bplL90kjDwEilMFJJjKRlUqkkRiKJlOYxPWOhaTjSAmN3udOC48Lh9uD2+XH5fDi9Plw+v2W8Plw+X9q2rsXJcFDVCaYl0sKDmx9k2Y5lRFNRLqi6gNtn3s684nmHFeh4JMzWN19n46sv09DaRqxyErrXz/XXXMPsM/pvTilNs1dMHp1xOSvPvoZnbl/I5BL/SN+eYoyhBGUIePMCzLn48iNHVGAaRp/AJJMYyQSptOgYadFJxmMkYjGSsWivbflFScZi/fy6W1to2b+XWChEMhYdtFyhaXhy8/DmBvDm5eHJC+DNC+DN7XP78gvw5xdic5zYTRVHgoZQAw9seoCndz5NSqa4cvyVfGH2F5iYN3HQNFJKGnZuY+Orf2X7u2+STMRxTJpBdFwhRUVFfPzjH6egoOCQNI0//gldzzzDn6Zeys7LbuHZT88n3zv2n6Fi+FGCohhWNF1H03XsDP8ErZFKEg+HiYa6iYdDxEIhYqFuYqFuIsEuwp2dRLo6CHd20FK7j0hXJ6ZhHJKPJzcPf0EROYVF+Av7bH9BITmFxXhyck/a3k5tsJY/bPoDz+96HgRcO/FaPjfrc1TnVGeNL6WkZf9etr/zBtvfe4uupkbsLjdTl1xAs9PLnv21zJgxg2uvvRan03lI2ob/+m+6Hn2UpyYtJf7Jz/HIDbNx2tTuwIrsKEFRjBl0mx1Pbh6e3KFtLihNk1g4RLizg3BHB6GONrpbWwi2tdDd2kJb/QH2bVhLMh7rX47dTk5hETlFJZZdWExOcdpdVIwvvwDtMPMSI8Gezj3c/+H9vLz3ZWzCxs1Tb+azMz9Lme/QOYxUIkHd1k3s27CWPWtX0dFQj9A0qmfN5ezrP0bRlBk89exztNYe4NJLL2Xx4sVZh8fqfvlrQg/8gZfGL6LoG9/gn5dOVPNcisOiBEVxyiI0Dbc/B7c/h8KqcVnjSCmJh8MEW5vpbmsh2NpCsKU5bTexe81eIl2d/dJouo4vv5CcoiJyi0rwFxb3unMKi/EXFqLb7MNyD7XBWn694df8ec+fcdlc3DbjNm6bcRtFnqLeONFQN407t3Nw53Yadm6jftsWUok4ut1O5fRZzL/qOiafvRhPTi7bt2/njw8/gqZpfOpTn2LixOxDZPsfeYzIr3/Fiur5zPqPH3PFnPJhuR/FqY06AlihOALJRNzq2bQ0p8XGsrvS7lB7W79FFQiBL5Bv9WyK0ibt9uTm4fJ6cXp8ODzuQXs6DaEG7tt4H8/uehYHdm6pvoFrSi5Di6QItbfRXl9H+8E62uoP0N3aki5Wo7B6HJUzZjF+7nwqZ8zC7rSGHk3TZOXKlaxcuZKysjJuueUW8vKy9/R2vfwa0a9/lU0lk6m57z4WTS0Z3geqGHMM9QhgJSgKxXFipJJ0t7WlBacpLTgtve7uttasczkADrcHu9PZuyTbRNKd7CYSC2EzNBymDWEcunLO5nSSX15JfnklhVXjKJs8jdJJk3G4Dv3WJBqN8vTTT7Nz507mzp3L1Vdfjd2evQe1+c3VxO/8As3eAioeeoTZU1XPRKHOlFcoThi6zU5eSSl5JaVZw03TINTeTrC1mWh3kHg4bJlIiFg4RCqRIJ6MsaN9O/s69iLtksryKmaUziHXG8DudOHNy8Obl483EMAXKMAXyB/SwoGmpiYef/xxOjs7ufLKK1m4cOGg8yCrVm0n/tWvIOxuJv3hfiYpMVEcJUpQFIoRRtP09OR+0SFhoUSIR7Y8wsNbnibsC3PlBVdy59w7B121dTRs2rSJ5557DqfTye2330519eB5Lt9QS+Tvv0J5Kk7RAw9SNXPw5ccKxWAoQVEoRoFIMsJj2x/jgU0P0BXv4pLqS7jzjDuZHJh83HkbhsFrr73GO++8Q1VVFTfffDM5OTmDxn92bR313/ouF3TWE/j5LyhfOPe466A4PVGColCcQBJGgmU7lvG7jb+jLdbGkool/P2Zf8/MgpnDkn84HObJJ59k7969LFy4kMsvvxybbfD/5g++vZf1P/8tXzqwhpy7vkL5Ry4dlnooTk+UoCgUJ4CkmeT5Xc/z242/pTHcyIKSBfzsgp8xr2TesJVx8OBBHn/8cUKhENdeey1nnnnmoHGllNzz6k5WPvZnfrL5RTwXX0z5XX83bHVRnJ4oQVEoRhDDNHh538v8Zv1vqO2uZXbhbH60+EcsKls0rB8Jrlu3jhdffBGv18vnPvc5KioqBo1rmpIfvbCZl19dx33rHsU1YTyV//EfJ+3uAIqxgxIUhWIEkFLyau2r3LvuXnZ37WZKYAr/c+H/cEHVBcMqJKlUir/85S+sXr2a8ePHc9NNN+H1Dn70btIw+cayDby0tpaHti/Do0mq7v0Vuk8d16s4fpSgKBTDiJSSN+vf5FfrfsXW9q3U5NTwX0v/i8vGXYYmhrcHEAwGWbZsGQcOHGDx4sVcfPHF6PrgW8JEEwZ3PbqW5duauS/yHvm1Oym75x4cNTXDWi/F6YsSFIVimHi/4X1+ue6XbGjZQIWvgp+c+xOuHH8lNm34/5vt37+fZcuWEY/Huemmm5g1a9Zh43dFk3zhoVWs3t/Br2rCVN/zLIFbP0HOFWoXbcXwoQRFoThO1jWv49519/J+4/uUeEr4l0X/wnWTrsOuDc9+XplIKfnggw/461//Sl5eHp/+9KcpKTn81igt3XFue+ADdjV385tLKxj/nS9jnz6d4m+dZIfGKcY8SlAUimNASsn7je9z/8b7WdW4igJXAd8+69vcNOUmnLrzyBkcA4lEghdeeIEPP/yQyZMnc8MNN+B2H/5Y3wPtET79h/dpCsb5wyfPpPpfv048maTy5z9Dc45MPRWnL0pQFIqjoGeO5L6N97GxZSPF7mK+tfBb3DjlxiOe2X48tLW18fjjj9Pc3MyFF17Ieeedh3aEVVnbG7u57YH3iSVN/veLZ1P17CO0rl9P+X//VM2bKEYEJSgKxRAwTIPlB5bzu42/Y2v7Vip8FXz/nO9z3aTrcOgje3rhtm3beOaZZ3q3nJ80adIR06yt7eCzf1yF06bxxJcWUdWwi/2/vY/c664j96qrRrS+itMXJSgKxWGIJCM8s+sZ/rTlT9SF6hiXM467l9zNVROuGpE5kkxM02T58uW89dZblJeX87GPfWzQLeczeXNnC196ZA1Ffid/+vzZVDgle277NvbSUkr++bsjWmfF6Y0SFIUiC43hRh7d9ihP7niS7kQ3ZxSdwdcXfJ2Lqi5CPwGnNYZCIZ566in27t3LvHnz+MhHPjLolvOZ/PnDBr722DomFft56HMLKfa7aPiXH5A8cIBxDz+E7vePeN0Vpy9KUBSKNKY0+aDxA5ZtX8by2uWYmFw67lI+PePTzC06cRsm7t69m6effpp4PM4111zDvHlD257lT+/t5/vPbWJ+dYA/3L6QXLed7uUr6HziCQq+8Hk8CxeOcM0VpztKUBSnPe2xdp7b9RxP7niS2u5acp253Dr9Vm6dfisVvsG3MBluDMNgxYoVvPXWWxQVFXHbbbcdcUkwWAsF/ue1Xfz81R1cPK2YX906D7dDJ9XWRsP3v49z2jQKv/rVE3AHitMdJSiK05KkkeSt+rd4cc+LrDiwgqSZZF7xPL4898tcVnPZiC39HYzOzk6efPJJ6urqmDdvHldccQUOx5En+430vlwPv7ufG+dV8u83zsauW6u/Gn/0r5jBIOV/fABtCHkpFMfLUQmKEELIUTwzWAgxBbgLuBCYAHQDq4DvSyk3jFa9FGMDKSXrW9bz0p6X+Mu+v9AV7yLgDPCxqR/jpsk3MSlw5NVTI8HmzZt54YUXkFIO6av3HuIpg68/sYGXNjbwpfMn8O2PTOvdJyz4l7/S/corFP2/r+OaMmUkq69Q9HK0PZTXgaVCiLuB1cBqKWX9sNdqcC7DEpOHgLVAHvBN4H0hxBIp5ZoTWBfFGCBlpljXvI7ltctZXrucg+GDuHQXF1ZdyNUTr2ZR+aIRX601GNFolD//+c98+OGHlJeXc9NNN5Gfnz+ktKF4ii8/soa3drXy3Sunccf5fScspjo6aLz7blwzZ1Lw2c+OVPUVikM4WkG5Im1Hgc8AvxRC2EmLC/CGlHLFMNZvII8B92b2koQQy4F9wNeA20awbMUYIZgI8kHDB6w4sIKVdSvpinfh0BwsKl/EnWfcySXjLsFrH93ddXft2sVzzz1HOBzmwgsv5Nxzzz3sxo6ZNHfH+MJDq9l8MMh/3zyXG+dX9gtv+rf/DyMYpPqBBxCHOVxLoRhujvhryxzmklJG0/a/ZYQXAQuBBcCPhRAbpJR3jkRlpZStWfy6hBA7gBM3e6o4qUgaSda3rOe9hvd47+B7bGrbhClNchw5LK1cyoXVF7KkfAkeu2e0q0oikeCVV15h9erVFBUV8YlPfILy8vIhp9/R1M1n/7iK9nCC3902n4um9Z+0716xguALL1B41124pqqhLsWJZSivL7uEEL8AHpBShgYGSilbgD+nzb8KIdYOcx0PixAiH5gF/PFElqsYPTpjnWxo2cCGlg2sb1nPptZNRFNRNKExq3AWX5z9Rc4pO4e5xXNHbTgrG7t27eLFF1+ks7OTRYsWcdFFFw3p25Ie3t7Vypf/tAaXXeeJLy1idmVuv3AjGKTxBz/EOXkyhV+6Y7irr1AckaEIyjPAD4AfCSF+B/xSSnngMPE/Niw1Gzq/BARwzwkuV3ECaI+1s719Ozs6drCtfRubWjexL7gPAF3oTMufxvWTrufssrNZWLoQv+Pk+3AvFArx17/+lQ8//JCCggJuv/12ao5yL60nVh3gu898yMQiHw98diEVeYfuG9b8X/9FqrWVynvvRahVXYpR4IiCIqX8hhDi+1hzJncB/yCEeBr4bynlqizxdw21cCHEJcDfhhB1pZTygizpvwPcCnx+sHKFEHcAdwBUV1cPtWqKE4iUkuZIM/uD+9nfvZ/9XfvZ3bWbHe07aI4298YrdhczvWA61066lrlFc5lZMPOkGMYaDCkl69ev55VXXiEej7N06VLOPffco+qVmKbkp69s59ev7+a8yYXc+8l55LgOTR9+9106lz1JwRe/gHv20FaJKRTDjTjaVcBCiIuBrwJXAu8BP5NSPnNMhQvhAYbSykeklLUD0n4Z+A3wPSnlT4ZS3oIFC+Tq1auPvqKK48IwDVqjrTRGGmkM95mmSBO1wVpqu2uJpqK98R2ag3G545gWmMbU/KlMCUxhav5U8l1DWwF1MtDQ0MDLL79MbW0tVVVVfPSjH6W4uPio8gjGkvzjY+t5bVsznzirin+9dlbvNyaZmIkEez96DRLJhOeeQ3O5hus2FAoAhBBrpJQLjhRvKJPy5UBggFkOBIGPA08Cx7S5kZQyAmw72nRCiE8Dv8bqJQ1JTBTDg5SSaCpKd6KbUDJEd6Kbzngn7bF22mPtdMQ6+tntsXbaom2kZKpfPm6bm1JvKZW+ShaWLmRczjjG5YyjJqeGEm/JsB+Xe6IIh8O89tprrF27Fo/Hw9VXX828efOOuNX8QHY1d3PHw2uobY9w97Uz+dQ54wY9i779D38gsX8/Vb//vRITxagylDmUOsAEQlgfEnZhiUkX8HTaPmEIIa7HmoD/vZTyGyey7JMJKSUpmSJlWsYwjd7rpJnM6hdLxYgbcWJGjHgqbrkH+MUM6zqeihNOhQklQoSSIUKJEN3JbsLJMKY0B62X2+Ym35VPwBmg0F3I5MBkSjwllHpLKfWW9rpzHDmDNpBjkVQqxapVq3j99ddJJpOcc845LF269IgHYGXjb1ua+MfH1+Oyazz6xXM4a/zgPbNEXR2tv70P/xVX4Dt3yfHcgkJx3AxFUNYBU4AHgXuklPtGskKHQwhxPvB/wEbgQSHEORnBcSnlupEod3fnbu5Zcw8mJlLKXrufG4kp0+HSRCL7ufv5cYR4A/IzpNFPPFJmCkMaw3qPAoHL5sKpO3HqTlw2Fx6bB7/DT6WvEp/Dh8/uw2v34nf4e699dh8BV8ASEVdgRA+ZOhkxDIONGzfy+uuv09XVxaRJk7j88sspKio66rxShsnPX93BvSt2M6cyl99+aj7lWSbfe5BS0nT3jxG6Tsl3vn08t6FQDAtDmZSfL4S4EPh/wHYhxLNYQ00fjHjtDuUiwAmcCbw9IGw/UDMShSbNJE2RJgA0oaEJDYFACHGIWxMaCNA41H1IvJ78SMdLu4UQCNJ5C4EudHShY9Ns2DU7Ns2GrunYhA2blmEGXOuajl3Ye697hMKlu3DanP1su2Y/pXoMI41pmmzdupXly5fT1tZGeXk511xzDRMnTjxy4izUd0b52v+tY/X+Dm5ZUMWPrp2Jy374keTQ8uWEVq6k+JvfxD6ETSQVipHmqCblhRDTsYTlk8Aa4KfAc6O5v9fRoCblFceLYRhs2bKFt956i6amJoqKirjooouYNm3aMQvyK5sb+acnN5IyTP7thtlce8aRv9E1IxF2X301utfH+KefQhzFyjGF4mgZtkn5TKSUW4UQXwT+E/gu8ARWz2DyMdVSoRgjJJNJ1q9fz9tvv01nZyeFhYVcf/31zJ49+6gn3HuIJFL8+8vbePjd/cyqyOFXn5hHTeHQtoRp/e19pA42UPG//3979x4fd13vefz1nUsuM7k3tyZNb7SF0ko5UvWUS7EqbNc9sgeEFXflqkBhK4gIxwUUWqBAj8hWRIWzhxYUhT2oILssPihVOEcKWMHULcTeSJu0zaW5TTJJJnP5nj8mSdPS0DSdyW8m834+Hj9+87vMzId+k3nn+/vN7/v7mcJEUsZYvuX1JyAPyAcKgKGDukN/js1OTmkizgsEAvzpT39iy5YtBINBqqurWb58OfPmzRt3kAC8tbuNW5/byt72Xr569ixuW34y2Z6xfVkytGsXbevXU3jhhfjOOGPcNYgk2lh6KO8AHceYRCYNay319fW8/fbb1NXVYa1l7ty5nHnmmcycOfOEzjX1DkRY+/Jf2fBGPdNLfDx77d/yqdlTjqu2ptX34MrNpfxbt4y7DpFkGMtJ+WsmohARp3V2drJ161Zqa2tpa2sjNzeXJUuWsHjx4jEPK/9RXnmvmVUvbqOxo48rz5zJbctPxpd1fKMBB/7P/6X3rbeovPsuPFPGHkQiE0FjW0tG6+vro66ujtraWurr64H4ED1nn302CxcuPK5hUkazt62XVS9u49W6FuZV5B13r2RItLub5rUPkvOxj1F0ySUnXJdIoilQJON0d3dTV1fH+++/T319PbFYjJKSEkWIWhoAABSzSURBVJYtW8Zpp51GcXFxQt4n0B/mn17fzWOv78brMtzx+flcedbMow6fMhatP3iE6ME2an70Y8wY750iMpEUKDLpRaNR9u/fz65du9i5cyeNjY0AlJSUsGTJEubPn091dXXCrsPpD0f52Zt7ePR3O+noDXPBoipu//x8KgvHPyxK/3vv0fH00xR/+VIN/igpS4Eik04sFqO1tZU9e/awa9cu6uvrCYVCAFRVVbFs2TLmz59PWVlZQi/m7A9H+dU7+/jhph3s7+rnnLml/MPyU1hYXXjsJ38EG4txYNUq3MXFlN10U4KqFUk8BYqkvb6+Pvbt20dDQwMNDQ00NjYyMDAAQFFREQsWLOCkk05i1qxZ+HyJH+4+0B/m6Tf38sQfPqC1O8SimiK+d8kizpxTmpDX7/zlL+mv3UrVgw/gLjyxcBJJJgWKpI1YLEZ7eztNTU00NzcPT11d8fFJjTGUl5dz2mmnUVNTQ01NDcXFxUkbUmZnSw+/eHsv//uPDXSHIpwzt5R1XzqdJSdNSdh7Rjo6aP3eQ/gWL6bgggsS8poiyaJAkZQSjUbp7u6mvb2dtra24XlbWxsdHR3EYvGRjo0xlJaWMn36dCoqKqiqqqK6uprs7Oyk1heKRPnttmaefnMPb33QjtdtWL5wKtctnX3Ch7aOpuWhh4gGg1R89zsaa01SngJFJkQkEqG3t3d46unpobu7m0AgQFdXF4FAgEAgQE9Pz2HP83g8lJSUUF5ezimnnEJpaSmVlZWUlZXh8UzMj28kGuMPu9p4sXY/v93WRHd/hJqSXG5bfjKXnFFDWX5yQqz3nXfpeu6XlHz1anLmzUvKe4gkkgJFPpK1lkgkQiQSIRwOEw6HCYVCx5z6+/sJBoPDATJ0UvxIWVlZFBYWUlBQQEVFBQUFBRQUFFBcXMyUKVPIz88/oSFOxqu7P8wfdh5kU10Lr77fQltwgPxsD/9hYSUXLKri7DmluFzJ6zHYSISm1avxVFZSdsMNSXsfkURSoIzBwMAAbW1tAMP3QRkaYPloj5O9LRaLHTZFo9EPrTvWtqH14XB4ODCGQmPk42h07PddcblcZGdnk52dTU5ODj6fj5KSEnw+32GT3+/H5/NRUFBATorcYXAgEuMv+7r4Y307r29v5Y/17YSjlvwcD+fOK+MLi6o4d17ZMYeUT5SOn/+cUF0d1evW4fKPbcBIEacpUMbgwIEDrF+/3ukyxszlcuF2u3G5XB+ajlzv9XrxeDzk5OTg8XiGl4emkcterxev1zscGkdOHo8nLY7zW2tpCvSzbV+ArY2dvF3fzp8bOukPx8/PzKvI4+qzZ/GZk8v5+IzicV+IOF7h5hZa1/0A/znnkH/+eRP63iInQoEyBmVlZXzpS18a/rA0xnzocaK2jWXf0cJhaJJDunrD7D7YwwcHg2xv7mHb/i7e2x+gLRj/WrHLwKlVBXz5k9P51KwSFs8soTQvuSf2j6XlwQex4TCVd96RFgEtMkSBMgY+n4/58+c7XYYcwVpLdyhCU1c/+zv7aOrq50BXP/s6+6g/GOSDg8Hh4ADwug1zy/P5zCnlLKgqYGF1IfOnFuDPTp1fg+DmzQReeonSlSvJmjHD6XJEjkvq/CZJxrLWMhCNEQxFCYYi9IQiBEMROnvDtPcO0BEcoKM3TEdwYHi5vXeA5q5+ggOHn+MxBsrysplZ6uf8BRXMKvUzqzSPWaV+ppf4yPKkbg8uNjBA0+p78E6fzpRrvuZ0OSLHTYGSYuIn4AcfDy0D1kL8EYe2H7HuyP2HXsQSf82otcRilkjMEo1ZYjb+eOS6aMwett/wNmuJRu2HXmMgEiMUiRKKxBiIxgiFR86jg9tjh82DA/HACIaiw+ERiX30XaSz3C6K/V6KfVmU+LOYX1nAufPKqCrMpbIwh6mFOVQW5lBRkDPh5zwSpf2J9Qx88AE1jz+GK8nX04gkgwJlDN7d28HVG/4IcOjD3Q59lDP4of3hD/OhD3I79MQj1o18DfvRn6dpJcvtIsvjIttz+DzL4yLL7SIv20NFfg7+bA952W782Z74lOUeXBdfLvIdChBflntSn08YaNzHwZ/8hPzzzydv6VKnyxEZFwXKGJT4s/i706oY+jwzMOKE+dA6gzFD20ZsH/zP0PZDzz/8OUMvNnIbw48Pf01Geb458n1G1GAMuF0mPhmDy2XwDC0fsW54mzm0fXibMXjch7bFA8N9KDjcrqRenzFZNa9ZAy4XFf/j206XIjJuCpQxmDHFzz1/ryHDJTm6N22iZ9Mmym/9Ft6pU50uR2Tc0vNgs8gkEevro/ne+8iacxIll1/udDkiJ0Q9FBEHHfzRjwnv38+Mnz6FScDthkWcpB6KiENCO3bQtn49hRdeiO8Tn3C6HJETpkARcYC1lqZVq3H5/ZTf+i2nyxFJCAWKiAO6nn+B3i1bKL/lm3hKSpwuRyQhFCgiEyza2UnL2rXknn46RRdf7HQ5IgmjQBGZYC0PfZ9oIEDlqrsxGsxTJhH9NItMoN5336XzX/6FkssuI+fkk50uRyShFCgiE8RGIjTdvQpPZSWlK1c6XY5Iwuk6FJEJ0v7TnxH661+p/sE63Hm6C6NMPuqhiEyA8L59tD7yCP5zl5J/nu7CKJOTAkUkyay1HLjrbgCmfve7k3rUZMlsChSRJAv85jcE/+3fKL/5ZrzV1U6XI5I0ChSRJIq0tdG85n5yTz+d4v/6ZafLEUkqBYpIEjXft4ZYby9T770H43Y7XY5IUilQRJKk+3e/I/DSS0xZcR3Zc+Y4XY5I0ilQRJIg2tND06rVZM+dS+k11zhdjsiE0HUoIknQ8tBDRJqbmbbuf2KyspwuR2RCqIcikmDBN96g8xfPUHL55eQuWuR0OSITRoEikkDRQID9t99B1uzZlN38DafLEZlQOuQlkkDN960h0trKzGd+gSsnx+lyRCaUeigiCdK9cSNdL7xA6XXXkvuxjzldjsiEU6CIJECkvZ0Dd91N9qnzKV2xwulyRByhQ14iJ8haS9NddxMLBKha/4S+1SUZK617KMaYLxtjrDGm0elaJHN1Pvcc3a+8Qtk3biJn3jynyxFxTNoGijGmCHgYaHK6FslcoZ07ab5vDf4zl1By1VVOlyPiqLQNFGAtUAv81ulCJDPF+vvZ981bcPl8TH3gAd0fXjJeWv4GGGPOAr4C/Hena5HM1bJ2LaHt26l64H685eVOlyPiuLQLFGOMF3gc+Edr7U6n65HMFHjlFTp+/gtKrrySvKVLnS5HJCWkXaAA/wBkA/ePZWdjzLXGmC3GmC2tra3JrUwywkBjIwfu/A45CxZQ/s2bnS5HJGU4GijGmM8NfkvrWNPvB/efA9wBrLTW9o/lPay1j1trF1trF5eVlSXx/0YyQay/n8av3wixGNXff0hfERYZwenrUN4A5o9hv97B+Q+ATcCbg9/yAsgCzOByyFrbl/gyRQavN7l7FaH332faj39E1owZTpckklIcDRRrbS9QdxxPORWYAXQcZVsHsA7QiHySFJ3PPkvX889TesMN5C9b5nQ5IinH6R7K8boUOHLEvW8DZwCXALrAUZKir7aWpvvW4F96DqUr9eVCkaNJq0Cx1r555DpjzJXED3X9fsILkowQPnCAhpUr8VZUUL12ra43ERlFWgWKyESL9gRpWHE9tq+fmieewF1UdOwniWSotA8Ua+2VTtcgk5ONRtl/yy2Edu6k5rHHyJ471+mSRFKa+u4io2h+8EF6XnuNyu/cSd7ZZzldjkjKU6CIHEX7k0/S8dRPKbniCoovvdTpckTSggJF5Aidzz9P8/0PkH/++ZTfdqvT5YikDQWKyAjdmzZx4I478Z+5hKrv/SPG7Xa6JJG0oUARGRR8+232feNmchYsYNojj+DSsCoix0WBIkI8TBpWXI+3poaax36Cy+93uiSRtKNAkYwXfPMtGq5bgbdqKjM2rMdTXOx0SSJpSYEiGS24eTMNK1aQNa2aGRs24NGI1CLjpkCRjNW9cSMNK64na/p0pm/YgKe01OmSRNKaAkUyUsczz9J4401kn3Iy05/cgGfKFKdLEkl7aT/0isjxsNZy8IePcvDRR/Gfu5RpDz+My+dzuiyRSUGBIhkjFgrR9N276HrhBQovvJCpq1dhvF6nyxKZNBQokhHCzc00rvw6/X/5C6U3fp3S66/HGON0WSKTigJFJr3ed96l8aYbscFepj36Q/I/+1mnSxKZlHRSXiYtG4tx8PF/Ys9ll+HK9THz2WcUJiJJpB6KTErhlhYOfPvbBN/YTP5/XM7UVatwFxQ4XZbIpKZAkUnFWkv3yy/TdM+9xHp7qbxnNUUXX6zzJSITQIEik0a4pYWm1avp2fgqOQsXUvXA/WTPmeN0WSIZQ4Eiac9GInQ+9xwt338YGwpRfuu3KLniCoxHP94iE0m/cZLWgm++RfOaNYS2b8f3yU9SuepusmfNcroskYykQJG0FNq5k9Z16+h+ZSPeqiqq160j//zzdK5ExEEKFEkrod27Ofjojwi89BKu3FzKbrqRkquuwpWT43RpIhlPgSJpoW/rVto3PEng5ZcxOTlM+drXKLn6Kt27RCSFKFAkZdlIhO6NG2l/8in63n0Xl99PyZVXMuWrV2t0YJEUpECRlBPavZuuX/+arudfINLairemhorbb6fwogtx5+U5XZ6IjEKBIikh3NRE98ZXCbz4In21teB2k7d0KUUXf5G8T38a43Y7XaKIHIMCRRxhrWXgg3p6Nr1K4JVX6K/dCkD23LmU33YbhRd8QXdQFEkzChSZMJH2doKbNxN84w2CmzcT2X8AgJyFCym7+Wbyz/sc2bNnO1yliIyXAkWSwkajhHbupO/PtfTVxqeBXbsAcBUU4P/Up/Bfey1555yDt7ra4WpFJBEUKHLCop2dhHbsoH/7dkI7dhDavoP+ujpsby8A7uJichctovALX8B/5hJyFizQORGRSUiBIsdkrSXa0UG4oYGBvQ0MNOwl3NAYn+/ZS6S1dXhfV0EB2fPmUnTRReQuWkTu6YvwTpumK9hFMoACJUNZa4n19BDt6iLa2UW0q5NoWxuRlhbCLS1EWlqJtLYSaWkh0tKCDYUOe76nvBxvTQ3+s84ie+4csufOJXvePDwVFQoPkQylQElx1lpsOIwdGDg0DS2HQsT6+oj19hIL9g4+DhLr7cX29cXX9Q5OPT1EOzvjAdLVRTQQgGj0qO/p8vnwlJfjKS8nd9EiPGVleCsr8NbUkFVTg3faNFy5uRP8LyEiqU6BMgYDjY10PvMMNhqDWPTo82gUGxttHoVobPT5yMAY8TgWDkM4PL6ijcGVm4vx+3Dl+nDl+fEUFeGpmoq7sBB3YdHgvBB30eB8yhQ8ZeW48/yJ/QcUkYygQBmDSEsr7U/9FNxujMs1trnbBS43uF2Yo829XlxDy94sTJYXV1ZWfH1W1uC6LIzXG59nHbkcn7t8vsHJj8uXO7xscnJ06ElEJpQCZQx8H/8bTtla63QZIiIpzeV0ASIiMjkoUEREJCEUKCIikhAKFBERSQgFioiIJIQCRUREEkKBIiIiCaFAERGRhDDWWqdrmDDGmFZgj9N1pIBS4KDTRciHqF1Sj9okboa1tuxYO2VUoEicMWaLtXax03XI4dQuqUdtcnx0yEtERBJCgSIiIgmhQMlMjztdgByV2iX1qE2Og86hiIhIQqiHIiIiCaFAERGRhFCgyDEZY243xvzVGBMzxvy90/VkIrVBajLGvGqMqTXG/NkY86/GmNOdrslJChQZi1eBzwOvO11IBlMbpKaLrLWLrLWnA98HNjhcj6MUKGnGGDPNGPOIMWazMabXGGONMTNH2bfGGPOcMabLGBMwxvzKGDP9eN/TWvuWtXbXidaeSRLdTmqDxEhCu3SNWCxIYulpQYGSfuYA/wXoAP51tJ2MMT5gE3AKcAVwGTAX+J0xxj8BdWY6tVNqSni7GGOeNsY0AvcAX0lS3WnB43QBctxet9ZWABhjvgacP8p+1wCzgZOttTsH998K7ACuI949xxjzDjBar+VvrLUNCaw9kyS0nSRhEt4u1tr/NuL1HgT+U9KqT3HqoaQZa21sjLteALw59Msw+NwPgD8A/3nEuo9ba0tHmRQm45TodpLESHK7/DNwnjFmyolVmb4UKJPXAuD/H2X9NuDUCa5FRqd2Sk3HbBdjTLExZuqIbV8EWoD25JeXmhQok1cJ8ePER2oHio/nhYwxdw4eI14C/C9jTKMxpjIBNcoY20ltMOHG0i7FwG+MMX8xxtQSPxT2dzaDhx/ROZTJ7Wg/2Oa4X8Tae4F7T7wcGcUx20lt4IiPbBdr7W7gExNXTupTD2Xy6iD+V9aRijn6X17iDLVTalK7jIMCZfLaRvw48JFOBd6b4FpkdGqn1KR2GQcFyuT1G+BvjTGzh1YMXsB11uA2SQ1qp9SkdhkHDV+fhowxFw8+/CywArgBaAVarbWvDe7jB2qBPuBO4seD7wHygdOstT0TXXemUTulJrVL8ihQ0pAxZrRGe81a++kR+00HHgbOI34y8VXgG9ba+mTXKGqnVKV2SR4FioiIJITOoYiISEIoUEREJCEUKCIikhAKFBERSQgFioiIJIQCRUREEkKBIiIiCaFAEXGIMebTxpiYMabeGPNFp+sROVEKFBHnvAN8DtgP/LPuIS/pToEi4hBrbcBau4n4eFKFxO/4J5K2NPSKSAowxmwDWqy1y5yuRWS8FCgiDjPGTAN2E7+D6pzBOwGKpB0d8hJx3s1AGIgAVzhci8i4qYci4iBjTBGwF3gCqAHOAGZZ/WJKGlIPRcRZ1wM5wEPEQ2UG8BlHKxIZJ/VQRBxijMkG6oH/Z6292hjjBvYAv7fWfsXR4kTGQT0UEedcDpQDawGstVHgKeAiY0yBk4WJjIcCRcQBxhgXcAvwa2tt3YhNTwC5wKWOFCZyAnTIS8QBxpgLgV8Bn7DWbjli22tAlrV2iSPFiYyTAkVERBJCh7xERCQhFCgiIpIQChQREUkIBYqIiCSEAkVERBJCgSIiIgmhQBERkYRQoIiISEIoUEREJCH+HSLn1KVHFvelAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(λ_valsL2, WsL2);\n",
    "plt.xlabel(\"λ\");\n",
    "plt.ylabel(\"$w_j$\");\n",
    "plt.title(\"L2 regularization\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEpCAYAAAC0kdQLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl4nGW9//H3PftkJvvWNmmb7ntL91Joy76VrSiyiYBaUET0qD85oh4VF0A4CEdRURFQgSL7JjstBUpL931v0jVN0iyTZPaZ5/79MZM0TSftpE0ySfp9Xddz3TPP+p00nU/uZ1Vaa4QQQoiTZUp1AUIIIXoHCRQhhBAdQgJFCCFEh5BAEUII0SEkUIQQQnQICRQhhBAdQgJFiC6klLpZKaWVUjf35G20o5Yn47WUpLoW0fkkUESHUUqVxL88Xkli3gKl1I+UUi8qpcriy9V1RZ2i47T4N38y1bWI1LOkugBxyhoN/AYwgO1AILXl9CovA0uB8lQXAvwIuA/Yn+pCROeTQBGpshmYA6zWWjcopcqArNSW1DtorT2AJ9V1AGity+kewSa6gOzyEimhta7QWi/WWjeczHpaHi9QSs1TSi1VSnmVUotazGNSSs1XSi1TSjXGhyVKqavaWOckpdR78flq47vlBiqlFimldKt52zxG0J7jB0qpryqlXlNK7VZKBZVSh5RSryqlprT3Myc6htJU+zGGn7eY9xyl1BNKqW3x9TbEf17XtK4DKI2/vanV+kqO9TNQSrmVUr+KbyOolKqK/5zHJfi8TbVblVK/aPEz2qiUuu54P1vRdaSHInqLa4BzgVeBRUAEQCmlgGeBLwGbgKfi888FXlRKfVdr/UjTSpRSE4HFgB34N7AHmAV8AtR0Yv2PAmuAd4FqYCBwJXChUuosrfXSBMsk/MxteDI+T2tXE9v96Gsx7ofAYGAZsV1VOcDlwAKlVF+t9cPx+dYAjwDfAdYCLY+dtXk8TCnlAD4Epsa38SJQTOzf6CKl1IVa608SLLoAmAS8Sey76zrgGaVUndb6rba2J7qQ1loGGTpkAEoADbxyAsuWAXUnsNzN8W1GgFkJpt8Wn/4oYG4x3kXsyywI9Gsx/tP4/HNbreev8fG61fgn4+NLEmz7qGkt6r251byDEiw/CmgA3m/nZ064jQTzXQZEgeWA8zi1uIiFhgdIS/Bv/mQb20j0M/hZfNzjgGoxfg6Hj6mZWoxfFJ9/CeBuNb8G3kn1774MsUF2eYne4mWt9ccJxn8LqAX+S2sdbRqptfYCvwRswFUQO2MJmAl8prV+s9V6fs6xewAnRWtdmmDcZmAhMEspZUuwWFuf+biUUuOBZ4gd37hCa+0/Ti1eYr27DGDaiWyzhZuInYRxt44nQ3wbHwGvAUOBMxIsd7fWurHV/GXAUbsFRWrILi/RW6xoPUIplQaMJbbb6u7Y3q8j5MfbkfF2Qrxd0npGrfV+pdQeYruCOpxSaihwN3A20I9Y0LWUy9EHt4/6zEluqxB4ndgx1Mu11gdaTc8gttvrCmKfN63VKvqeyHZbrHsQsZMxKhLMsii+3dOA1mG5OsH8+4EBJ1qP6FgSKKK3qEwwLhtQxI5H/OwYy7ribXq8rTrGNjo8UJRSw4DP49t/n9hpv43Edv9cSSzo7G3U095tOYgd6+gPXK21XtVqug34iNgX+kpiu6xqiO0aO43Yl32iWpKVEW8ThQnAwVbzNdOxs9daiyAnF3UbEiiit0j0pLj6ePup1vrMJNbRdMZZfhvTCxKMM+KtOcG0o74U2/BdYqdMX6+1frblBKXUdA73nFo7kafj/R2YAfxEa/1igulNvYO/aK1va1XLXfHpJ6Pp36SwjemFreYTPYgku+i1dOyU5C3AWKWUO4lF1sbbma0nKKWKSLxrpelspqJW85toOwhaGxJvX2+1Diexs5o6hFLqf4idGfW01vrX7aklLtFxjabjUokC9Sha63pipxqPUkolCu458XZNMusT3YsEiujtfg9kAn+M7+45glJqjFKqAEBrXUbsCvPTlVJzW836cxL36FfG26+0Gn8nye8e2xNvm7+w46c7/4bEvaJ2U0pdTewzfAZ8rT21xJe/ithZYa3VxtuiBNPa8g/AAdzTahtnEtvFt5PY2Xaih5FdXqIzTDrGvZ3e1Vo/A7GL3lqMzwNsLcdprW/ugFr+RKzHcSNwllLqQ2L76fsC44nt3jmdw8cj7iB2HcrLSqmW16GUAOuA1hfevQzsBr6mlCoG1gMT4+v9iMN/cR/Ln4FbgJeUUs8ROzV3FrGD14uAs9r3kRN6itjxpLXAjxKcoLBIa72IWM9kD3CXUmoMsR7eGOAiYp91XsuFtNaNSqnlwByl1N+IhYEG/tTGMQ+A+4FLgW/EzzZbTCyQriF29tdXtdZGG8uK7izV5y3L0HsGDl+TcKzh4RbzH3PeJLd5M8ldc3EDsVNwa4lde7IHeAf4JuBqNe8UYgfHvfH5X4p/tvWAJ8G6hwJvEDuQXk/s1NdhtO86lHOJnV3WQOzCxhfbu45jTU/i3+XnLeYdQiw8quL1fAxceIzaR8Z/lp4W6yuJTzuq/vj4dGI9sB1AKP6ZXwImJPg8i9r6fTjWNBm6flDxfxQhxDHEj8FUAhu01id7HYYQvZIcQxGihfj9onJbjTMR203jJHabEyFEAtJDEaIFpVQesI/YPbW2EbtG5UxiF0huAabqFldrCyEOk0ARooX4mWC/I3ZMoy+xi/j2Ejsu8kutdWfeIFKIHk0CRQghRIeQYyhCCCE6xCl1HUpeXp4uKSlJdRlCCNGjrFy58pDWuq1bEjU7pQKlpKSEFStO6AatQghxylJK7U5mPtnlJYQQokNIoAghhOgQEihCCCE6hASKEEKIDiGBIoQQokNIoAghhOgQp9RpwycqEmnE69uJSVlQyoJSVkymxK9jrZkEz5sQQoheTQIlCY2NW1i56pp2LdMULiZTU2uLDw5MJjtmkwOTOfa69XuzyR6bz3x4XrPFhcXswmx2Yba4sZhdWCxuzGYXJpOtkz65EEIkTwIlCS7XUCaM/xtahzF0BG1EkngdRusIho63RhjDCGIYAQwjSDQaIBJpwDAOYRgBotFAfHpsHq2jxy8sTikbFkssbJpCxmrJxGLNxGrNxmqJt/H3FmsmVkvsvdnskt6UEKJDSKAkwWrNIi/v7C7dpmFE4uETIBoNEo16iUYbiUS8RKNeItFGopGWbdP0RqJRL8FgBY3erYTDdUSj3ja3YzLZsNnysdkKsNvysNkLsNvysdnysdsLsNnysDv6YbPmSvAIIY5JAiUJmxv9/GDrXtLMJtLMJlxmM2mmptex1mlu8T4+Lc1sbp6eFh9vN6mkvphNJgsmkxtwn3T9hhEiHPYQjtQRDtcRCcfacKSOUKiaUKiKULAKn7+MOs8KwuHaBPXYcTiKcDiKcDqKcTiKcThjr53OgdhsOSddpxCiZ5NASVK62Yw3alATDuGLGnijBr74YLRjPWZFi8CJh9MRIXTkkG42k201k221kG0xk2W1kG01k2WxYDUl12MwmWzY7fnY7ce9txsAhhEkFKomGKoiFKwkECwn4N+HP7CfQGAfDQ0bjgodiyULl2swaWlDcKUNJs0Va53OAShlbsdPSAjRU51Sz0OZMmWK7uibQ2qtCRo6FjDG4ZDxRqPNr1uOP2q6YRwVUE3v/caxoyrdbGoOmByLhXy7hUKblUKblYL46wKblUKbBZelY7/UIxEvgXjA+HxleH078flK8fl2Egodap7PZHLgdo8kPX00bvco0tPH4HaNwGx2dGg9QojOo5RaqbWecrz5pIdykpRSOMwKh9lE7vFnbxdDaxqjBrXhCLXhKHWRWNv0vjYSoS4cpSYcoSYcZbsvQGUoQjjBHwkus4k+Niv9HTb6O22xtsWQb7NgascxEovFhds9HLd7+FHTwmEPPt8uvN6dNHq30tCwkYqK19m//5n4HCZcrqFkZk4kK3MymZmTcToHyjEaIXo46aH0MobW1IajVIbCVIYiVITCVATDVIbCHAiG2RcIsS8QpjocOWI5u0kxwGFjaJqDoWl2hqY5GJZmZ0ianUzryf/dobWO7S5r3ERDwyYa6tfhqV9NJNIAgM2WR2bmJLIyp5KTcyYu1zAJGCG6iWR7KBIopyhvNMq+QJi9gVBs8Ico8wfZ4QtS6g8e0cspsFkYnuZgTLqTcW4nY9xOhqU5sCR5DKctWht4vdvxeFZR51mJx7MSv38PAHZbITk5Z5CTM4ucnJnYbHkntS0hxImTQElAAiU5EUOzJxBihy/Adl+QHb4AWxoDbPb6CRix3xeHSTHS5WRcupPJGWlMy3QzyGk76V5FIHCAmppPqK75hJqaT4lE6gDIzJxEQcElFORfiMPR76Q/oxAieRIoCUignJyIodnhD7Chwc/6Rn+89VEfiZ08kGu1MC3TxZRMFzMyXUxITzupXozWURoaNlJd/RGVVe/Q2LgZgKzMqRQVXU9BwYWYTPYO+WxCiLZJoCQggdLxDK3Z7guy3OPlc08jyz1eSv0hIHYW2qzsdObkxIYS58l9+ft8pVRWvsWB8ufx+/dgtebQt+8XKOp3LWlpJR3waYQQiUigJCCB0jWqQmGW1nlZXNvAwpp69gXCAAx02DgvN4OL8zOZkek+4d6L1gY1tUvYv/8ZDh16H62j5OWdR8nA28jMnNSRH0UIgQRKQhIoXU9rTak/xKKaehbVNLC4toGAocmxmrkgN5NL8jOZk5OO3XRiT1IIBivYt/8Z9u37F5FIHZmZUygZeBu5uWehlDydQYiOIIGSgARK6nmjURZWN/DWIQ/vHvLQEDXIspi5oiCLL/XJYVJG2gkd2I9EvJSXP8+ePY8TCB4g3T2GIUO+T07ObDn9WIiTJIGSgARK9xIyDD6ubeTFilreqqrDb2gGO+18qU821/fNpcBubfc6DSNMRcXr7Cr9PwKBvWRlTmXIkB+QlXXc/wtCiDZIoCQggdJ9NUSivF5Vx/MHa/iszotVKS4ryOKWojymnECvxTBCHDjwPKVlvycUqiI//wKGDf0xTmdxJ30CIXovCZQEJFB6hh2+AE/uP8Rz5TU0RA3GuZ3c1j+fKwqyk74hZpNo1M/evU9QWvZHwGDggFsZOPA2zGZn5xQvRC8kgZKABErP4o1EeaGilsf3HWKbL0CR3cpt/fO5oW9uu292GQiUs2PHfVRUvoHDUcTIEb8iN3d2J1UuRO8igZKABErPZGjN+9X1/HFPJUs9XrIsZm4pyuOrxXnk29p3nKW2dhlbtv4Un28nfftezbChd2O1ZnRS5UL0DhIoCUig9HwrPF7+uKeStw55sJsU1/TJ4ZsDCtp10WQ0GqS07Pfs2fMXrNZcRo38TZc/kVOInkQCJQEJlN5jhy/An/ZU8vzBWiJaMzc/izsHFjAuPS3pddTXr2fz5rto9G6lf/HNDB36Q7mVixAJSKAkIIHS+1QEw/xtXxVPHThEfcTgC4XZ3DWoDwOS7LEYRpDtO+5n376nSE8fw9gxj5CWNqiTqxaiZ5FASUACpfeqj0R5dE8lj+2txNDw1eI8vjOwkOwkn+VSVfUemzbfhdYRRo26j8KCSzq5YiF6jmQDRe5NIXqFDIuZHw3uy5Lpo/hCn2we21vFGcs282x5Ncn80ZSffz7Tp72B2zWcDRu+zc5dD6H1sR/BLIQ4kgSK6FX6OWz8buQAPpg6gqFpDv5ry16uXL2DzY3+4y7rcPRj0qSn6dv3asrKHmXd+m8SiTR2QdVC9A4SKKJXGu128srEoTw0sj/bfQHOX7GV35aWEzaO3VsxmeyMGnkvw4f9lOrqhaxcdQ3BYGUXVS1EzyaBInotk1Jc3zeXj6eN4sqCbB4qq2Duym1s8R67t6KUon//m5kw/nH8/j2sWHk1Pl9pF1UtRM8lgSJ6vVybhT+MHsjfx5awPxjmwhXb+POeSozjHFvJzZ3FpIlPE436WLHyS9TXr+uiioXomSRQxCnjkvwsFk0bwdk56fx85wFuWl9KbThyzGUyMsYzZfK/MZvTWLX6RjyeNV1UrRA9jwSKOKXk26w8MXYQvx5WxKKaBs5fsZXV9b5jLpOWNojJk5/DZsth9Zqb8NSv7aJqhehZJFDEKUcpxdeK83l10lC0hitWbeeZA9XHXMZh78OkiU9js+awZs1NsvtLiAQkUMQpa1KGi/enjmBmlpvvbd3LL3bsJ3qM4ypNpxVbLFmsXnMTDY1burBaIbo/CRRxSsu2WvjX+MHcUpTHn/ZW8dUNpXgj0Tbndzj6MWni05jNaaxd81X8/v1dWK0Q3VuPCxSlVH+l1AtKKY9Sql4p9ZJSakCq6xI9l8WkuHd4Mb8eVsR7h+qZt2YHh0JtH6x3Oos4bcITRA0/a9beTDhc24XVCtF99ahAUUqlAR8CI4GbgBuBYcBCpZQrlbWJnu9rxfk8NW4Q27wB5q3ezv5AqM153e7hjB//FwKBfaxZO59o9PhX4gvR2/WoQAHmA4OBK7XWr2itXwUuBwYCt6W0MtErnJ+XyYIJQzgYDHP5qu3s8gXbnDc7aypjxjxMff0aNm3+YVL3DBOiN+tpgXI5sFRrvaNphNa6FPgUuCJlVYleZUaWmxcnDiVgaC5ftZ01xzituCD/QoYO+SGVlf+htOz3XVilEN1Pcvf27j7GAK8mGL8RuLqzNhoJRTn00nailT60oTEMjTZo8To+dFYBpyqtif+gY4NhgI7GXncyJ/B/aWa+f1oW85Zt4eLyQJt/fWnGEsy+m3DpQawb/g66fc+77xoaQ4NGE1IRjPhvq4pNSmp50bMNrPVy783f69Rt9LRAyQESHQGtAbITLaCUuhW4FWDAgBM7dl+5u57tyw6SaVZtzqOUQrU9WbR2xO4hffzxzUzxofN/2PkB+N2nDfxmoot3+ziOM/dEtA6iXT3hlvcSDqeikNrU6dvoaYECif83tPntorX+C/AXiD1g60Q2mN3XRd/rRmJ1WLDZzVgdZmwOC1aHGavdjNVmRpl6eZpoDZEghH2xIdgIwXoIeA4Pze/rjxznrwPfIfDV0OY/nzMbXHngKgB3PrgSDE3jbW66Mr17+tPm11Wt44b/3MD3JnyPircrGDhwINdddx3Vf3+Cyt/+liHvvYutf/82l3/5t/ewbd163h37dd6/6wKU/OXUQ53Z6VvoaYFSS6yX0lo2iXsuHcLptjF8Wp/OWbnWYEQgGo61TUOnvA+DET1yWjR8OCRCPgh7Iew//Drki70Pe5Pb1WSygCMzNtgzYm3+CHCdAWl58XDIjbVN753ZYO5pv4o9x5MbnyTdms5Az0B2B3Yze/ZsABreeQf76FHHDJPKsl3sWvk5K7KnMnfyIAkTcUw97X/xRmLHUVobDXRef+7QDvj0dxBt+oJu/cV8vPcJvuSbXuu2L6LrNMoU++I3WWNf5Na02GBrat3gLgSrM/7eFZ/Hefi1zQWOLHBkHBkeVmeX9h7Ese2p38P7u9/nq6O+yopFKxg6dChFRUWEDx7Ev3Yt+d/97jGXX/rSArA5WJs+jgcmFnVR1aKn6mmB8hrwoFJqsNZ6F4BSqgQ4A/jvTttq0AM7Pox9+ZqssS9jsxVM5iPf29JavLcc/tJu93tzfP0tvvRPeF0J3pt62sl94kQ9ufFJLCYLYwNj+cT3yeHeybvvAZB+4QVtLnto7262L1vCzqLTGTuokJI8udRLHFtPC5S/AncAryqlfkJsh/wvgb3AY5221aLJ8P3NnbZ6ITpDhbeCV3a8whWDrmDN0jWUlJQ0n5hS/+472IcNwz5oUJvLL31xAWa7gw/NI/nxpOKuKlv0YD3qT1WttRc4B9gG/BN4GigFztFay8O/hWjhiY1PoLVmJjNpbGxs7p1Eqqrwr1xF+oUXtrlsZdkutn72MYFhM4nanFw2vm9XlS16sJ7WQ0FrvQf4QqrrEKI7O+Q/xAvbXuDSQZeyYfkGiouLGRTvjdS/9TZoTcZFbQfKJ88+hcOdzvPR4ZwzsoCsNFtXlS56sB7VQxFCJOepjU8RNsLMMc/B4/Ewe/bs5jO0PG++gX3kSOxDhyZcdu+m9ZSuWUn69As5GFBcO03uvSqSI4EiRC9TG6jlua3PcfHAi9m4fCN9+/Zl2LBhAIT27iWwdh2Zl85NuKzWmo+feRJ3Ti7vGEMoynIye1h+V5YvejAJFCF6mSc3PkkgEuBc27nU1tYyZ86c5t5J/ZtvApBxySUJl92xYinl27cy7KIv8GlZPddN64+5t1+0KzqMBIoQvUiFt4KnNz/N3EFz2bJiC4WFhYwYMQKI9T48b7yBc/JkrP36HbVsJBxm8b/+Tk6/YpaZB2M2Ka6e0vZFj0K0JoEiRC/y2LrHiOooFzoupLq6+ojeSXDbNkI7dra5u2v1W69Rd7CcM278Gs+vLue8UQUUZhzvHmZCHCaBIkQvsbt+Ny9tf4kvDvsiGz7fQH5+PiNHjmyeXv/GG2CxkH7RRUct662rZelLCxg8aSpbzUXUeENcJwfjRTtJoAjRSzy6+lFsZhtzzHM4dOgQc+bMwRS/K4I2DOrf/A+umadjyT76xtyfLPgHkVCYOTd+nX98Vkb/HDkYL9pPAkWIXmDjoY28VfYWNwy/geWfLKdPnz6MHj26eXpgwwbCBw6QOffo3V3lO7ayYdH7TLrkcvYbbpaX1XLT6SWY5GC8aCcJFCF6OK01931+H7mOXE4LnobH4+G8885r7p0AeJcuA8A1a9YRyxrRKO/99VHcWdnMuOpanvi0FJfNzJemysF40X4SKEL0cG+VvsWaqjXcMe4Oli1ZxqBBgxgyZMgR8/iWLcM+bBiWnCOf/rDqrdeoKtvF2bfchidq4vV1B/ji5GIyHNau/Aiil5BAEaIH84V9PLTyIUbnjianIgefz8d55513xHNLdDiMb9Uq0qZNO2LZyrJdfPrvfzF40lSGTZvJ00v3EI5qbppZ0sWfQvQWEihC9GBPbHyCCl8Fd466k8+WfMbo0aMpKjryuSX+9RvQfv8RgdJQc4iX7/8FDnc658+/g1DU4OlluzlnZAGD891d/TFELyGBIkQH0Vqjddc9r31P/R7+vv7vXFxyMQfXHMQwDM4///yj5vN9/jkAadOmAhAK+Hn5/nsI+nxcddfPcOfk8urqAxxqDHHLGSVdVr/ofXrc3YaF6E601lTs3M6WJR+x9bNP8HnqsDocWB1ObPZ463DExh3x3onV4Wh+bXM4sKW5cLjcONzpOFxu7GlpqDYehqa15p6l92Az27ih3w28vPBlZs2aRXaCU4J9n3+OffhwLNnZRCMR3nzktxzaU8a8u35G/sBBRA3Nnz/ayZh+GZw5NK+zf2SiF5NAEaKdtNYc2rubrUs+ZuuSxdRVlGO2WCg5bTK5xQMIBwOEAwFCgQDhgJ9wIEBjbQ3h+PvY+ABaG8fekFI40lw43OnYXW4cbndz4OwzKqit2MitY+bx8VuLcLtczJg+7ahV6FAI3+rVZH3xi4SDAV7/3X2Url7BeV+/nUGnTQbgnY0H2XXIy6PXT5JnxouTIoEiRBIi4TD7Nq1n58rP2bVqOfVVFShlYsC4CUybdzXDps3E4Ur+2IPWmkg4dDhk/H6CPi8Br5dAYwNBbyOBxgYC3kYCjY3xtoH6qgp89fUEvY3MJJeDezcR7FeCY/8u/vy167A503BlZZGWmY07OwdHJErQZaNfbiYf3ftz9m3ZyPnz72D8eRc11/HHRTsYnOfiorF9OuvHJ04REijilKK1JhIKxnoQfj+heA8i1voJ+n0EW3yB+xsbCTR4KN+xnXDAj8VmZ8C4CUy/8mqGTJmOK+voXUzJUEphtdmx2uyQkdmuZX/66U95c/sb/Hnq//HBqx9T4HZx5uwb8Hs8eD21+Orq8NbVUlG6g4bKCqJFeWxZ+BYms5m5d/4/Rs6c3byuxdsPsWF/Pfd/YZzcVVicNAkU0Wm01mjDwIhGMaIRjKgRb6MYRhQjEm+jLYf49KZ54/NFI2EioRDRcJhIOEQkFIq/DxEJh2NtKDataVwsOILNYRHy+5Pb1QQokyl2LMOdjsPlYtSZcxgyeTr9x4zDak/dDRMX71vMKzte4evjv87eLZWEIxGuvu568vMT3yal7OabCdTVkfd/D2NzppGRX3DE9D8u3EGfDAfzJsoz48XJk0BJgqfyIKvffiP+TqM1oDUaDfGTemJn9xyednje2DyxeXV83iOnNa0k9l7HRx15xlDraU0raqqh6QwjrY3YeMM4PK75tYE2WsxnxMdp4q2GtpbTOr5sgvma548FgY4eDorOZjJbsNismK02LFZbi9exNi0jg8zCPs0Hxm2OtFjrdGJzxIbmg+jxcQ53Ojans9sdT6gN1PI/n/4Pw7KHcYHrAp5b/xxz5sxpM0yMUIjAmrVkfelq8gaUHDV96a5qlpXW8NNLR2OzyAmf4uRJoCTBW1fLuvffhvgXTKxRsS8cBYr4F49SoOLvmudt0armOVvMG19H85eXii3aYtrh9Rxj/SYTSimUMqFMqvk1pvg4pVAmhclsjr82Ha6paWi5DqWg+b1KuP7DyzaNM2Eym1AmM2aLJdaazSizGZO5xWuTGZOlqbXEWrMJk9mC6ajpsdZisx0OCltTYFgxmcyd9c/erWit+eXSX+IJeXj0rEd559l3yMvLY1arW6m05F+9Bh0I4JoxI+H6Hnp3G4UZdm6YLncVFh1DAiUJ/YaP4s5/vJDqMsQp7I1db/De7vf47qTvsnf1Xurq6rjllluwWNr+L+xbthRMJtKmTj1q2sfbD/F5WQ2/vHIsDuupEcqi80k/V4hubm/DXu5ddi+n5Z/GnLQ5LF26lKlTpzJw4MBjLuf9bCmOcWMxp6cfMV5rzf++u5WiLCfXyBMZRQeSQBGiGwtFQ/zgox+Agp9P/Tmvvfoaubm5Ca+Ibyna6MW/fj2uGacfNe39zZWs3efhO+cNk2MnokPJb5MQ3dgDyx9gU/UmfnXGr1j78VoaGhqYN28eNpvtmMv5ViyHSATXjOlHjI9EDR54ZwuD8lxcNbGojaWFODESKEJ0U2+Xvs2CrQu4afRNFNQXsG7dOubMmUNx8fGVKDhrAAAgAElEQVRP8fUtXYay2XBOnHjE+Gc/38O2ikbuumgkFrP89xcdS36jhOiGdtTu4GdLfsaE/Al8ZdBXeP311ykqKjrmWV0teZcuxTlpEibH4Wtm6nwh/ve9bZw+OJcLxxR2VuniFCaBIkQ3Uxuo5Y4P7yDNmsZ9M+/jpRdeQinF1Vdfjdl8/DOyIjU1BLdsOWp318Pvb6feH+Z/Lhvd7a6xEb2DBIoQ3Ug4GuZ7i75Hla+KR85+hFWLV3Hw4EGuuuoqsrKyklpH0+3qW15/sqOygX8u3c110wYwqm9Gp9QuhFyHIkQ3obXm18t+zYqKFdw36z6M/QarVq3izDPPZPjw4Umvx/vZUkxuN46xYwHw+MPc/vQq3HYL3zs/+fUI0V7SQxGim/jr+r/y4vYXmT9uPhNsE3j99dcpKSnh7LPPTnodWmu8n3xC2rRpKIuFQDjKbf9cQekhL3+6YRK5bnsnfgJxqpNAEaIbeGHbC/x+9e+5bPBl3Dj4Rp599lkyMjKSPm7SJFRaRnj/ftyzZxGOGnzr6VUsK63hwasnMFMeniU6mQSKECn2we4P+OXSX3Jm0Zn8eOqPWfDsAiKRCNdddx0ul6td6/J+8jEAzjPO5Hv/XssHWyr51ZVjueI0ueZEdD4JFCFSaMn+Jfxw8Q8ZmzuWB2Y9wGsvv0ZlZSVf/OIXKSgoOP4KWmlc/DG2wYP5xfIaXl97gB9dPJIbph/7Fi1CdBQJFCFSZMn+JXz7w28zKHMQfzjnD3zw9gds3bqViy++mGHDhrV7fUYggG/5clb3GcGzn+/ljrOHctucIZ1QuRCJyVleQqRAyzD56wV/ZcXHK1i9ejVz5sxh2rSjnw2fjPqly9DBIM9QzJ3nDuO/zmt/KAlxMqSHIkQXW7xv8RFhsmnlJj799FOmTJnCWWeddULrDEai/OdvLxIwW7nw+ov53vnD5eJF0eUkUIToQq/ueJU7P7yTIVlD+OsFf2Xzqs289957jBkzhksuueSEQsDjC/OVxz8nf/NqAmNO47bzRnVC5UIcn+zyEqILaK35+4a/8/Cqh5nedzoPn/Uwa5ev5b333mP06NFcddVVmEzt//tuT7WPm5/8nMievRR7D1F42W2dUL0QyZFAEaKTRYwIDyx/gGe2PMPFgy7m12f8mmWfLeP9999n7NixzJs3r13XmjRZvaeWrz+1goiheWKwHwD3rDM7unwhkia7vIToRJ6gh9vfv51ntjzDV0Z/hXvPvJdFHy466TBZ8PkernlsKS67hZdun0ne2mXYBg3CVlLS8R9CiCRJD0WITrKjdgd3LryTcm8598y8h8sHX86rr7zKunXrmDx5MnPnzm33bq5AOMrPXt3Icyv2Mnt4Po9ccxoZRpBty5eT85UbO+mTCJEcCRQhOsFbpW/xi89+gcPs4IkLn2BU5iieeeYZdu7cydlnn83s2bPbfQC+9JCXO59dzfr9Hu48ZyjfOW84ZpPC8+Z7EA6Tfu65nfRphEiOBIoQHcgX9nH/8vt5aftLnJZ/Gg/MeQBb0Mbjjz9OVVUVl19+OZMmTWrXOrXWLFi+l3te34TNYuJvX5nCeaMPPyCr8cOFmHNycE6Y0NEfR4h2kUARooNsrN7I3R/fTamnlPnj5nP7abezu3Q3Tz7/JABf/vKXGTKkfVeuVzcGuevF9by/uYIzh+bx4NUT6JN5+CmMOhymcfFi0s8/H3UCx2KE6EgSKEKchIgR4aO9H/Hs1mdZVr6MPGcej53/GDP6zuCzzz7jvffeIz8/n2uvvZacnJyk16u15uXV+/nVm5tpDET4ydxRfPWMQZhMR+4m8y1fjtHQQPq553T0RxOi3doVKEoppbXWnVWMED1FhbeCV3a8wvPbnqfCV0EfVx/unHgn1468FlPYxDPPPMP27dsZNWoUV155JXZ78s8hKTvk5SevbOCTHYeYOCCL+64az4g+6QnnbfjgQ5TdjmvmzI76aEKcsPb2UBYBc5RSvwRWACu01vs7vKo2KKWGA98CzgYGAw3AcuCnWuu1XVWHODX5wj7e3/M+r+18jc/LP0ejmdlvJndPv5vZxbOxmCzs3LmTl19+Gb/fzyWXXMLUqVOTPvjuDUZ4bPEuHvtoJzaziV9eMYYbpg88qlfSRGtNw8IPcc2cicnp7MiPKsQJaW+gXBRv/cBNwO+VUlbi4QIs1lov7MD6WruAWJg8BawCsoAfAsuUUmdorVd24rbFKcgT9LDkwBIW7l3Ior2L8Ef8FLuL+caEb3DZ4Mvon9EfgPr6ehYvXsyKFSvIz8/ny1/+Mn369ElqG1FD8+LKfTz47lYqG4JcOr4vP710NIUZjmMuF9y8mciBctJvv/2kP6cQHeG4gdJyN5fW2h9vf9Niej4wFZgC/EoptVZr3Vm/4QuAR1vudlNKfQiUAd8BvtJJ2xWngIgRYWfdTtZWrWVt1VrWVa2jrL4MgGx7NnMHz+XyIZdzWv5pzb2O8vJyPvvsMzZs2ADA9OnTOffcc7HZbMfdntaaRVur+O07W9lcXs/EAVn86cuTmTwwO6l6699+B8xm3HK6sOgmkumh7FBKPQL8XWvd2Hqi1roK+E98uEcptaqDa2y5rUMJxnmUUtsAeSSdSJov7GNH3Q621m5la81WttVuY0vNFvyR2C1Mchw5jM8fzxVDr2BK4RTG5Y3DbIqdRRWNRtmxYwdLly6ltLQUq9XK1KlTmTFjBtnZxw8DrTULt1by8PvbWbfPQ3G2k99fN5FLx/dNeveY1pr6t9/GNX06liS2KURXSCZQXgZ+BvxCKfVX4Pda673HmP9LHVJZkpRSOcBY4Imu3K7oGeoCdZTVl1FWX8bu+t2UecrYXredPfV70MQ6um6rm+HZw5k3dB7j88czPn88xe7io77ca2trWbVqFWvWrKGhoYH09HTOO+88Jk+ejDOJYxiRqME7Gyt4bPHO5iC576pxXDWpGJulfVfMBzdvJrxnD7nzv96u5YToTMcNFK31D5RSPyV2zORbwHeVUi8B/6u1Xp5g/h0dX+Yx/R5QwMOJJiqlbgVuBRgwYEAXliW6gj/i56D3IOXecsobyyn3lnOg8QC7G3azu343nqCneV6LslCcXsywrGHMHTyX4dnDGZE9giJ30TF7BmVlZSxevJhdu3ahlGLo0KFccsklDB8+PKn7cHn8YZ5bvoenluxmf52fgblp/PYL45k3qQir+cRup1f/1ttgNpN+3nkntLwQnUG19yxgpdS5wJ3AJcBS4CGt9csntHGlzgPeS2LWj7TWZyVY/kfAb4Cvaa3/fryVTJkyRa9YsaLddYquo7XGG/ZSG6ylLlBHXTA21AZqm19X+6sp95Zz0HuQ2mDtEcublImCtAIGpg9kYMZASjJLYm1GCf3c/bCYkj8P5cCBA3zwwQfs3LmT9PR0Jk+ezMSJE8nMzEzqc6zf7+G55Xt5efV+fKEo0wfl8LUzB3HuqELMbZy5lQytNTsvvAhb//4MePxvJ7weIZKllFqptZ5yvPmSOSjfD8huNXwI1APXAi8AJ3qJ7hIgmacB+RLU9Q1iYfKTZMJEdKyIESEYDRKIBGJtNEAwEiQYPTw0TfOFfTSGG/GGvW0OjeFGfGEf3ogXQxsJt2lSJrLsWWTbs+nr7svYvLH0dfWlj6sP/dz96OvqS35aPlaT9aQ+W1VVFQsXLmTTpk04nU7OP/98pk2bhtV6/PXWeEO8vHo/z6/Yy5aDDTisJuaO68ctZ5Qwtuj4QZSMwKZNhPfsIe/W+R2yPiE6SjJ/ru0DDKCR2HUfHmJh4gFeircnRGvtA7a0dzml1I3AH4ntdvv1iW7/ZBjaIKqjsdaIHn6to0e+b/G65TIGBoYRb+PzaXRseovxRyzTaojqKFrro9bfNC5shAkbYSJGhIgRafN1W9NaLh+KhmKhEQ0SjASJ6Ei7f2ZmZSbNmobb6sZldeGyunDb3BS6Cpvfu6wu0q3pZDliwZHlyCLLHhvSbemYVOc9caGuro5Fixaxdu1arFYrs2fPZubMmTgcxz59F2DTgXr+9vEu3lhXTihqMKE4k1/PG8tlE/qR4Ti5gGutQc7uEt1UMoGyGhgOPAk8rLUu68yCjkcpNY/YAfi/aa1/0BXbXFO5hlvfu5WocTg0mg7o9gQKhcVkwWqyYjFZjnjdum167bA4jhhvM9mwW+w4zA7sZjt2ix2n2XnUuKbXDku8NTtwWp24rC4cZke3fM55TU0NS5YsYfXq1UDs1N9Zs2bhcrmOu+yKshr+uGgnH26pJM1m5rpp/blu+gBG9snolFq11tS/8w6u00+Xs7tEt5PMQfnJSqmzge8DW5VSrxDrGXze6dW1opSaDTwLrAOeVErNaDE5qLVe3RnbzXPm8aXhX8JkMmFW5ubBpEyYTS1eK3PC902vTcrUPBzxHlPzuhUKsyneKjMmU3x6gnU0jVNKHdE2jbearViUpfl0V3GkAwcO8Omnn7Jp0yZMJhOnnXYas2fPPu4xkqihWbilkr8s3sXnZTXkuGx8//zhfOX0EjLTOrY30pp/zZrY7q5vfKNTtyPEiUjqCGX86veFSqlRxILlI6XUSuBB4NUuvL/XOYAdmAh82mrabqCkMzZanF7MD6Z2SWdIdLJIJMKWLVtYsWIFZWVl2O12zjjjDKZPn056euL7ZTWp8YZ4bvlenl62m321fvpmOvifS0dz7bT+pNm65j6rntdeQzkcpF9wQZdsT4j2aNf/Aq31ZqXUfOC3wN3Av4l9kQ/rhNoSbf/nwM+7Yluid6mpqWHlypWsWbMGr9dLVlYW559/PpMnTz7uMZLd1V7+8OEOXl17gFDEYMbgHH508SguGFN4wqf9nggdCtHwn7dIP/dczO7j744Toqslc5bXSsANpAMZQNMVXE07wwd3TmlCnByfz8emTZtYv349u3fvRinF8OHDmTJlCkOGDDnu43f31/n5w4fb+feKfVjNimum9OfG0wcyvPDYPZnO0rh4MVGPh8wrLk/J9oU4nmR6KKuA2uMMQnQLgUCA7du3s379enbs2IFhGOTm5nLWWWclfQ1JRX2APy3ayTPL9gBw44yB3H7WEAqOc7PGzuZ59TXMeXlyq3rRbSVzUF5OdhfdWm1tLVu3bmXbtm2UlZVhGAbp6elMnz6dcePG0bdvcvfIqmw4HCQRQ/OlKcXccc4wirJSf2v4aF0djYsWkX39dSiLPBdPdE/ymyl6nGAwyO7duyktLWXnzp1UVlYCkJuby4wZMxgxYgT9+/c/7i6tJuUeP49/XMq/lu0mHNVcNbGIb58zjAG5aZ35Mdql/u230eEwGZfJ7i7RfUmgiG4vFAqxd+9eysrKKC0tZf/+/WitMZvN9O/fnwsuuIARI0aQm5vbrvWu21fH45+U8ua6cgytmTexmG+fM5SSvO53wLvu5ZexDRmCY8zoVJciRJskUES3YhgG1dXV7Nu3r3morKxEa41SiqKiIs4880wGDRpE//79k7odSkvBSJR3N1bwz6W7+by0Brfdwk0zS7h5Zgn9c7pPj6SlwNZtBNauo+Cuu7rlhaFCNJFAESkTCoWoqqqioqKCgwcPNrfBYBAAu91OUVERs2bNon///gwYMKBdz2ZvaevBhviNGvdR6wtTlOXkJ3NH8aWp/Tv81igdre6FF1BWK5lXXpHqUoQ4JgkU0am01ni9Xqqrq5uHmpoaKisrqa6ubp7ParVSWFjI2LFjKS4upri4mNzc3KSPgyRyoM7Pf9aX8/raA6zd58FqVlwwug/XTO3PGUPzTuqOv13FCAbxvPYa6eefJ7daEd2eBIo4KVprAoEAHo/nqKGmpobq6urmHgeAyWQiJyeH/Px8xo4dS58+fSgsLCQrK+ukwqPJQU+AtzaU88a6clbujp3RPqZfBj+ZO4qrJhWT4zr+o3m7k4Z338XweMi6+upUlyLEcUmgiIQMw8Dn8+H1emlsbDyqbWxspL6+Ho/HQygUOmJZk8lERkYGOTk5jB8/ntzc3OYhMzMzqYdSJStqaNbsrWXhlioWbq1k44F6AEb2SecHFwxn7vh+DOqGB9mTVffv57H270/a9OmpLkWI45JA6aW01kQiEUKhUPPg9/sJBAJHta3H+f1+fD4fiW7RZjKZcLvduN1ucnNzGTx4MJmZmUcMLperQ3obbX2usmofS3dVs2RnNR9vr6LOF8ZsUkwemM1dF43k/NGFDC1wd8r2u1KwtBTf8uXk/9d/oTrp5ylER5JA6SRaawzDIBqNNg+RSOSI922NO968LYMiHA4fERoth2Tu2Wm323E6nTgcDpxOJ/n5+TidTtxuNy6X66jW4ejaW9D7Q1E2lXtYt8/D6j11LCutpqI+tgstP93OuSMLOXtkPrOG5nf6nX67Wt1z/waLhcx5V6a6FCGSIoGShMrKSj744IOkA6FpXGewWCxYLBZsNhs2mw2r1YrNZsPtdjePa2toCo2m1m63d1pP4kR4fGG2VTawpbyedfs8rN/vYVtFA0Y8FwvS7UwfnMuMwTnMGJzL4DxXrz2N1vB6qXvxRTIuuABrQUGqyxEiKRIoSYhGo3g8HsxmMxaLBavVisPhwGw2Nw8Wi+WI950xvjt9+Z8orTWHGkPsrvayvbKRbRUNbK+ItZUNhw/e57hsjC/O5ILRhYwrzmJ8cSaFKb6XVlfyvPYaRkMD2Td+OdWlCJE01XWPMkm9KVOm6BUrVqS6jF6vIRCmoj7A3lo/e2t87K72safGx5546w8f7r05rWaGFboZVpDO8EI3wwvTGVbopijL2Wt7H8ejtWbX3EsxpaVR8vy/T9mfg+g+lFIrtdZTjjef9FBEUgxD0xCIUO0NUusLcagxREV9gIOeAAdbtBWeAN7Qkbv7HFYTA3LSGJDj4oyheQzMTWNAbhpD82PBYeoB14N0Je+SJYR27aLf/fdJmIgeRQLlFKK1xh+O0hCI0BAIUx+INL9u2db6QtR4Ww5han0hosbRvVmLSVGQbqcw08GIwnRmD8unT6aDPhkOirKdDMxJIz/dLl+M7VD7r6cx5+aSfvHFqS5FiHaRQEkhw9CEDYNwVBOKGISjxuE2ahCOaEItxgUjBv5wlEAoij8cH0Kt2gTTvaFYcDQGIkQShEJLSkGW00qOy0auy86gPBeTB9rJcVnJcR1uc102CjLs5Lns0sPoQKHdu2lctIi8b34Tk61nXYQphARKErZXNHDvW1swtMbQsSAwtCZqaLSGqI69j42PXWxn6MPTItF4aLQIh3B8XEdwWE04rWbSbJbYa5sZp9WM226hIN1Oms1CuqNpsB7Z2luOs+CyWSQgUqj670+grFayr7s21aUI0W4SKEkIRQ0qGwKYlUIphUmB2RR7bTYprCaFScUGsyk2vem9yQQWkwmbxYTVbMJmVs2vrebYeJvZhNWssMZft55uNavm8Wk2M454eDitZuwWkwRALxGpqsLz8stkzpuHJT8/1eUI0W4SKEkY0y+TN749K9VliF6u5h//REci5H71llSXIsQJ6fkXNgjRC0QbGqh99lnSL7gA28CBqS5HiBMigSJEN1D33HMYjY3kfv3rqS5FiBMmgSJEihmBANVPPUXa6TNwjh2T6nKEOGESKEKkWO2CBUSrDpH3zW+muhQhTooEihApZPh8VP/1b6TNmIFr2rRUlyPESZFAESKFap9dQLS6mvxv35HqUoQ4aRIoQqSI4fNR/fjjuGbOJG3y5FSXI8RJk0ARIkVqnn6aaE0NedI7Eb2EBIoQKRCtq6P6r3/DNWc2aRMnprocITqEBIoQKXDoT3/GaGyk4PvfT3UpQnQYCRQhulho3z5qnnmGzHlX4hg+PNXlCNFhJFCE6GJVD/0OZTaTf+edqS5FiA4lgSJEF/KvW0f9f/5Dzi03Yy0sTHU5QnQoCRQhuog2DA7+8leY8/PI/drXUl2OEB1OAkWILlL3wgsE1q+n8Ic/xOx2p7ocITqcBIoQXSBSW0vVQ78jbcoUMi69NNXlCNEpJFCE6AJVDz9CtKGBwp/+FKXkCZuid5JAEaKT+devp+7f/ybnyzfgGCGnCYveSwJFiE5kBAIcuOu/sRQUkHeH3GJF9G7yTHkhOlHlQw8R2rWL/o//DXN6eqrLEaJTSQ9FiE7i/ewzav/xT7JvuAH3GWekuhwhOp0EihCdIFpfz4Ef3Y1t0CAKfiD36xKnBtnlJUQnOPjLXxGpqqJkwbOYnM5UlyNEl5AeihAdrP6tt6h//XXyvvlNnOPGpbocIbqMBIoQHShcUUn5z3+BY9w48m67NdXlCNGlenSgKKWuU0pppdS+VNcihNaa8h//GB0M0u/++1FWa6pLEqJL9dhAUUplAb8DDqa6FiEA6hYswPvJJxT8vx9gHzwo1eUI0eV6bKAAvwXWAu+kuhAhgqWlVNz/W1xnnEH29denuhwhUqJHBopS6gzgy8C3Ul2LEDoU4sAP70LZ7fT9za/lXl3ilNXjThtWSlmBvwAPaK13yH9ekWqVD/2OwPr1FD3yiDw0S5zSemIP5S7ADtybzMxKqVuVUiuUUiuqqqo6tzJxymlYuJCaJ58k+/rryLjwglSXI0RKpTRQlFLnxc/SOt6wKD7/UODHwB1a60Ay29Ba/0VrPUVrPSU/P78TP4041YQPHqT8v3+EfeRICu66K9XlCJFyqd7ltQQYlcR8vnj7f8CHwNL4WV4ANkDF3we11v6OL1OII+lQiP3f+z5GOEzR7x7CZLenuiQhUi6lgaK19gFb2rHIaGAgUJtgWi3wCPDdDihNiGM6+Ovf4F+1iqKH/hf7IDlFWAhIfQ+lva4FHK3G/TcwGbgakAscRaerXbCAuueeI3f+fDIuuSTV5QjRbfSoQNFaL209Til1M7FdXYu6vCBxyvEtX87BX/0a15zZ5H/3O6kuR4hupSee5SVESgR3lbLv23di69+fogcfRJnNqS5JiG6lR/VQEtFa35zqGkTvF6mqYu/8+WAy0f/Pf5KnLwqRQI8PFCE6W7TRy97bvkGkpoaB/3gK28CBqS5JiG5JAkWIYzD8fvbf+W0CW7fS/4+PyvNNhDgGCRQh2mB4vez95u34li+n772/wT1nTqpLEqJbk0ARIoFoQwN759+Kf/16+j3wAJmXzk11SUJ0exIoQrQSqapi723fILB9O0W/e4iMC+QeXUIkQwJFiBaC27ez57bbiNbW0f/RP+CePTvVJQnRY0igCBHnXbqUfd++E+WwM/Cf/8Q5dkyqSxKiR5ELG8UpT2tN7YIF7Jl/K9Y+hQx67jkJEyFOgPRQxCnNCAQ4+It78Lz8Mq7Zsyh68EHMGRmpLkuIHkkCRZyyQvv2sf/O7xDYtIm8228n745voUzSaRfiREmgiFOS5/XXOfiLe0Apiv/4R9LPOTvVJQnR40mgiFNKtL6eg7+4h/o338Q5cSL9HvgttuLiVJclRK8ggSJOGY0ff0L5z/6HSEUl+d+5k9z581EW+S8gREeR/02i14tUV1Nx733Uv/EGtkGDKHn2GZzjx6e6LCF6HQkU0Wtpw8Dz0ktUPPAghs9H3re+Re5tt2Ky2VJdmhC9kgSK6JW8yz6n4v77CG7ajHPyZPre8wvsQ4akuiwhejUJFNGrhMrKqHjwQRrf/wBL3770e/BBMi65WE4HFqILSKCIXiG0Zw+H/vwYnldfxWS3k//d75Jz802YHI5UlybEKUMCRfRoLYNEmc1k33A9efPnY8nPT3VpQpxyJFBEj6O1xr9yJTVPPUXD+x+grFayb7ie3K99HWthQarLE+KUJYEiegwjEKDhvfeoeeofBDZswJyZSe78+WR/+QasBRIkQqSaBIro1nQ0im/5cjyvvU7DO+9geL3YBg2iz89/RuYVV2ByOlNdohAiTgJFdDtGKIRv2TIaPvyQxg8+JFJZicnlIv3CC8m8/DLSpk2Ts7aE6IYkUES3EK6oxLtkCY0LF+L95BMMnw/ldOI+8wwyLr4Y99lnS29EiG5OAkWkRLSuDu/y5fg+W4p36VJCu3YBYCkoIOOyy0g/52zSZszAZLenuFIhRLIkUESn04ZBqKwM/5q1+NfGhuDWraA1yukkbcoUsr7wBdJmTMcxapTszhKih5JAER1KR6OE9uwhuHUrga1bCazfgH/dOoz6egBMbjfO8eNJ/9a3cJ0+A+e4cSi5t5YQvYIEijghOholfOAAobLdhEp3Edi2jeDWbQS3b0cHArGZzGbsQ4eScdFFOCdMwHnaBGyDBkkPRIheSgJFtMnw+wmXHyRcfoDwvv2Edu8mVFZGqKyM8N696HC4eV5zdjb2kSPIvuYa7CNG4Bg5AtuQIXIMRIhTiATKKUhHo0Tr6ogcqiZaU03kUDWRyop4eJQTLj9A5EA50bq6I5ZTNhu2gQOwDxlM+rnnYBs4EFtJCbaSEsy5uSilUvSJhBDdgQRKD2cEgxj19UTr64l66onWezAaGg6/9tQTqakhWn0oFhzV1URra8EwjlqXKT0da58+WPr1xTl+PNY+fbH264u1b1+s/fph6dtXdlcJIdokgdKJdDiMEQyi/f7DbSCIDsRaI+BHJ9UG0IHA4dbvj4VGfT06GDxmDSotDUtODpbcXKzFxTgnTMCcl4slNw9Lbg7m3FwseXlY8vMxp6d30U9GCNEbSaAkIbRvH3ULFhw/BPyBIwKESOSEtqfsdpTDgcnhQDnsmOwOlNOBye7AlJeL1e7AlJGOOSMTc0YG5swMTOmx1pyRgSkj1prT0+UMKiFEl5FASUKkqoqaf/7ryC95h7O5NeW6sDqcmBx2VOvW7sDkdBy/bVq33S67lYQQPZIEShLSJk5k5No1qS5DCCG6NflTWAghRIeQQBFCCNEhJFCEEEJ0CAkUIcT/b+/eYuUawzCO/x/HqtASFRE9UUIlIg5Jy41TL7igSoQ4iwskIkokpHdckV65IA2CxB11uhBpVOsQJVWVKFrHVtMLFeCu3SQAAAWhSURBVKWkJK2+LmYkYzqz916zvlnfmr2eX7KSvb/1fTPvfrL2erPW3jNjloQbipmZJeGGYmZmSbihmJlZEm4oZmaWhCIidw2VkbQT2NoxNA34vcD3xwG/DKG07udJuW68Of329xqvS169nivVGudVfM1Y85xXsXll8uoeS5nX7IiYMe6siGjsBqwo+P36KupIuW68Of329xqvS16DZua8hrNmrHnOq7q8useqzOu/rem3vN4s+H1VdaRcN96cfvt7jdclr0Gfy3kNZ81Y85xXsXll8uoeqzIvoGG3vMqStD4izstdx6hwXsU4r2KcVzFV5NX0K5SiVuQuYMQ4r2KcVzHOq5ih5+UrFDMzS8JXKGZmloQbipmZJeGGMiSSpkh6TdJXkjZKelvSybnrqjNJj0jaLGm/pMW566kTSadI+kDSFkmfSfIfo8fgY6mYVOcrN5TheioizoiIs2n9C98zuQuquXeAK4D3chdSQ08Dz0fEacBDwEuSlLmmOvOxVFzp81VjGoqkkyQ9KekjSXskhaQ5febOlPSypN8l7Za0UtKsIs8XEX9HxNsdQ+uAkblCqTovgIj4OCK+K1t7HaTMT9IMYAHwAkBErGrvOnfIP0ZlUh9vk+lY6idlZqnOV41pKMA84DpgF/B+v0mSpgKrgdOBW4GbgVOBdyUdWeL57wVeL7G+arnzGnUp85sF7IiIvR1Lt7bHJwsfb8UNM7PBzlfDfil+XTbgoI6v7wQCmNNj3n3AP8C8jrG5wD5gacfYBlrvi9Nrm9n1mA8DHwFTc+cwInmtARbnzqAu+dG6EtnctW4VsCT3z1nHvCbbsZQhs4HPV425QomI/ROceiWwLiK+7Vj7A/AhcFXH2DkRcVyf7af/5kl6ELgGuDwi9qT5aYYvV16TReL8tgEnSjq0Y93s9vikkPp4a4JhZFb2fNWYhlLAmcAXPcY3AfOLPJCkpcANwKKI+C1BbXWULK+GGje/iNgJfALcBiBpESDg02pKrBUfb8VNKLMU5ys3lAMdS+ueZLdfgWMm+iCSTgKWA9Np3avcKGl9mhJrJUleAJKWSdoOLASekbRd0gkJaqyzieZ3F3C7pC3AE8CN0b4/0TATyquhx1I/42aW6nx1SJkqJ7Fev6iF/kUzIrYXXTPCSucFEBGPAY+VL2fkjJtfRHwDXFBNObU3kbyaeiz1M2Zmqc5XvkI50C5aHb3bMfTu8k3nvMpxfsU4r+Iqy8wN5UCbaN1z7DYf+LLiWkaB8yrH+RXjvIqrLDM3lAO9ASzofNuB9ouFLmzvs/9zXuU4v2KcV3GVZdaot6+XdG37y0tp/ZHzHmAnsDMi1rbnHAl8DvwFLKN17/FR4CjgrIj4s+q6c3Fe5Ti/YpxXcbXLLPeLc6rc2kH22tZ0zZsFvALsBv4AXqPHC4Ym++a8nJ/zqvdWt8wadYViZmbD47+hmJlZEm4oZmaWhBuKmZkl4YZiZmZJuKGYmVkSbihmZpaEG4qZmSXhhmKWiaSLJO2X9KOka3LXY1aWG4pZPhuAy4AdwLMN/Ex0m2TcUMwyiYjdEbGa1nswTaP10atmI8tvvWJWA5I2AT9HxMW5azEblBuKWWbtj1/9ntYnqM6LiO8zl2Q2EN/yMsvvfmAvsA+4NXMtZgPzFYpZRpKmA9uA54CZwLnA3PAvpo0gX6GY5XU3MAVYTqupzAYuyVqR2YB8hWKWiaTDgR+BtyLiDkkHA1tpfTjSTVmLMxuAr1DM8rkFOB54HCAi/gFeBJZIOjpnYWaDcEMxy0DSQcADwKsR8XXHrueAI4DrsxRmVoJveZllIOlqYCVwfkSs79q3FjgsIhZmKc5sQG4oZmaWhG95mZlZEm4oZmaWhBuKmZkl4YZiZmZJuKGYmVkSbihmZpaEG4qZmSXhhmJmZkm4oZiZWRL/Ajm/PptQkkRiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(λ_valsL1, WsL1);\n",
    "plt.xlabel(\"λ\");\n",
    "plt.ylabel(\"$w_j$\");\n",
    "plt.title(\"L1 regularization\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the weights \"snap\" to zero in the L1 case whereas they only approach 0 asymptotically in the L2 case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Bonus) Another question is what happens to regularization as $n$ gets large.\n",
    "\n",
    "- As $n$ gets large, the loss term starts to dominate the regularization term for fixed $\\lambda$, because it's a sum over $n$ things.\n",
    "  - This appeals to our intuition, since we don't expect to overfit if we have lots of training data.\n",
    "  - There's also a Bayesian interpretation coming later in the course.\n",
    "- In theory, $\\lambda$ should be $\\Omega(1)$ and $O(n^{1/2})$ (hopefully those are the correct symbols!). \n",
    "  - If it grew linearly with $n$ then the loss and regularization would grow the same amount, which violates the above intuition about lots of training data.\n",
    "- In practice, if we're choosing $\\lambda$ with (cross)-validation anyway, we'll just get the value that works best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why not use Huber regularization?\n",
    "\n",
    "- Don’t confuse the L1 loss with L1-regularization!!!\n",
    "  - L1-loss is robust to outlier data points.\n",
    "    - You can use instead of removing outliers.\n",
    "    - \"sparse residuals\"\n",
    "  - L1-regularization is robust to irrelevant features.\n",
    "    - You can use instead of removing features.\n",
    "    - \"sparse coefficients/weights\"\n",
    "- And note that you can use them together.\n",
    "- Why aren’t we smoothing and using \"Huber regularization\"?\n",
    "   - With the L1 loss, we cared about its behavior far from 0.\n",
    "   - With L1 regularization, we care about its behavior near 0.\n",
    "      - It’s precisely the non-smoothness that sets weights to exactly 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple choice question\n",
    "\n",
    "Which of the following is true about L1 regularization? (There may be multiple that are true.)\n",
    "\n",
    "1. For any value of $\\lambda$, some of the $w$-values will be $0$.\n",
    "2. For $\\lambda \\rightarrow \\infty$ all of the $w$-values will be zero.\n",
    "3. All the weights will be smaller than with L2 regularization, because you're not squaring them in the loss.\n",
    "4. The regularization path is **not** a continuous function.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uniqueness of the solution: L1 and L2\n",
    "\n",
    "- With no regularization, the solution is not unique. \n",
    "  - Example: two identical columns $i$ and $j$. \n",
    "    - If $(w_i,w_j)$ is a solution then $(w_j,w_i)$ is also a solution.\n",
    "- With L1 regularization, the solution is also not unique. \n",
    "  - The same example applies.\n",
    "- With L2 regularization, the solution is unique. \n",
    "  - If we have two identical columns $i$ and $j$ then $w_i=w_j$.\n",
    "  \n",
    "**The experiments shown below bonus material.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr.seed(0)\n",
    "\n",
    "n = 20\n",
    "X = npr.randn(n,1)\n",
    "y = npr.randn(n,1) + 0.5*X**2 + X\n",
    "X = np.concatenate((X,X),axis=1) # identical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR0AAAEmCAYAAABf10QVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztnXu4HFWZr9/f3tkhV5AIooKCiJeAo/IQFUVFES8wQkBuPl4QnWNAxcsoR0XxTBx1OB4QHRhFGQdRB0URFLyPRkAHhSFRATXKwSEwMHgEEnNPyE6+88eqnfTudFVXdVd3Ve31vc/TT++uWutbq7qr3r3qq5vMDMdxnGExUnUHHMeJC5eO4zhDxaXjOM5Qcek4jjNUXDqO4wwVl47jOEPFpeNUjqT9JJmky5rcRoG+nJb05bSq+1IFLp0B0LKCm6QrU8pMrHjvGnb/nMEjaYWkFVX3o45Mq7oDEXCCpAVmtrTqjkTOfcB8YHXVHQG+CdwE3F91R6rARzqD5a7k/dxKe+FgZlvM7PdmVvmGbmark77UQYBDx6UzWG4DrgKOlPSSvJUkvVDSDyStkrRR0u2SzpKUa2Tamr+Q9FeSvpPEsrZyL5b0PUkPSdok6XeS3t+pHUmPkvQFSQ9KWi/pxqT+4qStF7WUTc1ZFMlnSDoiafOOpM21kn4u6ZSiy9wpp9PS97TX9S1lnyzpPEm/TuJukvRbSedIGmvvB7AvsG9bvNNyfD9/I+mWZHnXSPqppIUdym3/3iW9VtKtSZ/ulfRRSaPdvt+q8N2rwXMOcDxhtPPsboUlnQx8BdgAfA1YBRwNnAc8X9Lxlv+CuQOAG4FlwOeBfVraORO4EHgQuCZp5/kt/XxVS9m5wE+BpwA/AW5OYv8AuC5nX3rhvcD+SXv3AfOAY4ErJD3GzD7VoU7qMnfg+pTpzyJ85xtapr0KeCNh+X8M7AIcDnwEWAAcl5T7C/BhYCJX19rHX2f0BUkXAWcCdwOfA6YDJwHfkvQeM7ugQ7W3Ay8l/IbXEb6fDxK27fdntVcZZuavkl/AfoAB30o+/3Py+YSWMqcl097VMm1Xwkq7DpjfMn0a8MOk/KkF2jfgAx3mHwRsIeQVdmuZLuCfknontkz/WDLtvLY4r2tp50Udlu20Dm3vNK+lv5e1lX1Ch/qzgVsJuZlZBZa5Yxsp5f4MrASe3DJ9b2B6W1m1/LbPb5u3AliR0kan7+DwZNqtwJyW6Y8mCHcLsH/L9MVJ+ZXAAS3T5wEPAWvb+1uXl+9eDYfFwCag27D3OGA34BIzWz4x0czGgfclH99QoN37CSOkdk4niOxMa8krWFhrP0BYmV/dUv61hP/6/9AW53LgdwX6Uwgzu6vDtPXAFwmC7jRyTFvmrkjaFfgOsDtwkpnd0dLufWb2cFtfDLg4+XhkL222MPG7fsjM1rW08SfC8kwj/A7tXGhmd7aUXwlcC8whjExrh+9eDQEzu0/SPwFnEVauS1OKPjN5v75DjF9LWt1SJg+3mtmWDtOfQxDLsZJe2WH+RuCpAJJ2I+QnlpnZqrY+maSbgAML9Ck3iQTeCywk7GbNaivymA7V0pa5W1ujwFcJo8C3mNmStvkjwN8QRikHEaSnLn0pQupv3zKt02//qw7T7kveH9FflwaDS2d4nAu8GVgs6fKUMrsm7/8vZf6fgCcWaPPPKdPnETaYD2XUnZ28z03eHyjYRl9Img7cQNjQlgGXEXYltibTFhLyKmX15xOEPM6FZvbZDvMvAt5KyLdcTfgtHiZs2O9M6UsRdgU2mdmaDvP+1FKmnU5HwMaT91omk106Q8LMVko6n5B4fBthA2pnYoXbKyXMXi1lcjWbMn0NYeOdbWabu8RYm7zvmTL/UR2mbUveO630nTacTiwkyOUSMzu9dYak9yXzO1H4rnSSTieI44fAuzvM3wt4CyHf8lwz29gy7zlJ3X5ZAzxR0q4dxLNXS5nG4zmd4fJJwijmbDpvfBNHN17YPkPS0wn/VTOPgOTkPwhCeFa3gknO525gvqRJw3VJAg7tUO0vyfveHeYdnLOPEyO6b3eYd1jOGF2RdAQhef474BQz29qh2BMII8MftwqnS1+2UmykkfrbE5LMrWUajUtniCRJ0I8CewDv6FDkGsJ/s0WSDpiYmOQbPp58/FIJXfkMYaP4tKSdchGS9pI0v2XSVwn5lA+2FX0NnfM5vySMOE6RtH23Q9Kz6ZwM7cQ9yfukjVrSq4BjcsbIRNKTgW8QdlGOsfST9Sb68txEtK31z06psxLYo3X5uzDxu35Y0vbclaRHEfJa44RTKRqP714Nn88RhvA75WbMbLWkM4B/BZZJuoIwajgaeBrhyErf0jGz2yW9nfAf/g5J3yMc4t0deBLhfJ0PARNH0M4FTgDOknQwYaR0AOGckB8CL2fHLhVmdq+kq4ATgVsk/Ygw6lkIfJcd57Rk8W3Cxv4+SQcBvyckcF9BuIzg+F6Xv4VPEZb5+8CpLT6ZYIWZXWZm/y1pos1bJF0HPJYdy39Ch9jXEc7fuUbSjYRD3t81s9s7dcTMrpd0MWE37jdJe9OBkwm7sO81sz/2t7j1wKUzZMxsi6T/BXw5Zf5XJd1P+A96CjAD+CPhRK8LksO0ZfTjYkm/JgjwhYQNaiVBPh+h5b+qma2R9ALCaOsY4LmEof5RhJPmXs6O3M8EpxFOPDyBkIC9LWnj0eSQjpmtTXZ9zidI8MVJm0cTjhSVIZ2JEcVRyaudGwgJbAhHHe9J2n074Tf5IEGOnaTzUeCRwF8TTt4bISSEO0on4W2Eo1FnEOSzLfn8FjO7Oucy1R6VtA4PHEkvJ5yrciDhv9MDwM+BxWY2sHNFnGwk/Qx4HuEkw3XdyjtOk3I68wiHTs8EXkYYCRwE3CRp3yo7FgMpuZ9XE0YhP3HhOHlpzEinE5KeQtjXP8vMPlF1f6Yykm4n5JduJZyf8nTgJYRLNl5gZlPiyIozeJo00unEQ8l74TNQncJcRsiBvJaQ03ga4YLUQ104ThEaN9JJDh+PEk7N/9+EfMIzzGwgZ8Y6jlMuTTx6dTNwSPL3ncARLhzHaQ5NHOnMJ5zNuz/hAsq9CLcVWNGh7CJgEcCoxg6ZPVrB9W9bO53gOnx2t43sQfvJtLCZEe5RDa4LHK3oMqGRkjMMo/3Fs5GdzhUaXN2CXbXR7PjrV977oJmlXS6zncZJp5XktPwVwBVmdkZW2d12ebQ9b+/Xwfr1Q+lbK9tWV3/JzPxtD3DBlh8hdlwaPfHLr2AuZ+xSykm+fTGyW97Lskpm9uzuZQpgc2f2VX/r7N6vHR2fM9a9UGv5WcXMs2V2evmbv3LWMjNb0C1GoxPJZvYXwi7WAd3KbqfkFSwPlW1MLSwf2ZMrR8KVDROymZDPfqzlk5t/UEm/WqlMziX/I9LanUeURRhd3+0a3HSmrSt2TGXahm3dC7Uwtr5Y+U40WjrJ1b9PJZwdmp9IxXPp2MF8fWT+9tvrwQ7xzGeli6dEXDzpNEY6kr4p6UOSFircEPx0wmnq44R7oRQjYvFcOPosF08nXDy56Uc8jZEO4X6+xxFuVfldwjVDNwDPbL2tZCEiFc/3pz0pdVdrPiv57OZOd5MYLi6ewFQUT2OkY2YfN7NDzOwRZjbLzJ5iZqd3OmpViEjFk7WrtR9rXTwl4uKZTGOkM1AiFo8nl1Nw8eSmqHhcOhNELB5PLqfg4slNEfG4dFqJWDyeXE7BxVM6Lp12IhVPt+Syi6c8YhePS6cTkYrHd7UycPGUhksnjYjF4yOeFFw8peDSySJi8fiIJwUXT9+4dLoRsXh8xJOCi6cvXDp5iFg8S5LbT7t42nDx9IxLJy+Riue86Yf5rlYaLp6ecOkUIVLx+K5WBi6ewrh0ihKxeHzEk4KLpxAunV6IWDw+4knBxZMbl06vRCweTy6n4OLJRTzS6fOG2R2JVDyeXM4gYvHkJR7p0P8NszsSqXi67Wr5/XjKY6qJJyrpgIunTPxGYBm4eFKJTjrg4ikTvxFYBi6ejkQpHXDxlIkfTs/AxbMT0UoHXDxl4jcCy8DFM4mopQMunjLxp0xk4OLZTvTSARdPmXhyOQMXD+DS2Y6Lpzw8uZyBi6c50pF0oqSrJN0taaOkP0g6V9Lcstpw8ZSHJ5cziFw8jZEOcBawFfgA8ArgYuAtwI8klbYcLp7y8ORyBhGLp0nSOcbMTjazy83sBjP7FPAO4DnAi8psyMVTHp5cziBS8TRGOmb2QIfJtyTve5fenounNDy5nMEUEk9eGiOdFA5P3pcPIriLpzw8uZxBZOJprHQk7Q38PfBjM1s6qHZcPOXhyeUMIhJPI6UjaQ5wDTAOvDGj3CJJSyUtfXi89x/VxVMenlzOIBLxNE46kmYA1wL7Ay83s3vTyprZJWa2wMwWjE2f01e7Lp7y8ORyBhGIp1HSkTQGXAU8GzjazG4vUn/r7F36at/FUx6eXM5giounMdJJzsW5HHgJsNDMbuoljosnUBfxeHI5hSksnsZIB/g0cBJwPrBe0qEtr32KBHLxBOoiHk8upzBFxdMk6RyVvH8Q+EXb638UDebiCdRFPJ5cTmEKiqcx0jGz/cxMKa/FvcR08QTqIB5PLmcwxcTTGOkMChdPoA7i8eRyBlNIPNFLB1w8E9RFPJ5cTmGKiMelk+DiCdRFPJ5cTmEKiMel04KLJ1AX8XhyOYUaiqeIfFw6bbh4AnUQT7fksounPPoVTxFcOh1w8QTqIB7f1cqgoeJx6aTg4gnURTw+4kmhgeJx6WTg4gnURTw+4kmhYeKJRjo2ou6FOuDiCdRFPD7iSaFB4olGOgDjc8Z6qufiCdRFPEu0L+Di2YmGiCcq6YCLp1/qIJ7zph/mu1ppNEA80UkHXDz9UgfxdNvV8ksmyqNs8UQpHXDx9EtdxOPXaqVQY/FEKx1w8fRLXcTj12qlUFPxRC0dcPH0S13E4zmeFGoonuilAy6efqmLePxarRRqJh6XToKLpz/qIB6/EVgGNRKPS6cFF09/1EE8nlzOoCbicem04eLpj7qIx5PLKdRAPC6dDrh4+qMu4vHkcgoVi8elk4KLpz/qIh5PLqdQoXjikU4PS+ri6Y86iMeTyxmULJ68xCMdYHxW8cV18fRHHcTjyeUMKhBPVNIBF0/M4vHkcgpDFk9jpCNpH0kXSfqFpA2STNJ+vcRy8cQrHk8upzBE8TRGOsABwMnAKuBn/QZz8cQrHk8upzAk8TRJOj81s73M7GjgyjICunjiFI8nlzMYgngaIx0z2zaIuC6eOMXjyeUMBiyexkhnkLh44hWPJ5dTGKB4prR0JC2StFTS0i0PZ3+JLp54xePJ5RQGJJ4pLR0zu8TMFpjZgrHp3TcqF0+84vHkcgoDEM+Ulk4vuHjiFI8nlzMoWTzRSMdG8z/3ysUTp3g8uZxBieKJRjoAW2bnX1wXT7zi8eRyCiWJp1HSkXSipBOBQ5JJRyXTDs8bw8VTgIjF48nlFEoQT6OkQzgp8ErgjOTzZ5LPHy4SxMVTgIjF48nlFPoUT6OkY2ZKeb2oaCwXTwEiFY8nlzPoQzyNkk7ZuHgKEKl4PLmcQY/iiVo64OIpRMTi8eRyCj2IJ3rpgIunEBGLx5PLKRQUj0snwcVTgIjF48nlFAqIx6XTgounAJGKp1tyOWrx5MSl04aLpwCRisd3tfojGulYgSV18RQgYvH4iKc3opEOwPjM/NdfuXgKELF4fMRTnKikAy4ecPGUiY94ihOddMDFAy6eMrl07GCWaF/AxZOHKKUDLh5w8ZTJedMP812tnEQrHXDxgIunTLrtakV9yUQLUUsHXDzg4ikTv1arO9FLB1w84OIpE79WKxuXToKLx8VTJn44PR2XTgsuHhdPmfi1Wp2JRjp5z0h28bh4ysRvBLYz0UgHYMusfEJx8bh4ysSTy5OJSjrg4imCi6c8PLm8g+ikAy6eIrh4ysOTy4FCa7+kJ0haIuk/JV0gaUbLvP8ov3uDw8WTHxdPeXhyufhI5zPA1cBJwDxgiaS5ybzetogKcfHkx8VTHrEnl4uu9XuZ2afNbJmZnQZ8hyCe3djx/TUKF09+XDzlEXNyuegaP2mtNbNzga8DS4C5HWuUhKTHSfqGpNWS1ki6WtLjy4jt4smPi6c8Yk0uF13b75D00tYJZnY+8BXgiaX1qg1Js4CfAE8F3gC8HngScJ2kUtZYF09+XDzlEWNyueuaLql1K3s18NP2MmZ2AfC4EvvVzpuB/YHjzOxbZnYNcCywL3B6ngCWwxUunvy4eMojtuRynrX8TknvkDTHzDab2eZOhczsvtJ6tTPHAjeZ2Z0t7d0F3AgszBtk64zuZVw8+XHxlEdMyeU8a/g3gb8D/kvS/5E0yBFNGgcBv+kw/bfAgUUCuXgm4+IJ1EE8sSSXu67dZnYWsA9wNnAU8EdJV0h6Vt+t52cesKrD9JXA7mmVJC2StFTS0q0bdzwMzMUzGRdPoC7imerJ5VxrtpltNLPPmtlfEcQzE/i5pJ9JOr6vHuSn0yH5zC3ezC4xswVmtmB05uSV2MUzGRdPoC7imcrJ5TyJ5MdKOkjS8yUdA+xNOJJ0BXAo8I2eW8/PKsJop53d6TwCyoWLZzIunkBdxDNVk8vTcpS5F9gGrAPWAquBNcn71cn7oPktIa/TzoHA7/oJvHUGjG7KLrNllhjb0P3cx/GZYtrGfOdIbpk9wtj6bbnKjs8aYdqGfGW315kzxrR1WwrVgSCe0fUdjxXkwubORGs39ly/I7NnF3pWdhmM7LZr5bf1/P60J/EYW8fJ25ZjBOmIIKGJ5PIZuxxTaR+3rV5TWNJ5/o3+CtgIXAa8wMyeZmbPM7OjzOwUM1tUvKuFuRY4VNL+ExMk7QcclszrCx/xTMZHPIG6jHimWnI5TyL5EMIh6wOAP0j6mqRn99a9nvlnYAVwjaSFko4FrgH+C/hcGQ24eCbj4gnURTxTKbmcN5F8nZm9EngmYRfrBkn/Lum4tpMHB4KZrQeOAO4AvgxcDtwFHGFm63LFyLGkLp7JuHgCdRFP3ZPLeSm0BpvZcsLZwc8A7iRcd3XHAPrVqe17zOwEM9vVzOaa2XFmtqJIjPFZ3cu4eCbj4gnURTx1Ty7nIc/Rq2WS/iDpvyWtA8aB5cCphET0/pkBaoaLpzgunkAdxNPtzOUmiCfP0atfEg5LZ70axfgsmLYhu4wf1ZqMH9UK1OGo1qVjB8MWOGnbcmDno1qf3PwD/naXV1TZxUy6SsfM3jyMjgwbF4+Lp1fqJJ60w+l1Fk+U90iewHe1iuO7WoE67Go1NbkctXTAxdMLLp5AXcTTtBxP9NIBF08vuHgCdRHPEu0LNEM88Uiny5K6eIrj4gnUQTznTT+sMbta8UiH7tJw8RTHxROog3iasqsVlXTAxZNZx8XTF3URT91HPNFJB1w8mXVcPH1RF/HUecQTpXTAxZNZx8XTF3URT12Ty9FKB1w8mXVcPH1RB/HUNbkctXTAxZNZx8XTF3UQT7ddrSruxxO9dMDFk1nHxdMXdRFPnW4EFo10ut1Px8WTUcfF0xd1EU9dbgQWjXQAts7KvjDSxZNRx8XTF3URTx1yPFFJB1w82+O6eMqP2YW6iKfqG4FFJx1w8WyP6+IpP2YX6iCeqh9hHKV0wMWzPa6Lp/yYXaiDeKpMLkcrHXDxbI/r4ik/ZhfqIp4qkstRSwdcPNvjunjKj9mFuohn2Mnl6KUDLp7tcV085cfsQl3EM8zkcjzSUfb9iV08SVwXT/kxu1AH8QwzudwY6Uh6t6RvS7pfkklaXDTGthldxOLiCXFdPOXH7EIdxDOs5HJjpEN4yN+jgG/1E8TFkw8XT7ziGXRyuUnSOcjMngO8vd9ALp58uHjiFc8gk8uNkY6ZFXu4UxdcPPlw8cQrnkEllxsjnUHg4smHiydO8QwquRy1dMDFkxcXT5ziGURyuRLpSDoyOQLV7XV9n+0skrRU0tKt69IfPeviyYeLJ17xlJlcllm+52uXiaRZwONzFN1gZve01Z0GbAE+bGaL87a5yxP2scec/c7MMiObsjeq0Q1d5nd57nm3Z6fniQHkenY6kPvZ6UDuZ6cDhZ+dDvT07HSgr2enA+U/Ox2G/ux0oPJnpwO8acuvOGnb8u3PTYcdElrOPA5i5TIzW9AtzrSB9TADM9sA/H7oDc/YCptGU2dvm7EtUzxbZ23LFM/WGdnSGJ/VXTzdYkAY8eQRz/hM5RbPltkjucUzPmuksHjG54z1JJ6ts3fpSzw2d2b54pk9e+jiGdlt18rFc+nYwdw/Pod3bL0FYLt8jJDjmQ9PzRMnvpzOjK2Zs31XKx++qxXnrlZWcnkW5PpSGiMdSQsknQi8Kpl0oKQTk1eOTbkFF096TBdPfiIVT1ZyOQ+NkQ5wJnAl8LXk80nJ5ysJZyoXw8WTHtPFk5+IxdM+4slLY6RjZqeZmVJeK3oK6uJJj+niyU/E4mkf8eShMdIZGC6e9JgunvxELJ53j72U27UnW+DhPHWikY5GMlzs4kmP6eLJT6TiWT6yJ/9z+ku5DW7PUz4a6QBMn5VxyNbFkx7TxZOfSMVThKikAy6eSe25eFw8FRCddMDFM6k9F4+LZ8hEKR1w8Uxqz8Xj4hki0UoHXDyT2nPxuHiGRNTSARfPpPZcPC6eIRCNdEYyDpm7eFrac/G4eAZMNNIBmDkj/dwlF09Ley4eF88AiUo64OJx8eTHxTMYopMOuHhcPPlx8ZRPlNIBF4+LJz8unnKJVjrg4nHx5MfFU2Ifqu5A1bh4XDx5cfGU1H6lrQ+R0ZH0jdzF4+LJi4unhLYra7kCdpuZfsdzF4+LJy8unj7braTVCnHxpOPiyY+Lp482h95iDXDxpOPiyY+Lp8f2htpajXDxpOPiyY+Lp4e2htZSDXHxpOPiyY+Lp2A7Q2mlBoyq88bs4knHxZMfF0+BNgbeQo3YfZfOj5d18aTj4smPiydn/IFGLwlJT5b0j5Juk7RO0v2SrpX0jKKxXDzFYoCLpwgunhyxBxa5XF4GvBj4InAM8FZgT+BmSYcUDebiKRYDXDxFcPF0iTuQqOVzBfAMM/uEmV1nZt8EXgFsBN7ZS0AXT7EY4OIpgosnI2bpEQeAmT1oZtY2bTVwB7B3r3FdPMVigIunCC6elHilRhsikuYBTwOW9xPHxVMsBrh4iuDi6RCrtEjD5yJAwKfyFB7LuODTxVMsBrh4iuDiaYtTSpSCSDpSkuV4XZ9S/2zgNcCZZnZnRjuLJC2VtHTjqk3sOWNdap9cPMVigIunCC6eHagtVTIUJM0CHp+j6AYzu6et7hnAxcA5ZvaxvG3ueeAetvBLrwTggU1zUsut2tx55Vi9MX0r3Lhpeuq8hzdkrOSbRtPnASObsjeq0Q1d5qf7EoBpG7Ln54kBMLYh3zo0bWP+dW1sfbZYJ8XdkL/s9jrrMv4pZDC6fnNP9SbQ2s7/3Ppi/fryY3Zh2+o1O037t81fWWZmC7rVrWSkY2YbzOz3OV7twnk98BngE0WE046PeAI+4imOj3gC/Yx4GpPTkXQ88AXg82Z2Vr/xXDwBF09xXDyBXsXTCOlIeiHwVeA24DJJh7a8Du41rosn4OIpjosn0It4GiEd4AhgF+Bg4EbgFy2vb+YJME2dN2wXT8DFUxwXT6CoeBohHTNbbGZKee2XN84e0zsLxsUTcPEUx8UTKCKeRkinTFw8mbNdPD3g4ilGdNIBF4+LJ6OOi2fgRCkdcPG4eDLquHgGSrTSARePiyejjotnYEQjnbGUo1cunszZLp4ecPFkE410AB4z/S8dp7t4Mme7eHrAxZNOVNIBF0/qfBdPeh0XT6lEJx1w8aTOd/Gk13HxlEaU0gEXT+p8F096HRdPKUQrHXDxpM538aTXcfH0TTTSSTt65eJJme/iSa/j4umLaKQD8OhpqztOd/GkzHfxpNdx8fRMVNIBF087Lp4kroun/JgpRCcdcPG04+JJ4rp4yo/ZgSilAy6edlw8SVwXT/kx24hWOuDiacfFk8R18ZQfs4VopJN29MrFMxkXTxLXxVN+zIRopAPw2GmrOk538UzGxZPEdfGUH5PIpAMunglcPPlw8ZQvnuikAy6eCVw8+XDxlCueKKUDLp4JXDz5cPGUJ55opQMunglcPPlw8ZQjnmikM0bnDcfFE3Dx5MPF0794opEOwKOnbeg43cUTcPHkw8XTn3gaIR1JcyV9XdKdktZL+oukmyW9rmgsF4+LJzWmiyc/fYinEdIBpgPjwLnAscBrgN8DX5b0t0WDuXhcPKkxXTz56VE8jZCOmT1kZq8xs38xsyVm9j0zewNwE/CmXmK6eFw8qTFdPPnpQTyNkE4GDwEZW9YOxrTzSuficfGkxnTx5KegeBolHQWmSXqkpEXAy4FP5a3/2NGdf2gXj4snNaaLJz8FxDOt/NYHytuAi5K/twDvNLMvpRVOxLQo+bhuzmPv+cOA+1eEPYAHq+5EjfHvpzt1+472zVNIZjbojuzcqHQk8KMcRW8wsxe11NuTsGB7EBLKpwNvNbPPDaKfg0TSUjNbUHU/6op/P91p6ndU1Ujn58D8HOUm7fuY2QPAA8nHH0iaBZwv6VIzy5XbcRynWiqRjpltIBzy7pelwBuAvYB7S4jnOM6AaVQiuQOHA+uAP1fdkR64pOoO1Bz/frrTyO+okpxOUSSdDhwK/JgwonkkcDJwCvB+M/t4hd1zHKcATTl6dTuwEDgfmEfI2C8HXmlm362yY47jFKMRIx3HcaYOTc/pNB5JT5b0j5Juk7RO0v2SrpX0jKr7NmwkPU7SNyStlrRG0tWSHl91v+qCpBMlXSXpbkkbJf1B0rmS5lbdtyL4SKdiJJ1JOIHxi8AvgUcA7wUOBg4zs2UVdm9oJKc/3ApsBs4BDPgoMAt4upmtr7B7tUDSTcA9wDWE3ObBwGLCkeDnmVn26eE1waVTMZL2AB6ylh9C0m7ACuDbZnZqVX0bJpLeCVwAPMXM7kymPQH4v8B7zeyCKvtXByTtmZyr1jrtVMI/rJeY2U+q6VkxfPeqYszsQWszv5mtBu4A9q6mV5VwLHDThHAAzOwu4EbCQYToaRdOwi3Je2MipuDDAAADJ0lEQVTWFZdODZE0D3ga4QhdLBwE/KbD9N8CBw65L03i8OS9MeuKS6eeXASIAlfQTwHmAZ0u7V8J7D7kvjQCSXsDfw/82MyWVt2fvLh0SkbSkZIsx+v6lPpnE+6MeGbrrkYkdEow5r8XRURImkNIKI8Db6y4O4VoysmBTaKni1kBJJ0B/ANwjpldWnbHas4qwminnd3pPAKKFkkzgGuB/YHDzaxR1x26dEqm14tZJb0e+AzwCTP7WOkdqz+/JeR12jkQ+N2Q+1JbJI0BVwHPBo40s9sr7lJhfPeqBkg6HvgC8HkzO6vq/lTEtcChkvafmCBpP+CwZF70SBoBLgdeAiw0s5sq7lJP+Hk6FSPphcC/Ef6bnwmTngq42cx+VUnHhoyk2YSTAzey4+TAjwBzCScHpt/fNRIkXQycAXwM+E7b7Hubspvl0qkYSYuBv0uZfbeZ7Te83lRLcsnDJ4GXEhLIS4B3mdmKKvtVFyStIP2WoB82s8XD603vuHQcxxkqntNxHGeouHQcxxkqLh3HcYaKS8dxnKHi0nEcZ6i4dBzHGSouHcdxhopLx6kESZ+X9Mu2acuSK/D3aZl2vqS7JI0Ov5fOIHDpOFWxinCJAwCSjgCeQrj8Yfdk2izgTcCFZra1ik465ePScapiknSA9wCXEp7YOnHTrlOBUeBfhts1Z5C4dJyqWAXsCiBpPvAywp0S17BDOmcSrrxfk5T7QPLYlW2Sjqugz04JuHScqlgFzExyNe8BrjGz/wRWA7tLOhJ4KnBhS50lwNHAT4fdWac8/CZeTlVM3A3wAOB1wIuTzxMjnVcBV5vZ3RMVzOxmAMnvYNpkXDpOVUxI5wPAL83sF8nn1cAhwF8TbuDlTDFcOk5VTEjntcApLdPXAK8Gbm7qnfGcbDyn41TFhHTuAb7VMn014YhV9E/0nKr4SMepBDN7kA6PlzGzNwNvHn6PnGHhdw50GoOkcwj3CN4TWAtsAhaY2Z8q7ZhTCJeO4zhDxXM6juMMFZeO4zhDxaXjOM5Qcek4jjNUXDqO4wwVl47jOEPFpeM4zlBx6TiOM1RcOo7jDJX/D11jmjDYrN4hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gridX = np.linspace(-3,3,1000)\n",
    "gridY = np.linspace(-3,3,1000)\n",
    "meshX, meshY = np.meshgrid(gridX, gridY)\n",
    "\n",
    "def make_plot(loss_fun, **kwargs):\n",
    "    loss = loss_fun(np.c_[meshX.ravel(), meshY.ravel()].T, X, y, **kwargs)\n",
    "    min_inds = np.nonzero(loss<=np.min(loss)*(1+1e-7))\n",
    "    loss = loss.reshape(meshX.shape[0],meshX.shape[1])\n",
    "    # plt.imshow(loss,extent=(np.min(gridX), np.max(gridX), np.min(gridY), np.max(gridY)), origin='lower')\n",
    "    CS = plt.contourf(gridX, gridY, loss, 30);\n",
    "    plt.plot(meshX.flatten()[min_inds], meshY.flatten()[min_inds], '.r');\n",
    "    plt.axis('square');\n",
    "    # plt.colorbar();\n",
    "    plt.xlabel('$w_1$');\n",
    "    plt.ylabel('$w_2$');\n",
    "make_plot(f)\n",
    "plt.title(\"No regularization\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a \"smear\" of solutions (red) along the $w_1+w_2=w_\\text{total}$ axis for some $w_\\text{total}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w_total = 1.7691\n"
     ]
    }
   ],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X,y)\n",
    "print(\"w_total = %.4f\"%np.sum(lr.coef_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR0AAAEmCAYAAABf10QVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztnXnYJGV5r+/f980Hs4IgIwqoI0IUUBFBQ1ADAgZXQA9IYlyQHIEknGgM0cDxHFGjHs+lHpdLNMQFzeUuqxoTBQSULQ7uihISFjEmggwDszDM8pw/qj66v57q7qru6q6urt99XX31dNVbz/t0T/X9ve9TSysiMMaYcTFTdQLGmGZh6RhjxoqlY4wZK5aOMWasWDrGmLFi6RhjxoqlYypD0smSQtLJde6jQC7np7msqjqXKrF0RoSkVekOdnGOto+QdJakCyTdlm537zjyNOXR9n9+ftW5TDKLqk7AALA/8E5gG/CvwAPVpjNVXARcD/y66kSAs4D/A/yq6kSqxNKZDG4CDge+HxH3S7oNeFi1KU0HEbEWWFt1HgAR8WsmQ36V4unVBBAR/xURV0fE/cPEaa9fSHqJpOslrZd0ZVubGUmvlXSDpHXp41pJL+0S82mSvpm2W5NOAR8r6UpJ0dG2a82iSD1D0imSLpV0u6RNku6WdImkQ4q+56yaznzuPR7ntLU9UtInJd2cxr0//bxO6swDuDV9+eqOeKt6fQaSlkv627SPTZLuSj/nJ2e83/nc5yS9te0z+qmkP+r32U4CHulMJycBRwGXAFcCWwAkCfgc8DLgZ8Cn0vYvBC6Q9PqI+MB8EEkHAVcDOwJfBO4Ang18B7hnhPl/GPgB8A3gt8BjgeOBYyQdERHXZ2yT+Z67cH7appMTSaa6G9qWvRHYG7iBZFq0K3As8HlJj4qI96ftfgB8AHgd8EOgvZbXtT4naTFwBfD0tI8LgL1I/o+eJ+mYiPhOxqafB54GfI3ke/xHwGcl3RsRX+/W30QQEX6M4AGsAgK4eIBtbwPuHWC7k9M+twDPzlh/Wrr+w8Bs2/JlJDv8JmCPtuXXpO1f2BHn79Pl0bH8/HT5qoy+t1vXlu/JHW0fl7H9fsD9wGUF33NmHxntXgxsBb4LLOmTyzISsawFlmb8n5/fpY+sz+At6bKPA2pbfjitGt9M2/Ir0/bXAss72gfwz1Xv+/0enl5NJxdFxLczlv85sAb4y4jYOr8wItYDbwd2AF4KyZEY4DDguoj4Wkecc+g9khiKiLg1Y9lNwLeAZ0vaIWOzbu+5L5KeAnyWpN5yXERs7JPLepJR4k7AMwbps41Xkxw4ODtSe6R9XAVcCuwDPDNju7MjYl1H+9uA7aagk4anV9PJ6s4FkpYCTyKZIp2dzLQWsDJ9fmL6fGD6fG1nw4j4laQ7SKYdpSNpH+Bs4DnAHiQybOfhbF+Q3e495+xrd+ArJPXNYyPiPzrW70QyxTqO5P0u7QjxqEH6bYv9OJIDCP+V0eTKtN+nAp1C/X5G+18Bjxk0n3Fh6Uwnv8lYtgsgkvrIW3psuyx9XpE+39Wjj9KlI2lf4F/S/i8jOeS9jmSqcTyJDHfskk/RvhaT1F4eDZwYEd/rWL8DcBXJl/5GkunRPSTTsKeSCCErl7zslD5nCQfgPzvaPUQkR+U62UINDg5ZOtNJ1p3Z7kufr4mIZ+WIMX8kbWWX9Y/IWLYtfZ7NWLfdF6cLryc5XeDlEfG59hWSfpfWCKyTQe5G9wngUODNEXFBxvr5UcZ5EXFaRy5vStcPw/z/ye5d1u/e0W4qmHgrmnKI5HD8z4EnSVqeY5Mfps+Hda6QtCfZw/j5ozR7drSfobssOnl8+vyVjhhLSI7WlIKk/01yxOczEfGOIrmkZNVZ5utkWdLdjoi4j+Qw+36SsuR+ePr8gzzx6oKl0yw+BOwMnJtOLRYg6QBJjwCIiNtIzuT9PUkv7Gh6Dtmj5BvT51d1LP8L8k/F7kifH/pSp4f630n26Kowkk4keQ/XAX9SJJd0+5eSHO3qZE36vGfGum58GlgMvK2jj2eRTCf/jeQo4tTg6dXoeVqPa3G+ERGfheTEsbbluwE7tC+LiJNLyOUjJCOXVwJHSLqCpG7wKOApJFOJ36NVHzmD5DydiyS1n6ezCvgR0Hny2kXA7cCfSNoL+DFwUBr3Klp/uXvxUeA1wIWSvkByWPrZJAXXK4Ejir3lTD5FUt/6IXBWRlH9yoi4kmSEcwfwJkkHkIwUDwCeR/JeX9K+UUSsk/Rd4HBJHyMRRgAf6VKDAXg38CLg9PQo2tUk0jqJ5KjWKRGxrcu29aTqY/bT+qB1zkavx/vb2vdsm7PPk8l3Tsofkxx+XkNybs4dwD8Dfwos62h7CElBd33a/sL0vf0YWJsRex/gqyTF3/tIDvvuS7HzdI4iOWp2P8nJgRcUjdFrfY7/l3Pa2j6eRDB3pfl8GzimR+5PTD/LtW3xVqXrtss/Xb6CZCR3C/Bg+p4vBA7MeD9Xdtsfeq2bpIfSZCceSccAbyI5Y3QXkp3gWpId5GdV5tY00prQb4CfRMSw56mYhlGn6dWuJDWDc0mE8xjgb4DrJT05Im6vMrlpRNIcsFNE/LZt2QzJlGAJySUHxhSiNiOdLCQ9gWSefWZEvLfqfKYNSbsBd5JcA3UzyTk8zyI5yfDnwNOj7axYY/JQp5FOFvN/gTdXmsX0sg74JEmN5TkkJ8L9Eng/8HYLxwxC7UY6kmZJzoN4LMkNkQ4jKbgVPiPVGDN+6jjSuQE4OP33LcCRFo4x9aGOI539SE6p3xs4k+RU8WdFcjJbZ9tTgVMBZjV38LK5XZMV2yo+7WHr1v5tmsBsrhN3R8PMCM6LnR0uZsxsd77Q6LYrkGrM5ou//p47746IbpfNPETtpNOOpIeRXM7/+Yg4vVfbnXd8ZBy25ytaC9avH2lu/di2dqoupxmKmZ3zXpZVMsuW9W9TkFixZKjtty4b7PrRLcvnim+zNL95Ni/r3/aGz555Y0T0vbVGrS+DiIh7SaZY+xTeeAQ7XBEq+6JNIJUJeAR/eHT/xv6NejC7ftNA2y1aV/xYyqIN+Uf8c+vLmx3UWjrpvVCeSHK6eXEsnonB4mkx7eKpjXQkXSTpf0k6TtJzJJ1Gcj3PFmDwc3QsnonB4mkxzeKpjXRIrng+nuRiva8BbyC9wVJE3DxU5IrFY1pYPC2mVTy1kU5EvDsiDo6Ih0XE0oh4QkSclnXUaiAqFI9HOwuxeFpMo3hqI52xYPFMDBZPi2kTj6XTicUzMVg8LaZJPJZOFhbPxGDxtJgW8Vg63bB4JgaLp8U0iMfS6YXFMzFYPC0mVTx5sXT6YfFMDBZPizqLx9LJg8UzMVg8LeoqHksnLxbPxGDxtKijeCydIlg8E4PF06Ju4rF0imLxTAwWT4s6icfSGQSLZ2KweFrURTyWzqBYPBODxdOiDuJpjnSGvJVkJhbPxGDxtJh08TRHOgx/K8lMLJ6JweJpMcniaZR0wOKZdiyeFpMqnsZJByyeacfiaTFO8eSlkdIBi2fasXhaTJp4GisdsHimHYunxSSJp9HSAYtn2rF4WkyKeBovHZg+8ZiFWDwtJkE8lk7KNInHo53tsXhaVC0eS6cNi2e6sXhaVCkeS6cDi2e6sXhaVCWe2khH0gmSLpB0u6SNkn4h6V2SVpTdl8Uz3Vg8LaoQT22kA5wJbAXOBp4HfAT4U+Cbkkp/HxbPdGPxtBi3eBYNtFU1vDgi7mp7fZWke0h+ZvgI4IqyO4wVS4b+D92OZctGsuP1Y2bnnar7ok0o29beV42Q168v/Q+Q7t841B/K2fWb2Lpsx8LbLVq3mS3L5wptU5uRTodw5vlu+rznyPr1iGeq8YinxbhGPLWRThcOT59vGmUnIxFPRVg822PxtBiHeGorHUl7Am8DLouI1f3ax4yG6q908fis5YnC4mkxqHjyUkvpSFoOXAJsAV7To92pklZLWr158/qB5qztWDzTjcXTYpTiqZ10JC0GLgX2Bo6JiDu7tY2I8yLikIg4ZG4u+YJbPC0snu2xeFqMSjy1ko6kOeAC4BnACyLix4PEsXhaWDzbY/G0GIV4aiOd9FyczwBHAcdFxPXDxLN4Wlg822PxtChbPLWRDvBh4ETgPcB6SYe2PfaqIiGLZ7qxeFqUKZ46Sef56fP/BK7rePz3QQIOO9oBi2cU7LftLj686R+5YNMX+esHr6k0F4unRVniqY10ImJVRKjL45xB41o8C6laPPttu4v3br6Mx3Mvy9jCUXE7n9x0caU5WTwtyhBPbaQzSiyehVQpnqds+w0zBALmz6x6FBssnhKpWjyWTorFs5CqxPOjmUcQQKSvLR6mTjyWThsWz0KqEM9NMyv54OzTHxJPsFA8H930lbHn1I7F02JQ8Vg6HVg81fP1Rfvyhrnnsqlt95wXzyrut3hKpArxWDoZWDwtqppm3TSzkr+ZOypzqmXxlMu4xWPpdMHiaVGleNqnWmDxALUXj6XTA4unRVXi+fqifXuKx8Xl8ij9hnVdaIx0Br21hcXTYhLF4+JyuYxDPI2RDlD4torzWDwtJlE8nmqVy6jF0yjpgMVTBhZPNhZPPhonHbB4ysDiycbi6U8jpQMWTxlMqnhcXC6PUYinsdIBi6cMJlE8vmSiXMoWT6OlAxZPGVg82Vg82TReOmDxlMGkiMfXaqVMsHgsnRSLZ3iqFI+v1cpgQsVj6bRh8QzPpF6r5alWeQwrHkunA4tneCbxWi3XeMplGPFYOhlYPMMzKTUesHiAiRKPpdOFKsVTOg0Xj4vLKRMinuZIZ4B3WpV4Sh/tQCPF4+JyBhMgnuZIB9iytPjbtXiGx8XlbJoqnkZJByyeJorHNZ4MRiCevDROOmDxNE08Li53oSLx1EY6kvaS9CFJ10naICkkrRo0nsXTbPG4uJxSgXhqIx1gH+BlwBrg22UEtHj8KxMuLjN28dRJOldHxO4R8QLgS2UFtXjGL54qf0HUxeUujFE8tZFORGwbVWyLp3nicY0ngzGJpzbSGTUWT7PE4+JyF8YgnqmWjqRTJa2WtHrzg/0/zMaLpwImSTwuLqeMWDxTLZ2IOC8iDomIQ+Z2yPeXvNHiadgRLXBxuSsjFM9US6edmM3/u1cWz/ipusbj4nIGIxJPY6QDsHlZ/rdr8YyfqsXjGk8GIxBPo6QDFk9uGigeF5e7ULJ4aiUdSSdIOgE4OF30/HTZ4UXiWDw5sXhcXJ6nRPHUSjokJwV+CTg9fX1u+vqtRQNZPDlpqHhcXM6gJPHUSjoRoS6PIwaJZ/HkpIHicXG5CyWIp1bSGQUWT04aKh7XeDIYUjyNlw5YPLlpoHhcXO7CEOKxdFIsnpxYPC4uzzOgeCydNiyenDRUPC4uZzCAeCydDiyenDRQPP2KyxZPPhojnSjwTi2enDRUPN1qPBZPPhojHYAtS/Jff2Xx5KSB4ulVXG704fScNEo6YPGAxVMG/Y5qNXbEk4PGSQcsHrB4yqDfiMfiyaaR0gGLByyeMrB4itNY6YDFAxZPGVg8xWi0dMDiAYunDFxczk/jpQMWD1g8ZeBLJvJh6aRYPBZPGVg8/bF02iginkJxLZ6xM0ni8bVaC2mMdPKekZxXPEVGO2DxVEHV4vG1Wtk0RjoAm5fmE8okiWdQLJ6Eqi+Z8I3AtqdR0oH6iWfQ0Q5YPPNULR7XeBbSOOmAxVMEi2d4XFxeSCOlAxZPESye4XFxuUWhPV/S4yRdLunfJb1P0uK2df9SfnqjpUniGRaLZ3hcXE4oOtI5F7gQOBHYFbhc0op0XXXfiCFoiniGHe2AxVMGLi4Xl87uEfHhiLgxIk4Gvkoinp1pfYa1w+LJj8UzPE0vLheVzoK9NiLeBXwRuBxYkblFSUh6tKQvS1or6T5JF0p6TFnxLZ78WDzD0+TiclHp3Czpue0LIuI9wGeBx5eWVQeSlgJXAE8EXg28EtgX+JakXHts5HCFxZMfi2d4mlpc7ruXS2r/hv0hcHVnm4h4H/DoEvPq5LXA3sDxEXFxRFwCHAs8Fjgtb5Cti/u3sXjyY/EMTxOLy3n28Fsk/YWk5RGxKSI2ZTWKiF+VnFs7xwLXR8Qtbf3dClwDHFckkMXTwuJpUXWNp0m/MpFn774IeAvwS0n/V9IoRzTdOAD4ScbynwL7j6JDiyc/Fs/w9PuViVM2f7+izBLKFE/fPTsizgT2As4Cng/8m6TPS3p6aVn0Z1dgTcbye4Bdum0k6VRJqyWt3rqx9RMZeUY7YPEUweIZnl7F5Wdu+2VFWbUoSzy59uqI2BgRH42IJ5OIZwlwraRvS3pJKZnkSCNjWc9ve0ScFxGHRMQhs0sW7sQWz0IsnhaTIJ5tsEA+18xUMcHYnjLEs6hfA0l7kIwm2h9XAPeRFJa/DMwOnUlv1pCMdjrZhewRUC62LobZB/q327xUzG3ofxrSliVi0cb+7TYvm2Fu/bY8KSZxl86waEOB9svnWLRuc+7282xdtiOz6zNLdrmJFUvQ/RuHirGAZcsG/s3sYZjZeafKrsL++qJ9uW3mYZyy5Qc8MtbxrZlVfGLuoEpyyWLb2vuGEnNf6QB3AtuAdcD9wFoS4awlOTt57cC95+enJHWdTvYHfjZMYItnIRZPiyrFc9PMSv56h+f2b1gRw4gnz9j9+8BG4Hzg2RHxpIg4LCKeHxEnRcSpA/VcjEuBQyXtPb9A0irgmem6ofBUayGearWocqo16Qwq5DyF5INJDlnvA/xC0hckPWOg3gbn74HbgEskHSfpWOAS4JfA3+UJ0O/OgRbPQiyeFhZPdwYRT95C8rci4kXAU0mmWFdJ+o6k4ztOHhwJEbEeOBK4GfgH4DPArcCREbEub5wtS3uvt3gWYvG0sHi6U1Q8hfbeiLiJ5OzgA4FbSK67urlQjwMSEXdExH+LiJ0iYkVEHB8RtxWNY/EUw+JpYfGUQ57LIG6U9AtJ/yFpHbAFuAl4FUkheu+eASYQi6cYFk8LiyebIqOdPEevvkdyWLrXo3ZsWQqLNnRf76NaC/FRrRZVHtWaBvpKJyJeO45EqsDiKYbF02J+xGP5FKex90iex1OtYniqtRBPt4rTeOmAxVMUi2chFk8xLJ0Ui6cYFs9CLJ78NEc6Od6pxVMMi2chFk8+miMd8knD4imGxbMQi6c/jZIOWDw921s8pWDx9KZx0gGLp2d7i6cULJ7uNFI6YPH0bG/xlILFk01jpQMWT8/2Fk8pWDzb02jpgMXTs73FUwoWz0IaLx2weHq2t3hKweJp0RjplHETL4unGBbPQiyehMZIB2Dr0t4XRVo8PdpbPKVg8TRMOmDxLIhr8VRC08XTOOmAxbMgrsVTCU0WTyOlAxbPgrgWTyU0VTyNlQ5YPAviWjyV0ETxNFo6YPEsiGvxVELTxNN46YDFsyCuxVMJTRJPc6Sj3vcmtnja4lo8lTCz806NkE9tpCPpDZK+IunXkkLSOUVjbFvcRywWTyuuxVMZ0y6e2kiH5Ef+HgFcPEwQiyc/Fk91TLN46iSdAyLid4H/MWwgiyc/Fk91TKt4aiOdiMj/w045sHjyY/FUxzSKpzbSGQUWT34aLx4f2SqNRksHLJ4iNFo8YPGURCXSkXR0egSq3+PKIfs5VdJqSau39vjpWYsnPxaPxTMsfX/LfERcC+yXo12PXxrvT0ScB5wHsOPj9goWb4UHZjPbblu8jZkHun+hti7dxuyGHutz/O65fzu9GBP52+lQ2e+nQyKeuv9+eiXSiYgNwM+r6Nvi6RLP4imGxTMwzazpLN7adZWnWvnxVMtTrUGojXQkHSLpBOCl6aL9JZ2QPvp8jTOweLLjWTzFsHgKUxvpAGcAXwK+kL4+MX39JZIzlYtj8WTHs3iKYfEUojbSiYiTI0JdHrcNHNjiyY5n8RTD4slNbaQzLJrpUUy1eLLjWTzFsHhy0RjpAOywtMfRE4snO94Ei2dY+Vg81dAo6YDFs6C/mosHhh/1WDzjp3HSAYtnQX8Wj8UzZhopHbB4FvRn8Vg8Y6Sx0gGLZ0F/Fo/FMyYaLR2weBb0Z/FMpXgmTT6Nkc5Mj0PmFk9bfxbP1IkHJmvU0xjpACxZ/GDXdRZPW38Wj8UzQholHbB4LJ78WDwjyqHqBKrA4rF48mLxjKD/SnuvEIvH4smLxVNy35X1PAFYPBZPXiyeEvutpNcJwuKxePJi8ZTU59h7rIjZme5fdIvH4smLxVNCf2PtrWJ2XtL95sMWj8WTF4tnyL7G1tOEYPF0x+LJj8UzRD9j6WXCsHi6Y/Hkx+IZsI+R9zChWDzdsXjyY/EMEH+k0Scci6c7dRRPVbc/tXgKxh5Z5Jpg8XSnbuKB6u67bPEUiDuSqBPIrLp/mS2e7lg8+YkVS8qXzxSKpzHSAdhlx+4/LWvxdMfiKYbF0ydeqdFGhKTfkfQBST+StE7SryVdKunAorEsnuIxwOIpisXTI1ZpkUbLHwDPAT4FvBj4M2AlcIOkg4sGs3iKxwCLpygWT5c4pUQZPZ8HDoyI90bEtyLiIuB5wEbgdYMEtHiKxwCLpyjTKJ5h5VML6UTE3RERHcvWAjcDew4a1+IpHgMsnqJMm3hguFFPLaSThaRdgScBN+VpP9flgk+Lp3gMsHiKYvG0bVdyHuPkQ4CA9+fdYOXidZnLLZ7iMcDiKYrFk24zgjz6IuloSZHjcWWX7c8CXg6cERG39OjnVEmrJa3euCaRh8WTYPEUZ2LFMwF1niKoo1QyFiQtBR6To+mGiLijY9vTgY8Ab46Id+Ttc+X+u8Vxn37RQ6/vemB5Zrs1m7rvFGs3dv8mbnxgh67rHtzQYyd/YLbrqpkHen+hZjf0Wd/dlQ+xaEPv9XliAMxtyLcfLdqYr93c+t5i3S7uhmLtARat6/FHoQez6zcNtF07ur/7H7mBWb++/JgF+KfffPTGiDikX7tF40imk4jYAPy86HaSXgmcC7y3iHCyWLl4XaZ4dtlxY1fx7Lzkga7iWbL4wa7i2WHp5u7iWby1q3i2Ld7WUzxbl27rKZ6ti/tLY8vS3uLJEwOSEU8e8WxZolzi2bxsppB4tiydKSyeLcvnBhLP1mU7Di2eWLGkfPEsW1a5ePJQm5qOpJcAnwQ+FhFnlhHTU60ET7WKM5FTLah8qpWHWkhH0u8DnwN+BJwv6dC2x0F5YixS9hfb4kmweIpj8QxGLaQDHAnsCBwEXANc1/a4KG+Q3XbIFozFk2DxFMfiKU4tpBMR50SEujxWFYll8fTG4imOxVOMWkinbCye3lg8xbF48tNI6YDF0w+LpzgWTz4aKx2wePph8RTH4ulPo6UDFk8/LJ7iWDy9aYx05rocMgeLpx8WT3Esnu40RjoAj9rh3q7rLJ7eWDzFsXiyaZR0wOLput7i6b2NxVMajZMOWDxd11s8vbexeEqhkdIBi6fr+pqLp4h8LB4qEU9jpQMWT9f1NRYPFBv1WDyMXTyNkU63o1cWT5f1Fk/vbSyegWmMdAAeuWht5nKLp8t6i6f3NhbPQDRKOmDxdGLxtMW1eMYinsZJByyeTiyetrgWz8jF00jpgMXTicXTFtfiGal4GiOdrEKyxbMQi6ctrsUzMvE0RjoAeyxas90yi2chFk9bXItnJOJplHTA4gGLx+IpSMniaZx0wOIBi8fiKUiJ4mmkdMDiAYvH4ilISeJprHTA4gGLx+IpSAniaYx05sj+8lg8Fo/FU5AhxdMY6QA8ssvv51o8Fo/FU5AhxNMo6YDFY/H0iGnxFGNA8dRCOpJWSPqipFskrZd0r6QbJL1ikHgWj8XTNeYEi2dY+UyKeGohHWAHYAvwLuBY4OXAz4F/kPSXgwS0eCyerjEnVDww/KhnEsRTC+lExG8j4uUR8fGIuDwi/jEiXg1cD5wyaFyLZ/LFkyeOxVOMqsVTC+n04LdAj29Vizll73AWz2SLJ28ci6cYIxFPTmolHSUskvRwSacCxwDvz7v9HrPZ/8kWj8XTNabFUzqLKul1cP4c+FD6783A6yLi090ap2I6NX25bvked/xixPkVYTfg7qqTmGD8+fRn0j6jx+ZppIgYdSLbdyodDXwzR9OrIuKItu1Wkryx3UgKyqcBfxYRfzeKPEeJpNURcUjVeUwq/nz6U9fPqKqRzrXAfjnaLZj3RMRdwF3py3+StBR4j6RPRESu2o4xploqkU5EbCA55D0sq4FXA7sDd5YQzxgzYmpVSM7gcGAd8JuqExmA86pOYMLx59OfWn5GldR0iiLpNOBQ4DKSEc3DgZcBJwF/ExHvrjA9Y0wB6nL06sfAccB7gF1JKvY3AS+KiK9VmZgxphi1GOkYY6aHutd0ao+k35H0AUk/krRO0q8lXSrpwKpzGzeSHi3py5LWSrpP0oWSHlN1XpOCpBMkXSDpdkkbJf1C0rskrag6tyJ4pFMxks4gOYHxU8D3gIcBbwQOAp4ZETdWmN7YSE9/+CGwCXgzEMDfAkuBp0TE+grTmwgkXQ/cAVxCUts8CDiH5EjwYRHR+/TwCcHSqRhJuwG/jbb/CEk7A7cBX4mIV1WV2ziR9DrgfcATIuKWdNnjgH8F3hgR76syv0lA0sr0XLX2Za8i+YN1VERcUU1mxfD0qmIi4u7oMH9ErAVuBvasJqtKOBa4fl44ABFxK3ANyUGExtMpnJTvps+12VcsnQlE0q7Ak0iO0DWFA4CfZCz/KbD/mHOpE4enz7XZVyydyeRDgChwBf0UsCuw/WX9cA+wy5hzqQWS9gTeBlwWEaurzicvlk7JSDpaUuR4XNll+7NI7ox4RvtUoyFkFRjz34eiQUhaTlJQ3gK8puJ0ClGXkwPrxEAXswJIOh14J/DmiPhE2YlNOGtIRjud7EL2CKixSFoMXArsDRweEbW67tDSKZlBL2aV9ErgXOC9EfGO0hObfH5KUtfpZH/gZ2POZWKRNAdcADwDODoiflxxSoXx9GoCkPQS4JPAxyLizKrzqYhLgUMl7T2/QNIq4JnpusYjaQb4DHAUcFxEXF9xSgPh83QqRtLvA98g+Wt+Biz4KdJNEfH9ShIbM5KWkZwcuJHWyYFvB1aQnByYfW/XBiHpI8DpwDuAr3asvrMu0yw6ntiNAAAB20lEQVRLp2IknQO8pcvq2yNi1fiyqZb0kof/BzyXpIB8OfD6iLityrwmBUm30f2WoG+NiHPGl83gWDrGmLHimo4xZqxYOsaYsWLpGGPGiqVjjBkrlo4xZqxYOsaYsWLpGGPGiqVjKkHSxyR9r2PZjekV+Hu1LXuPpFslzY4/SzMKLB1TFWtILnEAQNKRwBNILn/YJV22FDgF+GBEbK0iSVM+lo6pigXSAf4K+ATJL7bO37TrVcAs8PHxpmZGiaVjqmINsBOApP2APyC5U+J9tKRzBsmV9/el7c5Of3Zlm6TjK8jZlIClY6piDbAkrdX8FXBJRPw7sBbYRdLRwBOBD7ZtcznwAuDqcSdrysM38TJVMX83wH2AVwDPSV/Pj3ReClwYEbfPbxARNwBIvoNpnbF0TFXMS+ds4HsRcV36ei1wMPBCkht4mSnD0jFVMS+dPwZOalt+H/CHwA11vTOe6Y1rOqYq5qVzB3Bx2/K1JEesGv+LntOKRzqmEiLibjJ+XiYiXgu8dvwZmXHhOwea2iDpzST3CF4J3A88ABwSEf9ZaWKmEJaOMWasuKZjjBkrlo4xZqxYOsaYsWLpGGPGiqVjjBkrlo4xZqxYOsaYsWLpGGPGiqVjjBkr/x/Pa/VRBaZkLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "make_plot(f1, λ=1)\n",
    "plt.title(\"L1 regularization\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This is very interesting! We still have $w_1+w_2=w_\\text{total}$ but...\n",
    "- we can't have one of them take the opposite sign of $w_\\text{total}$ (in this case positive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR0AAAEmCAYAAABf10QVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztnXm0JVV97z/fe/tCjyDIoIDaIg6AgggaghIQiTgC+kQSo4LmCSaSaBKWA8/3JBrjsJRoWGJCHNAsEQdAUN97UcEGlSE2DigyhIRBjHk00nTTt5u2u+/v/bHr9j339BmqzlSnbn0/a9Wqe6v2/u3fObfO5+7atauOIgJjjBkVE2UnYIypF5aOMWakWDrGmJFi6RhjRoqlY4wZKZaOMWakWDqmNCSdLikknV7lNgrkclGWy8qycykTS2dISFqZHWBfy1H2aEnnSfqxpLWSNkn6uaT3SFo6inxN/zT8zS8qO5dxxtIZD74C/BmwFvgMcCGwCDgX+L6kZeWlVnkuBw7M1mXzLlIuvyo7kTJZVHYCBoDzgM9FxP+b3SBpEfBV4CTgLcCHS8qt0kTEOmBd2XkARMSvgV+XnUfZuKczBkTEhxuFk23bCnwo+/X38sRpHL+Q9ApJN0ialrSqocyEpDdJulHShmy5TtIr28R8lqRvZ+XWSrpU0hMkrZIUTWXbjlkUGc+Q9EZJV0q6R9JmSQ9IukLSEUVfc6sxndncOyznNpQ9TtJnJd2RxX04e79Obc4DuCv79bSmeCs7vQeSlkv6m6yNzZLWZO/zM1q83tncpyT9dcN7dIukP+z23o4D7umMN1uy9daC9U4FXgBcAayarS9JwBeBVwO/AD6XlX8pcKmkt0XEx2eDSDoMuBbYGfgycC9wNPB94MHiLyc3nwB+AnwL+A3wBOBk4ARJx0bEDS3qtHzNbbgoK9PMKcBBwMaGbW8H9gduJJ0W7Q6cCFwi6bER8bGs3E+AjwNvBX4KNI7lPdQuEUmLgauBZ2dtXArsR/obvUjSCRHx/RZVLwGeBXyT9Dn+Q+BiSQ9FxP9p195YEBFehrAAK4EAvtZHjPOzGG/JWf70rPxW4OgW+8/M9n8CmGzYvox0wG8G9mnY/oOs/Eub4vxTtj2atl+UbV/Zou0d9jXke3pT2Se2qH8g8DDwnYKvuWUbLcq9HNgG/BBY0iWXZSSxrAOWtvibX9SmjVbvwXuybZ8G1LD9GGAG+DdgomH7qqz8dcDypvIB/EvZx363xadXY4qk44E/AW4nHZBFuDwivtdi+1tIg9V/ERHbZjdGxDTwPmAn4JVZ+yuBo4DrI+KbTXHOpXjvKzcRcVeLbbcC3wWOlrRTi2rtXnNXJB0CXEwabzkpIjZ1yWWa1EvcBXhOL202cBrwCHBOZPbI2rgGuBI4AHhui3rnRMSGpvJ3Azucgo4bPr0aQ7Jz+S8DG4BXR8QjBUOsbhFzKfB00inSOelMax57ZuunZetDs/V1zQUj4leS7iWddgwcSQcA5wDPB/YhybCRR7PjgOwOrzlnW3sDXyeNb54YEf/ZtH8X0inWSaTX2zyF4bG9tNsQ+4nAj6NpTC9jVdbuM4Fmof64RflfAY/vNZ9RYemMGZKeCnwbmAJeGBE39xDm/hbbdgNEGh95T4e6s5fnV2TrNR3aGLh0JD0Z+Nes/e+QLnVvIJ1qnEyS4c5t8ina1mLS2MvjgFMi4kdN+3cCriF96G8inR49SDoNeyZJCK1yycsu2bqVcAD+q6ncdiJdlWtmKxW4OGTpjBGSnkI6hVgBvDgiru8xVKsns63P1j+IiOfliPFwtt6zzf69WmybydaTLfbt8MFpw9uARwGviYgvNu6Q9DvM9cCa6eVpdJ8BjgTeHRGXttg/28u4MCLObMrlHdn+fpj9m+zdZv/eTeUWBGNvxbqQnVJcTfrAvTwirh1k/Ih4GLgNeLqk5Tmq/DRbH9W8Q9K+tO7Gz16l2bep/ATtZdHMk7L115tiLCFdrRkIkv4X6YrPFyLi/UVyyWg1zjI7TtZKujsQEetJl9kPlNRK7sdk65/kiVcVLJ0xQNL+pB7OHsDJEXH1kJo6H9gVuCA7tWjO42BJewFExN3ADcDvSnppU9Fzad1Lvilbv75p+5+T/1Ts3my9/UOdXer/W1r3rgoj6RTSa7ge+OMiuWT1X0m62tXM2my9b4t97fg8sBh4b1MbzyOdTv476SrigsGnV8PnWWp/L863IuJiUg9nP9KcmKMkNfcuHoq5+SD98ElSz+V1wLGSriaNGzwWOIR0KvG7zI2PnJXldLmkxnk6K4GbgebJa5cD9wB/LGk/4GfAYVnca5j7z92JfwDeAFwm6Uuky9JHkwZcVwHHFnvJLfkcaXzrp8C7Wgyqr4qIVaQezr3AOyQdTOopHgy8iPRaX9FYKSI2SPohcIykT5GEEcAn24zBQJoA+jLgzdlVtGtJ0jqVdFXrjREx06ZuNSn7mv1CXZibs9Fp+VhWtlu5u3O2eTr55qT8EalntZY0N+de4F9Il+iXNZU9gjSgO52Vvyx7bT8D1rWIfQDwDdLg73rSZd8nU2yezgtIV80eJk0OvLRojE77c7zf5zaUfRJJMGuyfL4HnNAh96dl7+W6hngrs3075J9tX0Hqyd0J/DZ7zZcBh7Z4Patomh+VZ984LcqSHXsknQC8gzRjdDfSQXAd6QD5RZm51Y1sTOh+4OcR0e88FVMzqnR6tTtpzOACknAeD7wTuEHSMyLinjKTW4hImgJ2iYjfNGybIJ0SLCHdcmBMISrT02lFNqflNuDsiPho2fksNCTtAdxHugfqDtIcnueRJhneBjw7GmbFGpOHKvV0WjH7H3hLx1KmVzYAnyWNsTyfNBHul8DHgPdZOKYXKtfTkTRJmgfxBOCDpKsxh0ZE4RmpxpjRU8Wezo3A4dnPdwLHWTjGVIcq9nQOJE2p3x84mzRV/HmRJrM1lz0DOANgUlOHL5t81AgzbWDbtu5l6sxkrgm8g2diCHNjJ/uLGRM7zBkaXr0CqcZk9/jTD973QES0u21mO5WTTiOSHkW6nf+SiHhzp7K77vyYOGrf18L09Ehya2Zm3YK6fWbgTOya99asIbBssI+gjhVL+qq/bVlv95BuXT5VvM7S/ObZsqxz2RsvPvumiOj6aI1K3wYREQ+RTrEOyF1pwAdYXkr9UFWAUqU84H9EenhT90IdmJze3FO9RRuKX09ZtDH/ZOep6cFMjK60dLJnoTyNNN08PxbPWGLxzLGQxVMZ6Ui6XNL/lHSSpOdLOpN0P89WoPgcHYtnLLF45lio4qmMdEh3PJ9Mulnvm8Bfkj1gKSLu6CmixTOWzKxbX558LJ5c9COeykgnIj4UEYdHxKMiYmlEPDUizmx11aoQFs/YYvEkFpp4KiOdoWLxjC0WT2IhicfSmcXiGVssnsRCEY+l04jFM7ZYPImFIB5LpxmLZ2yxeBLjKp68WDqtsHjGFosnUWXxWDrtsHjGFosnUVXxWDqdsHjGFosnUUXxWDrdsHjGFosnUTXxWDp5sHjGFosnUSXxWDp5sXjGFosnURXxWDpFsHjGFosnUQXxWDpFsXjGFosnMe7isXR6weIZWyyexDiLpz7S6fPZtTtg8YwtFk9iXMVTH+nQ/7Nrd8DiGVssnsQoxZOXWkkHLJ46YfEkxk08tZMOWDx1wuJJjJN4aikdsHjqhMWTGBfx1FY6YPHUCYsnMQ7iqbV0wOKpExZPomzx1F46YPHUCYsnUaZ4LJ0Mi6c+WDyJssRj6TRg8dQHiydRhngqIx1Jr5J0qaR7JG2SdLukD0haMch2LJ76YPEkRi2eykgHOBvYBpwDvAj4JPAnwLclDfR1WDz1weJJjPSWiZ5aKoeXR8Saht+vkfQg6WuGjwWuHmRjsWJJ33/IeSxbNvADLQ8Tu+5S7veDV4CZdevLEfT09ED/IenhTX39w5yc3sy2ZTsXrrdowxa2Lp/KXb4yPZ0m4czyw2y971DadI+nNrjHkxhFj6cy0mnDMdn61mE1YPHUB4sn0at48lJZ6UjaF3gv8J2IWD3Mtiye+mDxJIYpnkpKR9Jy4ApgK/CGDuXOkLRa0urfbu3vj2rx1AeLJzEs8VROOpIWA1cC+wMnRMR97cpGxIURcUREHDG10/K+27Z46oPFkxiGeColHUlTwKXAc4CXRMTPitTvZWS+GYunPlg8iUGLpzLSyebifAF4AXBSRNzQSxyLZw6LpzsWT2KQ4qmMdIBPAKcAHwGmJR3ZsOxXJJDFM4fF0x2LJzEo8VRJOi/O1v8DuL5p+e9Fg1k8c1g83bF4EoMQT2WkExErI0JtlnN7iWnxzGHxdMfiSfQrnspIZ1hYPHNYPN2xeBL9iKf20gGLpxGLpzsWT6JX8Vg6GRbPHBZPdyyeRC/isXQasHjmsHi6Y/EkiorH0mnC4pnD4umOxZMoIh5LpwUWzxwWT3csnmJYOm2weOaweLpj8eTH0umAxTOHxdMdiycftZFOTKinehbPHBZPdyye7tRGOkCh57g2YvHMYfF0x+LpTK2kAxbPILB4umPxtKd20gGLZxBYPN2xeFpTS+mAxTMILJ7uWDw7UlvpgMUzCCye7lg886m1dMDiGQQWT3csnjlqLx2weAaBxdMdiydh6WRYPP1j8XTH4rF05mHxmFFQd/FYOk1YPP3h3k4+6iweS6cFFk9/WDz5qKt46iOdgq/U4ukPiycfdRRPfaQDbF1a7OVaPP1h8eRjoYgnL7WSDlg8o8biyUedxFM76YDFM2osnnzURTyVkY6k/SSdL+l6SRslhaSVvcazeEaLxZOPOoinMtIBDgBeDawFvjeIgBbPaLF48rHQxVMl6VwbEXtHxEuArwwqqMUzWiyefCxk8VRGOhExM6zYFs9osXjysVDFUxnpDBuLZ7RYPPlYiOJZ0NKRdIak1ZJWb/lt9zfR4hktFk8+Fpp4FrR0IuLCiDgiIo6Y2infh8riGS3jLJ4DZ9Zw6tZbOHBmTdmpLCjxLGjp9IrFM1rGUTwHzqzhg1uu5rRtP+WDW662eAZIbaQTk8W+98riGS3jJp5DZu5nim1MAouY4ZCZ+8tOCVgY4qmNdAC2LCsoEotnpIyTeG6e2IstTLIVsZUJbp7Yq+yUtlN18SwaSJQRIelV2Y+HZ+sXS1oDrImIa/LE2LJsgqnp/Fffty6dYNHGAuWXT7Fow5bc5WfZtmxnJqc3F643VJYtG/0U+V13Ke9D1cCtE3vyzqnjOGTmfm6e2ItbJ/YsO6V5zKxbX46kp6f7/odUKemw46TAC7L1NcCxeYMsVPHEiiWD/3K0motn3GTTSFXFU6nTq4hQm+XYorEW6qnWwE+zoPanWuNMFU+1KiWdQWPxFMDiGVuqJp5aSwcsnkJYPGNLlcRTe+mAxVMIi2dsGYdxsDxYOhkWTwEsnrGlFPFMTxfq8Vg6DVg8BbB4xpZx7/FYOk1YPAWweMaWcRZPbaQTBV5pUfEUxeLpD4snH+MqntpIB2Drkvz3XxURT9HeDlg8/WLx5GMcxVMr6YDFAxZP3Rg38dROOmDxgMVTN8ZJPLWUDlg8YPHUjXERT22lAxYPWDx1YxzEU2vpgMUDFk/dKFs8tZcOWDxg8dSNMsVj6WRYPBZP3ShLPJZOAxaPxVM3yhBPbaSTd0ayxWPx1I1Ri6c20gHYsjSfUCwei6dujFI8tZIOWDxFsHjqxajEUzvpgMVTBIunXoxCPIWOfklPlHSVpP+QdJ6kxQ37/nXw6Q0Piyc/Fk+9GLZ4ih75FwCXAacAuwNXSVqR7evtE1EiFk9+LJ56MUzxFD3q946IT0TETRFxOvANknh2BWLg2Y0Aiyc/Fk+9GJZ4ih7x847aiPgA8GXgKmBFyxoDQtLjJH1V0jpJ6yVdJunxg4ht8eTH4qkXwxBP0aP9Dkm/37ghIj4CXAw8aWBZNSFpKXA18DTgNOB1wJOB70oayBFr8eTH4qkXgxZP1yNdUuOn7A+Aa5vLRMR5wOMGmFczbwL2B06OiK9FxBXAicATgDPzBIgcrrB48mPx1ItBiifPUX6npD+XtDwiNkdEyy/bjohfDSyrHTkRuCEi7mxo7y7gB8BJeYNsW9y9jMWTH4unXgxKPHmO8MuB9wC/lPRhScPs0bTjYODnLbbfAhxUJJDFMx+LJ2Hx5GMQ4ul6dEfE2cB+wLuAFwP/LukSSc/uu/X87A6sbbH9QWC3dpUknSFptaTV2zbNfRmYxTMfiydh8eSjX/HkOrIjYlNE/ENEPIMkniXAdZK+J+kVfWWQn1aX5Dt+4iPiwog4IiKOmFwy/yC2eOZj8SQsnuGTZyB5H0kHS3qepJcD+5KuJF0CHAl8dcg5Qurl7N5i+2607gHlwuKZj8WTsHi6009vZ1GOMvcBM8AG4GFgHbA+W1+WrYfNLaRxnWYOAn7RT+Bti2Hykc5ltiwVUxu7z33cukQs2pRvjuSWZRNMTc/kKrt16QSLNuYru73O8ikWbdhSqA4k8UxOt7xWkItYsQQ9vKnn+i1ZtqzQd2UPgolddyn9sZ7jzsy69T0JOs+/0R8Dm4CLgKMj4ukRcVREvDgiTo2IMwq3WpwrgSMl7T+7QdJK4LnZvr5wj2c+7vEk3OPpTi9izjOQfDjpkvUBwO2SviTpOcXT64t/Au4GrpB0kqQTgSuAXwL/OIgGLJ75WDwJi6c7RcWTdyD5uxHxMuCZpFOsayR9X9LJTZMHh0JETAPHAXcA/wx8AbgLOC4iNuSKkeOVWjzzsXgSFk93ioin0BEcEbeSZgcfCtxJuu/qjiIxeiUi7o2I/xYRu0TEiog4OSLuLhJj69LuZSye+Vg8CYtncOS5enWTpNsl/aekDcBW4Fbg9aSB6P07BhgzLJ7iWDwJi2cw5Ll69SPSZelOS6XYuhQWbexcxle15uOrWglf1eqfrtKJiDeNIpFRY/FYPL1i8fRHLZ+RPItPtYrjU62ET7V6p9bSAYunFyyehMXTG7WXDlg8vWDxJCye4tRHOl1eqcVTHIsnYfEUoz7Sobs0LJ7iWDwJiyc/tZIOWDwd61g8fWHx5KN20gGLp2Mdi6cvLJ7u1FI6YPF0rGPx9IXF05naSgcsno51LJ6+mNh1F8unDbWWDlg8HetYPH1j8exI7aUDFk/HOhZP31g886mNdLo9T8fi6VDH4ukbi2eO2kgHYNvSzjdGWjwd6lg8fWPxJGolHbB4tse1eAYfMwcWTw2lAxbP9rgWz+Bj5qDu4qmldMDi2R7X4hl8zBzUWTy1lQ5YPNvjWjyDj5mDuoqn1tIBi2d7XItn8DFzUEfx1F46YPFsj2vxDD5mDuomnvpIR52fT2zxZHEtnsHHzEGdxFMZ6Uj6S0lfl/RrSSHp3KIxZhZ3EYvFk+JaPIOPmYO6iKcy0iF9yd9ewNf6CWLx5MPisXiGRZWkc3BE/A7wZ/0GsnjyYfFYPMOgMtKJiGJf7tQFiycfFo/FM2gqI51hYPHkw+KxeAZJraUDFk9eLB6LZ1CUIh1Jx2dXoLotq/ps5wxJqyWt3rah/VfPWjz5sHgsnkHQ9bvMh8R1wIE5ynX5tvHORMSFwIUAOz9xv44TdWYWzzDxSPsP1balM0xu7LC/y/ee+7vT/d3p/bCQvj+9FOlExEbgtpE3vHgbPDLZdrfFY/HkwuLpi/qN6Sze1nG3T7Xy4VMtn2r1SmWkI+kISa8CXpltOkjSq7Ilx0e5AYunfUyLJz8WT09URjrAWcBXgC9lv5+S/f4V0kzlYlg87WNaPPmxeApTGelExOkRoTbL3T0FtXjax7R48mPxFKIy0hkaFk/7mBZPfiye3NRGOprocBXH4mkf0+LJj8WTi9pIB2CnpR0u2Vo87WNaPPmxeLpSK+mAxTOvPYvH4imB2kkHLJ557Vk8Fs+IqaV0wOKZ157FY/GMkNpKByyeee1ZPBbPiKi1dMDimdeexWPxjIDaSGeiwyVzi6ehPYuHWLFk8PKxeLZTG+kALFn827b7LJ6G9iweYAi9HosHqJl0wOKxeIph8Qye2kkHLB6LpxgWz2CppXTA4rF4imHxDDCHshMoE4vH4imCxTOg9kttfQyweCyeIlg8A2i7tJZHzORE+w+5xWPxFMHi6bPdUlotiV2XtH/iucVj8RTB4umjzZG3WDIWT3ssnmJYPD22N9LWxgSLpz0WTzEsnh7aGllLY4bF0x6LpxgWT8F2RtLKmGLxtMfiKYbFU6CNobcwJkyq9YfZ4mmPxVMMiydn/KFGHzN227n118taPO2xeIph8eSIPbTIA0TSUyR9XNLNkjZI+rWkKyUdWjSWxVMsBlg8RbF4usQdStTB80Lg+cDngJcDfwrsCdwo6fCiwSyeYjHA4imKxdMh5sAjDodLgEMj4qMR8d2IuBx4EbAJeGsvAS2eYjHA4imKxdMm3kCjDYmIeCAiomnbOuAOYN9e41o8xWKAxVMUi6dFrIFFGjGSdgeeDtzaTxyLp1gMsHiKYvE0xRlIlHI4HxDwsTyFpzrc8GnxFIsBFk9RLJ6GGAPIozCSjpcUOZZVbeq/C3gNcFZE3NmhnTMkrZa0etPaR9hz8Ya2OVk8xWKAxVMUiyehpqGSkSBpKfD4HEU3RsS9TXXfDHwSeHdEvD9vm3setEec9PmXAbDmkeVty63d3PrAWLep/adw0yM7td33240dDvJHJtvvAyYe6fyhmtzYZX97XwKwaGPn/XliAExtzHcMLdqU/1ibmu4s1nlxN+Yvu73Ohg7/FDowOb25p3qN6OHW/+B6Znp6sPFyMrNu/bzfv7X54psi4ohu9Urp6UTExoi4LcfSLJzXARcAHy0inGbc40m4x1Mc93jm6LXHU5kxHUmvAD4LfCoizu43nsWTsHiKY/HM0Yt4KiEdSb8HfBG4GbhI0pENy2G9xrV4EhZPcSyeOYqKpxLSAY4DdgYOA34AXN+wXJ4nwCK1/mBbPAmLpzgWzxxFxFMJ6UTEuRGhNsvKvHH22Km1YCyehMVTHIunOJWQziCxeDrutnh6wOIpRu2kAxaPxdOhjsUzdGopHbB4LJ4OdSyeoVJb6YDFY/F0qGPxDI3aSGeqzdUri6fjbounByyeztRGOgCP3emhltstno67F5R4isrH4hk8tZIOWDxt99dEPFC812PxDJbaSQcsnrb7LZ725S2egVFL6YDF03a/xdO+vMUzEGorHbB42u63eNqXt3j6pjbSaXf1yuJps9/iaV/e4umL2kgH4DGL1rXcbvG02W/xtC9v8fRMraQDFk8zFk9DbItnJNROOmDxNGPxNMS2eIZOLaUDFk8zFk9DbItnqNRWOmDxNGPxNMS2eIZGbaTT7uqVxTMfi6chtsUzFGojHYB9Fq1tud3imY/F0xDb4hk4tZIOWDyzWDz5sXgGS+2kAxbPLBZPfiyewVFL6YDFM4vFkx+LZzDUVjpg8cxi8eTH4umf2khnitYfHIsnYfHkx+Lpj9pIB+Axiza23G7xJCye/Fg8vVMJ6UhaIenLku6UNC3pIUk3Snpt0VgWj8XTNqbFk58+xFMJ6QA7AVuBDwAnAq8BbgP+WdJfFA1m8Vg8bWNaPPnpUTyVkE5E/CYiXhMRn46IqyLif0fEacANwBt7iWnxWDxtY1o8+elBPJWQTgd+A3T4ZM0xpR0POovH4mkb0+LJT0HxVEo6SiyS9GhJZwAnAB/LW3+fyR3/0BaPxdM2psWTnwLiWTTYlofOW4Dzs5+3AG+NiM+3K5yJ6Yzs1w3L97n39iHnV4Q9gAfKTmKM8fvTnXF7j56Qp5AiYtiJ7NiodDzw7RxFr4mIYxvq7Ul6YXuQBpTPBP40Iv5xGHkOE0mrI+KIsvMYV/z+dKeq71FZPZ3rgANzlJt37hMRa4A12a//V9JS4COSPhMRucZ2jDHlUop0ImIj6ZJ3v6wGTgP2Bu4bQDxjzJCp1EByC44BNgD3l51ID1xYdgJjjt+f7lTyPSplTKcoks4EjgS+Q+rRPBp4NXAq8M6I+FCJ6RljClCVq1c/A04CPgLsThqxvxV4WUR8s8zEjDHFqERPxxizcKj6mE7lkfQUSR+XdLOkDZJ+LelKSYeWnduokfQ4SV+VtE7SekmXSXp82XmNC5JeJelSSfdI2iTpdkkfkLSi7NyK4J5OyUg6izSB8XPAj4BHAW8HDgOeGxE3lZjeyMimP/wU2Ay8Gwjgb4ClwCERMV1iemOBpBuAe4ErSGObhwHnkq4EHxURnaeHjwmWTslI2gP4TTT8ISTtCtwNfD0iXl9WbqNE0luB84CnRsSd2bYnAv8GvD0iziszv3FA0p7ZXLXGba8n/cN6QURcXU5mxfDpVclExAPRZP6IWAfcAexbTlalcCJww6xwACLiLuAHpIsItadZOBk/zNaVOVYsnTFE0u7A00lX6OrCwcDPW2y/BThoxLlUiWOydWWOFUtnPDkfEAXuoF8A7A60urX/QWC3EedSCSTtC7wX+E5ErC47n7xYOgNG0vGSIseyqk39d5GejHhW46lGTWg1wJj/WRQ1QtJy0oDyVuANJadTiKpMDqwSPd3MCiDpzcDfAu+OiM8MOrExZy2pt9PMbrTuAdUWSYuBK4H9gWMiolL3HVo6A6bXm1klvQ64APhoRLx/4ImNP7eQxnWaOQj4xYhzGVskTQGXAs8Bjo+In5WcUmF8ejUGSHoF8FngUxFxdtn5lMSVwJGS9p/dIGkl8NxsX+2RNAF8AXgBcFJE3FBySj3heTolI+n3gG+R/pufBfO+FXBzRPy4lMRGjKRlpMmBm5ibHPg+YAVpcmD757vWBEmfBN4MvB/4RtPu+6pymmXplIykc4H3tNl9T0SsHF025ZLd8vB3wO+TBpCvAt4WEXeXmde4IOlu2j8S9K8j4tzRZdM7lo4xZqR4TMcYM1IsHWPMSLF0jDEjxdIxxowUS8cYM1IsHWPMSLF0jDEjxdIxpSDpU5J+1LTtpuwO/P0atn1E0l2SJkefpRkGlo4pi7WkWxwAkHQc8FTS7Q+7ZduWAm8E/j4itpWRpBk8lo4pi3nSAf4K+AzpG1tnH9r1emAS+PRoUzPDxNIxZbEW2AVA0oHAC0lPSlzPnHTOIt15vz4rd072tSszkk5xERR/AAABAElEQVQuIWczACwdUxZrgSXZWM1fAVdExH8A64DdJB0PPA34+4Y6VwEvAa4ddbJmcPghXqYsZp8GeADwWuD52e+zPZ1XApdFxD2zFSLiRgDJTzCtMpaOKYtZ6ZwD/Cgirs9+XwccDryU9AAvs8CwdExZzErnj4BTG7avB/4AuLGqT8YznfGYjimLWencC3ytYfs60hWr2n+j50LFPR1TChHxAC2+XiYi3gS8afQZmVHhJweayiDp3aRnBO8JPAw8AhwREf9VamKmEJaOMWakeEzHGDNSLB1jzEixdIwxI8XSMcaMFEvHGDNSLB1jzEixdIwxI8XSMcaMFEvHGDNS/j+1SjbaXoSKaQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "make_plot(f2, λ=0.1)\n",
    "plt.title(\"L2 regularization\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With L2, the solution is unique. Why? This isn't a general proof but, try minimizing $w_1^2+w_2^2$ subject to $w_1+w_2=w_\\text{total}$. You'll get $w_1=w_2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "----------------\n",
    "\n",
    "This is the dividng line between lecture 17 and lecture 18.\n",
    "\n",
    "----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intercept and the mean value\n",
    "\n",
    "One question is why we bother fitting an intercept: can't we just center the target values?\n",
    "\n",
    "The answer is that the intercept isn't always going to be equal to the mean of the target values! For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEBCAYAAACQbKXWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAD41JREFUeJzt3W1s3eV5x/HvlZqMtUVTCO5UQRNj9YkHDU2xWktIhQJFRRNF1aAvQMAqhYxWSK0qtK0MaawT7ZuWtkIBLUVt0ZTuoWMrYWqllRIyNGpUW9V4KBSiNGZIvDDBYqJBTVxfe+GTzRg755zE9zn/c+7vRzo69nm67/zl/Hyf67rP35GZSJLqsqHfE5Ak9Z7hL0kVMvwlqUKGvyRVyPCXpAoZ/pJUIcNfkipk+EtShQx/SarQSL8nsJYzzjgjx8bG+j0NSRooMzMzr2TmaLvHNTb8x8bGmJ6e7vc0JGmgRMRsJ4+z7CNJFTL8JalChr8kVahY+EfE1RHxQETMRsQbEfHLiPhKRJxWakxJUmdKrvxvBX4L3AZ8HLgX+Azw44jwHYck9VHJ3T5XZubcsu/3RcSrwP3AxcAjBceWJB1HsRX4iuA/5met6zNLjStJg2pmdp6de/czMztffKxe7/O/qHX9bI/HlaRGm5md57r7pjiysMjGkQ3s3j7Jtq2bio3Xs9p7RJwJfAl4ODNX/fRWROyIiOmImJ6bW+2NgyQNp6kDhziysMhiwtGFRaYOHCo6Xk/CPyLeCTwILACfXutxmbkrMycyc2J0tO2nkyVpaEyOb2bjyAbeFnDKyAYmxzcXHa942SciTgX2AOPARZn5UukxJWnQbNu6id3bJ5k6cIjJ8c1FSz5QOPwj4hTgAeBDwGWZ+VTJ8SSpyWZm548b7tu2bioe+scUC//WXv7dwKXAH2XmVKmxJKnpet3QbadkzX8ncA3wVeDXETG57HJWwXElqXF63dBtp2T4X9G6/kvgpysu2wuOK0mN0+uGbjvFyj6ZOVbqtSVp0PS6odtOY/+YiyQNm142dNvxBGuStE56eXqGk+XKX5LWQdN287Tjyl+S1kHTdvO0Y/hL0jpo2m6ediz7SNI6aNpunnYMf0nqUJNOz3CyDH9J6sCgNXTbseYvSR0YtIZuO4a/JHVg0Bq67Vj2kaQODFpDtx3DX5Jahqmh247hL0kMX0O3HWv+ksTwNXTbMfwlieFr6LZj2UeSGL6GbjuGv6Rq1NTQbcfwl1SF2hq67Vjzl1SF2hq67Rj+kqpQW0O3Hcs+kqpQW0O3HcNf0tCwods5w1/SULCh2x1r/pKGgg3d7hj+koaCDd3uWPaRNBRs6HbH8Jc0NGzods6yj6SBMTM7z869+5mZne/3VAaeK39JA8HdPOvLlb+kgeBunvVl+EsaCO7mWV+WfSQNBHfzrC/DX1JjeHqG3jH8JTWCDd3esuYvqRFs6PaW4S+pEWzo9laxsk9EnAX8OTABXAD8LnB2Zh4sNaakwWVDt7dK1vzfC3wKmAEeAy4vOJakAWBDtzlKhv9/ZObvA0TEdgx/qWo2dJulWM0/MxdLvbakwWNDt1ls+ErqCRu6zeI+f0k9YUO3WRoV/hGxA9gBsGXLlj7PRlK3bOgOjkaFf2buAnYBTExMZJ+nI6kLNnQHizV/SevChu5gMfwlrQsbuoOlaNknIq5ufbmtdX1FRMwBc5m5r+TYknrLhu5gKV3z//6K7+9pXe8DLi48tqR1ZkN3eBQN/8yMkq8vqXds6A4Xa/6SOmJDd7gY/pI6YkN3uDRqn7+k5rKhO1wMf0kds6E7PCz7SPo/M7Pz7Ny7n5nZ+X5PRYW58pcEuJunNq78JQHu5qmN4S8JcDdPbSz7SALczVMbw1+qiKdn0DGGv1QJG7pazpq/VAkbulrO8JcqYUNXy1n2kSphQ1fLGf7SELGhq04Z/tKQsKGrbljzl4aEDV11w/CXhoQNXXXDso80JGzoqhuGvzRAbOhqvRj+0oCwoav1ZM1fGhA2dLWeDH9pQNjQ1Xqy7CMNCBu6Wk+Gv9QgNnTVK4a/1BA2dNVL1vylhrChq14y/KWGsKGrXrLsIzWEDV31kuEvNYgNXfWKZR+ph2Zm59m5dz8zs/P9nooq58pf6hF386hJXPlLPeJuHjWJ4S/1iLt51CSWfaQecTePmsTwl9aRp2fQoCga/hHxHuDrwMeAAB4GPp+ZL5YcV+oHG7oaJMVq/hHxduAR4IPAjcD1wPuAvRHxjlLjSv1iQ1eDpOTK/yZgHPhAZu4HiIgngReAPwXuKji21HPHGrpHFxZt6KrxIjPLvHDET4BTM/PCFbfvA8jMi473/ImJiZyeni4yN6mUdjV/qbSImMnMiXaPK7nyPw94cJXbnwGuKTiuVMz3nniRHz39Mlec/26u/fCWt9xvQ1eDomT4nw6s9hn2VwH/d2jgfO+JF7ntX58C4LEXXgFY9ReANAhKf8hrtZpSrPXgiNgREdMRMT03N1dwWlL3fvT0y8f9XhokJcN/nqXV/0qbWP0dAZm5KzMnMnNidHS04NSk7l1x/ruP+700SEqWfZ5hqe6/0rnALwqOKxVxrMRzvJq/NChKhv8e4KsRMZ6ZBwAiYgy4EPiLguNKxVz74S2GvoZCybLPt4CDwIMRcVVEfIKl3T//DfxtwXElSW0UC//M/DVwCfA88HfAbuBXwCWZ+XqpcSVJ7RU9t0/rHD5/XHIMSVL3PJ+/JFXI8JekChn+klQhw1+SKmT4S1KFDH9JqpDhL0kVMvwlqUKGvyRVyPCXpAoZ/pJUIcNfkipk+EtShQx/SaqQ4S9JFTL8JalChr8kVcjwl6QKGf6SVCHDX5IqZPhLUoUMf0mqkOEvSRUy/CWpQoa/JFXI8JekChn+klQhw1+SKmT4S1KFDH9JqpDhL0kVMvwlqUKGvyRVyPCXpAoZ/pJUIcNfkipk+EtShYqFf0R8ISIeioiXIyIj4o5SY0mSulNy5X8T8C7gBwXHkCSdgJGCr31eZi5GxAhwc8FxJEldKrbyz8zFUq+t9TMzO8/OvfuZmZ3v91Qk9VDJlb8abmZ2nuvum+LIwiIbRzawe/sk27Zu6ve0JPVAo3b7RMSOiJiOiOm5ubl+T2foTR04xJGFRRYTji4sMnXgUL+nJKlHOgr/iListWOn3eXRk5lMZu7KzInMnBgdHT2Zl1IHJsc3s3FkA28LOGVkA5Pjm/s9JUk90mnZ53HgnA4ed/gk5qIe27Z1E7u3TzJ14BCT45st+UgV6Sj8M/Mw8FzhuagPtm3dZOhLFWpUzV+S1BvFdvtExAQwxv//gjk3Iq5uff3D1rsJSVIflNzqeQtw47Lvr2ldAM4GDhYcW5J0HCU/5PUnmRlrXA6WGleS1J41f0mqkOEvSRUy/CWpQoa/JFXI8JekChn+klQhw1+SKmT4S1KFDH9JqpDhL0kVMvwlqUKGvyRVyPCXpAoZ/pJUIcNfkipk+EtShQx/SaqQ4S9JFTL8JalChr8kVcjwl6QKGf6SVCHDX5IqZPhLUoUMf0mqkOEvSRUy/CWpQoa/JFXI8JekChn+klQhw1+SKmT4S1KFDH9JqpDhL0kVMvwlqUKGvyRVqEj4R8T7I+KbEfFkRLweES9HxJ6IuKDEeJKk7pRa+V8OfBS4H7gS+CwwCjwREdsKjSlJ6tBIodf9B2BnZuaxGyLiEeAg8DnghkLjSpI6UCT8M/OVVW57LSKeB84sMaYkqXM9a/hGxOnA+cCzJceZmZ1n5979zMzOlxxGkgZaqbLPau4GAvhGqQFmZue57r4pjiwssnFkA7u3T7Jt66ZSw0nSwOpo5R8Rl0VEdnB5dI3nfxG4FrglM/cfZ5wdETEdEdNzc3Nd/2OmDhziyMIiiwlHFxaZOnCo69eQpBp0uvJ/HDing8cdXnlDRNwMfBm4PTO/fbwnZ+YuYBfAxMREHu+xq5kc38zGkQ0cXVjklJENTI5v7vYlJKkKHYV/Zh4Gnuv2xSPieuAe4GuZeWe3z+/Wtq2b2L19kqkDh5gc32zJR5LWUKzmHxGfBL4D3JeZt5YaZ6VtWzcZ+pLURpHwj4iPAH8PPAl8NyIml939m8z8eYlxJUmdKbXyvwT4HeAPgf9ccd8sMFZoXElSB4rs88/MOzIz1riMlRhTktQ5z+opSRUy/CWpQoa/JFUolp14s1EiYo6l5vCJOAN4y8nltCaPV3c8Xt3xeHXnZI/X1swcbfegxob/yYiI6cyc6Pc8BoXHqzser+54vLrTq+Nl2UeSKmT4S1KFhjX8d/V7AgPG49Udj1d3PF7d6cnxGsqavyTp+IZ15S9JOo6hDv+IeH9EfDMinoyI1yPi5YjYExEX9HtuTRURX4iIh1rHKiPijn7PqQki4j0R8c8R8VpE/E9E/EtEbOn3vJoqIs6KiLsj4qcRcbj1szTW73k1UURcHREPRMRsRLwREb+MiK9ExGklxx3q8AcuBz4K3A9cCXwWGAWeiIht/ZxYg90EvAv4Qb8n0hQR8XbgEeCDwI3A9cD7gL0R8Y5+zq3B3gt8CpgHHuvzXJruVuC3wG3Ax4F7gc8AP46IYhk91DX/iDgDOJTL/pER8XvAQeChzLyhX3NrqojYkJmLETECHAX+OjPv6PO0+ioiPgfcBXzg2J8hjYizgReAP8vMu/o5vyY69nPU+no78C3g7Mw82NeJNVBEjGbm3IrbbmBp0XppZj5SYtyhXvln5iu54rdbZr4GPA+c2Z9ZNdux/7B6k08AU8v//nRm/oql05Vf1bdZNZg/R51bGfwtP2tdF8upoQ7/1UTE6cD5wLP9nosGxnnA06vc/gxwbo/nojpc1LoullPVhT9wNxDAN/o9EQ2M01mqXa/0KuDfDNW6iogzgS8BD2fmdKlxBir8I+Ky1q6BdpdH13j+F4FrgVuWv4UfVid7vPQmqzXHouez0FCLiHcCDwILwKdLjlXsD7gX8jhwTgePO7zyhoi4GfgycHtmfnu9J9ZQJ3y89CbzLK3+V9rE6u8IpK5FxKnAHmAcuCgzXyo53kCFf2YeBp7r9nkRcT1wD/C1zLxz3SfWUCd6vPQWz7BU91/pXOAXPZ6LhlBEnAI8AHwIuCwznyo95kCVfU5ERHwS+A5wX2be2u/5aCDtASYjYvzYDa0PLF3Yuk86Ya29/LuBS4GrMnOqJ+MO+T7/jwD/ztLq7BZg+faz32Tmz/sysQaLiAlgjKWFwT8C3wf+qXX3D1vvJqrS+iDXfwFvALezVP//G+A04A8y8/U+Tq+xIuLq1peXAjez9CHLOWAuM/f1bWINExH3snR87gT+bcXdL5Uq/wx7+N8B/NUad89m5ljvZjMYIuK7LH2KdTXVfkindSqHrwMfY6nR+xPg87Uej05ExFrhsi8zL+7lXJosIg4CW9e4u9iHLIc6/CVJqxv6mr8k6a0Mf0mqkOEvSRUy/CWpQoa/JFXI8JekChn+klQhw1+SKmT4S1KF/hf4MtAAJvOU8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.append(np.linspace(-2,0,3), np.linspace(0,2,20))\n",
    "y = x\n",
    "plt.plot(x,y,'.');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By visual inspection, the intercept here should be zero. Indeed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(x[:,None],y)\n",
    "lr.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And yet the mean is nonzero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7391304347826086"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling features\n",
    "\n",
    "We've seen cases where scaling the features is important to get good performance. We can test this out by comparing the fits with and without scaling.\n",
    "\n",
    "#### Without regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 100\n",
    "d = 10\n",
    "X = np.random.randn(n,d) \n",
    "X[:,0] = X[:,0]*1000 # make the first feature on a huge scale\n",
    "y = np.sum(X,axis=1) + np.random.randn(n)\n",
    "Xtest = np.random.randn(1,d)\n",
    "\n",
    "lr = LinearRegression(normalize=False)\n",
    "lr.fit(X,y)\n",
    "\n",
    "lr_scale = LinearRegression(normalize=True)\n",
    "lr_scale.fit(X,y)\n",
    "\n",
    "np.allclose(lr.predict(Xtest), lr_scale.predict(Xtest))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L2 regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge = Ridge(normalize=False)\n",
    "ridge.fit(X,y)\n",
    "\n",
    "ridge_scale = Ridge(normalize=True)\n",
    "ridge_scale.fit(X,y)\n",
    "\n",
    "np.allclose(ridge.predict(Xtest), ridge_scale.predict(Xtest))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "IPython.notebook.set_autosave_interval(0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autosave disabled\n"
     ]
    }
   ],
   "source": [
    "%autosave 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L1 regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso = Lasso(normalize=False)\n",
    "lasso.fit(X,y)\n",
    "\n",
    "lasso_scale = Lasso(normalize=True)\n",
    "lasso_scale.fit(X,y)\n",
    "\n",
    "np.allclose(lasso.predict(Xtest), lasso_scale.predict(Xtest))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Intuition: consider the simple case of $n=2,d=2$ with no intercept for simplicity. The data set consists of two points:\n",
    "\n",
    "$$X=\\begin{bmatrix} 1 & 0 \\\\ 0 & 1000 \\end{bmatrix} \\quad y=\\begin{bmatrix} 1 \\\\ 1 \\end{bmatrix}$$\n",
    "\n",
    "The optimal weights are $$w=\\begin{bmatrix} 1 \\\\ 0.001 \\end{bmatrix}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.   , 0.001])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.array([[1, 0],[0, 1000]])\n",
    "y = np.array([1,1])\n",
    "lr = LinearRegression(fit_intercept=False)\n",
    "lr.fit(X,y)\n",
    "lr.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5  , 0.001])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge = Ridge(fit_intercept=False)\n",
    "ridge.fit(X,y)\n",
    "ridge.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The big coefficient got pushed down, but the small one pretty much got left alone. Therefore the _ratio of the coefficients_ changed. If we scaled the second feature by 1/1000 then we'd get coefficients that are the same. We can do the same with L1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso = Lasso(fit_intercept=False,alpha=0.1) # if alpha=1 then we snap one of the weights to zero\n",
    "lasso.fit(X,y)\n",
    "lasso.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusion:\n",
    "\n",
    "- Un-regularized linear regression: not affected by scaling\n",
    "- L1 or L2-regularized linear regression: both affected by scaling (and it's usually a good idea)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: a common mistake is to scale test data with the mean/std of the test set. Need to use the same transformation as you used for the training data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple choice question\n",
    "\n",
    "If I'm **not** using regularization, what happens if I multiply one of the features by $2$ and re-compute $w$? (There may be multiple that are true.)\n",
    "\n",
    "1. The corresponding weight is multiplied by $1/2$.\n",
    "2. The corresponding weight is multiplied by $2$.\n",
    "3. The predictions on the training data are changed.\n",
    "4. The predictions on the training data are unchanged. \n",
    "5. The predictions on any test data (with the same transformation applied) are unchanged. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling regression targets (bonus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This one is a little more insidious than the feature scaling. It seems like it would never matter. But, let's compare the predictions with/without scaling the targets by a factor of 10.\n",
    "\n",
    "#### No regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions are the same with/without scaling?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 100\n",
    "d = 10\n",
    "X = np.random.randn(n,d) \n",
    "y = np.sum(X,axis=1) + np.random.randn(n)\n",
    "Xtest = np.random.randn(1,d)\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(X,y)\n",
    "\n",
    "lr_scale = LinearRegression()\n",
    "scale_factor = 10\n",
    "lr_scale.fit(X,y*scale_factor)\n",
    "\n",
    "print(\"Predictions are the same with/without scaling?\",)\n",
    "np.allclose(lr.predict(Xtest), lr_scale.predict(Xtest)/scale_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L2 regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions are the same with/without scaling?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge = Ridge()\n",
    "ridge.fit(X,y)\n",
    "\n",
    "scale_factor = 10\n",
    "ridge_scale = Ridge()\n",
    "ridge_scale.fit(X,y*scale_factor)\n",
    "\n",
    "print(\"Predictions are the same with/without scaling?\",)\n",
    "np.allclose(ridge.predict(Xtest), ridge_scale.predict(Xtest)/scale_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L1 regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions are the same with/without scaling?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso = Lasso()\n",
    "lasso.fit(X,y)\n",
    "\n",
    "scale_factor = 10\n",
    "lasso_scale = Lasso()\n",
    "lasso_scale.fit(X,y*scale_factor)\n",
    "\n",
    "print(\"Predictions are the same with/without scaling?\",)\n",
    "np.allclose(lasso.predict(Xtest), lasso_scale.predict(Xtest)/scale_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(To read over after class...) What's going on here?! With feature scaling it mattered for Ridge and Lasso, but here only Lasso is affected. Why? \n",
    "\n",
    "- If I scale the values of $y$ then I can account for this by scaling all the weights by the same amount\n",
    "  - (it's cleaner to think about this without an intercept)\n",
    "- So, for no regularization this is fine, predictions are also scaled\n",
    "- For Ridge, if you scale the weights, _both terms in the loss_ are scaled by this factor squared\n",
    "  - $f(w)=||Xw-y||^2_2 + \\lambda ||w||_2^2$.\n",
    "  - $f(cw)=c^2||Xw-y||^2_2 + c^2 \\lambda ||w||_2^2 = c^2 f(w)$.\n",
    "  - Thus it's just an overall scaling of the loss, which has no affect\n",
    "- For Lasso, if you scale the weights, the first term is affected more than the second\n",
    "  - $f(w)=||Xw-y||^2_2 + \\lambda ||w||_1$.\n",
    "  - $f(cw)=c^2||Xw-y||^2_2 + \\lambda c ||w||_1$.\n",
    "  - This, in essence, we've scaled $\\lambda$ by $1/c$. Let's test this:\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso = Lasso() # default is alpha=1\n",
    "lasso.fit(X,y)\n",
    "\n",
    "scale_factor = 10\n",
    "lasso_scale = Lasso(alpha=1*scale_factor)\n",
    "lasso_scale.fit(X,y*scale_factor)\n",
    "\n",
    "np.allclose(lasso.predict(Xtest), lasso_scale.predict(Xtest)/scale_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The universe makes sense -- nice!\n",
    "\n",
    "If we were picking $\\lambda$ (a.k.a. `alpha`) with cross-validation then this should not end up mattering, so long as we did any scaling _before_ we picked $\\lambda$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collinearity and conditioning (bonus)\n",
    "\n",
    "#### The \"dummy variable trap\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "      <th>embark_town</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>Cherbourg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>Southampton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>54.0</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>Southampton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>Southampton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>58.0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>Southampton</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     age     fare  embark_town\n",
       "1   38.0  71.2833    Cherbourg\n",
       "3   35.0  53.1000  Southampton\n",
       "6   54.0  51.8625  Southampton\n",
       "10   4.0  16.7000  Southampton\n",
       "11  58.0  26.5500  Southampton"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "titanic = sns.load_dataset(\"titanic\").dropna()\n",
    "titanic_small = titanic[[\"age\",\"fare\",\"embark_town\"]]\n",
    "titanic_small.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We talked about converting categorical to numerical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "      <th>embark_town_Cherbourg</th>\n",
       "      <th>embark_town_Queenstown</th>\n",
       "      <th>embark_town_Southampton</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>54.0</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>58.0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     age     fare  embark_town_Cherbourg  embark_town_Queenstown  \\\n",
       "1   38.0  71.2833                      1                       0   \n",
       "3   35.0  53.1000                      0                       0   \n",
       "6   54.0  51.8625                      0                       0   \n",
       "10   4.0  16.7000                      0                       0   \n",
       "11  58.0  26.5500                      0                       0   \n",
       "\n",
       "    embark_town_Southampton  \n",
       "1                         0  \n",
       "3                         1  \n",
       "6                         1  \n",
       "10                        1  \n",
       "11                        1  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic_small_dummy = pd.get_dummies(titanic_small)\n",
    "titanic_small_dummy.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we're predicting age from the fare and town."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = titanic_small_dummy.values.astype('float')\n",
    "y = X[:,0]\n",
    "X = X[:,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0244833 ,  1.05209929,  1.76389238, -2.81599166])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X, y)\n",
    "old_coef = lr.coef_\n",
    "old_coef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25056447.18721889"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xnew = X.copy()\n",
    "Xnew[0,-1] += 1e-8 # tiny perturbation\n",
    "lr.fit(Xnew, y)\n",
    "new_coef = lr.coef_\n",
    "np.abs(new_coef[-1] - old_coef[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, what just happened??? We changed a value by $10^{-8}$ and the coefficient changed by a giant amount?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But wait, what it's actually doing is fitting an intercept, meaning a column of ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xint = np.append(X, np.ones((X.shape[0],1)),axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 71.2833   1.       0.       0.       1.    ]\n",
      " [ 53.1      0.       0.       1.       1.    ]\n",
      " [ 51.8625   0.       0.       1.       1.    ]\n",
      " [ 16.7      0.       0.       1.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 13.       0.       0.       1.       1.    ]\n",
      " [ 35.5      0.       0.       1.       1.    ]\n",
      " [263.       0.       0.       1.       1.    ]\n",
      " [ 76.7292   1.       0.       0.       1.    ]\n",
      " [ 61.9792   1.       0.       0.       1.    ]\n",
      " [ 83.475    0.       0.       1.       1.    ]\n",
      " [ 10.5      0.       0.       1.       1.    ]\n",
      " [  7.65     0.       0.       1.       1.    ]\n",
      " [263.       0.       0.       1.       1.    ]\n",
      " [ 61.175    0.       0.       1.       1.    ]\n",
      " [ 34.6542   1.       0.       0.       1.    ]\n",
      " [ 63.3583   1.       0.       0.       1.    ]\n",
      " [ 77.2875   0.       0.       1.       1.    ]\n",
      " [ 52.       0.       0.       1.       1.    ]\n",
      " [247.5208   1.       0.       0.       1.    ]\n",
      " [ 13.       0.       0.       1.       1.    ]\n",
      " [ 77.2875   0.       0.       1.       1.    ]\n",
      " [ 26.2833   0.       0.       1.       1.    ]\n",
      " [ 53.1      0.       0.       1.       1.    ]\n",
      " [ 79.2      1.       0.       0.       1.    ]\n",
      " [ 26.       0.       0.       1.       1.    ]\n",
      " [ 66.6      0.       0.       1.       1.    ]\n",
      " [ 33.5      0.       0.       1.       1.    ]\n",
      " [ 30.6958   1.       0.       0.       1.    ]\n",
      " [ 28.7125   1.       0.       0.       1.    ]\n",
      " [ 39.       0.       0.       1.       1.    ]\n",
      " [ 26.       0.       0.       1.       1.    ]\n",
      " [ 27.7208   1.       0.       0.       1.    ]\n",
      " [146.5208   1.       0.       0.       1.    ]\n",
      " [ 10.4625   0.       0.       1.       1.    ]\n",
      " [ 31.       1.       0.       0.       1.    ]\n",
      " [113.275    1.       0.       0.       1.    ]\n",
      " [ 76.2917   1.       0.       0.       1.    ]\n",
      " [ 90.       0.       0.       1.       1.    ]\n",
      " [ 83.475    0.       0.       1.       1.    ]\n",
      " [ 90.       0.       1.       0.       1.    ]\n",
      " [ 52.5542   0.       0.       1.       1.    ]\n",
      " [ 10.4625   0.       0.       1.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 86.5      0.       0.       1.       1.    ]\n",
      " [ 79.65     0.       0.       1.       1.    ]\n",
      " [  0.       0.       0.       1.       1.    ]\n",
      " [153.4625   0.       0.       1.       1.    ]\n",
      " [135.6333   0.       0.       1.       1.    ]\n",
      " [ 29.7      1.       0.       0.       1.    ]\n",
      " [ 77.9583   0.       0.       1.       1.    ]\n",
      " [ 91.0792   1.       0.       0.       1.    ]\n",
      " [ 12.875    1.       0.       0.       1.    ]\n",
      " [151.55     0.       0.       1.       1.    ]\n",
      " [247.5208   1.       0.       0.       1.    ]\n",
      " [151.55     0.       0.       1.       1.    ]\n",
      " [108.9      1.       0.       0.       1.    ]\n",
      " [ 56.9292   1.       0.       0.       1.    ]\n",
      " [ 83.1583   1.       0.       0.       1.    ]\n",
      " [262.375    1.       0.       0.       1.    ]\n",
      " [164.8667   0.       0.       1.       1.    ]\n",
      " [134.5      1.       0.       0.       1.    ]\n",
      " [135.6333   1.       0.       0.       1.    ]\n",
      " [ 13.       0.       0.       1.       1.    ]\n",
      " [ 57.9792   1.       0.       0.       1.    ]\n",
      " [ 28.5      0.       0.       1.       1.    ]\n",
      " [153.4625   0.       0.       1.       1.    ]\n",
      " [ 66.6      0.       0.       1.       1.    ]\n",
      " [134.5      1.       0.       0.       1.    ]\n",
      " [ 26.       0.       0.       1.       1.    ]\n",
      " [263.       0.       0.       1.       1.    ]\n",
      " [ 13.       0.       0.       1.       1.    ]\n",
      " [ 55.       0.       0.       1.       1.    ]\n",
      " [ 75.25     1.       0.       0.       1.    ]\n",
      " [ 69.3      1.       0.       0.       1.    ]\n",
      " [ 55.4417   1.       0.       0.       1.    ]\n",
      " [211.5      1.       0.       0.       1.    ]\n",
      " [120.       0.       0.       1.       1.    ]\n",
      " [113.275    1.       0.       0.       1.    ]\n",
      " [ 16.7      0.       0.       1.       1.    ]\n",
      " [ 90.       0.       1.       0.       1.    ]\n",
      " [  8.05     0.       0.       1.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 55.9      0.       0.       1.       1.    ]\n",
      " [120.       0.       0.       1.       1.    ]\n",
      " [263.       0.       0.       1.       1.    ]\n",
      " [ 81.8583   0.       0.       1.       1.    ]\n",
      " [ 30.5      0.       0.       1.       1.    ]\n",
      " [ 27.75     1.       0.       0.       1.    ]\n",
      " [ 89.1042   1.       0.       0.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 38.5      0.       0.       1.       1.    ]\n",
      " [ 13.7917   1.       0.       0.       1.    ]\n",
      " [ 91.0792   1.       0.       0.       1.    ]\n",
      " [ 90.       0.       0.       1.       1.    ]\n",
      " [ 29.7      1.       0.       0.       1.    ]\n",
      " [ 30.5      0.       0.       1.       1.    ]\n",
      " [ 78.2667   1.       0.       0.       1.    ]\n",
      " [151.55     0.       0.       1.       1.    ]\n",
      " [ 86.5      0.       0.       1.       1.    ]\n",
      " [108.9      1.       0.       0.       1.    ]\n",
      " [ 26.2875   0.       0.       1.       1.    ]\n",
      " [ 34.0208   0.       0.       1.       1.    ]\n",
      " [ 10.5      0.       0.       1.       1.    ]\n",
      " [ 93.5      0.       0.       1.       1.    ]\n",
      " [ 57.9792   1.       0.       0.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 49.5      1.       0.       0.       1.    ]\n",
      " [ 71.       0.       0.       1.       1.    ]\n",
      " [106.425    1.       0.       0.       1.    ]\n",
      " [110.8833   1.       0.       0.       1.    ]\n",
      " [ 39.6      1.       0.       0.       1.    ]\n",
      " [ 79.65     0.       0.       1.       1.    ]\n",
      " [ 51.4792   0.       0.       1.       1.    ]\n",
      " [ 26.3875   0.       0.       1.       1.    ]\n",
      " [ 55.9      0.       0.       1.       1.    ]\n",
      " [110.8833   1.       0.       0.       1.    ]\n",
      " [ 40.125    1.       0.       0.       1.    ]\n",
      " [ 79.65     0.       0.       1.       1.    ]\n",
      " [ 79.2      1.       0.       0.       1.    ]\n",
      " [ 78.2667   1.       0.       0.       1.    ]\n",
      " [ 56.9292   1.       0.       0.       1.    ]\n",
      " [153.4625   0.       0.       1.       1.    ]\n",
      " [ 39.       0.       0.       1.       1.    ]\n",
      " [ 52.5542   0.       0.       1.       1.    ]\n",
      " [ 32.3208   0.       0.       1.       1.    ]\n",
      " [ 77.9583   0.       0.       1.       1.    ]\n",
      " [ 30.       0.       0.       1.       1.    ]\n",
      " [ 30.5      1.       0.       0.       1.    ]\n",
      " [ 69.3      1.       0.       0.       1.    ]\n",
      " [ 76.7292   1.       0.       0.       1.    ]\n",
      " [ 35.5      1.       0.       0.       1.    ]\n",
      " [113.275    1.       0.       0.       1.    ]\n",
      " [ 25.5875   0.       0.       1.       1.    ]\n",
      " [ 52.       0.       0.       1.       1.    ]\n",
      " [512.3292   1.       0.       0.       1.    ]\n",
      " [ 76.7292   1.       0.       0.       1.    ]\n",
      " [211.3375   0.       0.       1.       1.    ]\n",
      " [ 57.       0.       0.       1.       1.    ]\n",
      " [110.8833   1.       0.       0.       1.    ]\n",
      " [  7.65     0.       0.       1.       1.    ]\n",
      " [227.525    1.       0.       0.       1.    ]\n",
      " [ 26.2875   0.       0.       1.       1.    ]\n",
      " [ 26.2875   0.       0.       1.       1.    ]\n",
      " [ 49.5042   1.       0.       0.       1.    ]\n",
      " [ 52.       0.       0.       1.       1.    ]\n",
      " [  7.65     0.       0.       1.       1.    ]\n",
      " [227.525    1.       0.       0.       1.    ]\n",
      " [ 10.5      0.       0.       1.       1.    ]\n",
      " [ 53.1      0.       0.       1.       1.    ]\n",
      " [211.3375   0.       0.       1.       1.    ]\n",
      " [512.3292   1.       0.       0.       1.    ]\n",
      " [ 78.85     0.       0.       1.       1.    ]\n",
      " [262.375    1.       0.       0.       1.    ]\n",
      " [ 71.       0.       0.       1.       1.    ]\n",
      " [ 53.1      0.       0.       1.       1.    ]\n",
      " [ 12.475    0.       0.       1.       1.    ]\n",
      " [ 86.5      0.       0.       1.       1.    ]\n",
      " [120.       0.       0.       1.       1.    ]\n",
      " [ 77.9583   0.       0.       1.       1.    ]\n",
      " [ 10.5      0.       0.       1.       1.    ]\n",
      " [211.3375   0.       0.       1.       1.    ]\n",
      " [ 57.       0.       0.       1.       1.    ]\n",
      " [ 30.       0.       0.       1.       1.    ]\n",
      " [ 79.2      1.       0.       0.       1.    ]\n",
      " [ 25.9292   0.       0.       1.       1.    ]\n",
      " [120.       0.       0.       1.       1.    ]\n",
      " [  0.       0.       0.       1.       1.    ]\n",
      " [ 53.1      0.       0.       1.       1.    ]\n",
      " [ 93.5      0.       0.       1.       1.    ]\n",
      " [ 12.475    0.       0.       1.       1.    ]\n",
      " [ 83.1583   1.       0.       0.       1.    ]\n",
      " [ 39.4      0.       0.       1.       1.    ]\n",
      " [ 26.55     0.       0.       1.       1.    ]\n",
      " [ 25.9292   0.       0.       1.       1.    ]\n",
      " [ 50.4958   0.       0.       1.       1.    ]\n",
      " [ 52.5542   0.       0.       1.       1.    ]\n",
      " [  5.       0.       0.       1.       1.    ]\n",
      " [ 83.1583   1.       0.       0.       1.    ]\n",
      " [ 30.       0.       0.       1.       1.    ]\n",
      " [ 30.       1.       0.       0.       1.    ]]\n"
     ]
    }
   ],
   "source": [
    "print(Xint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"dummy variable\" columns add up to the intercept column; we've introduced perfect collinearity! (The rank of this matrix is 4, not 5.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.matrix_rank(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.matrix_rank(Xint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas' `get_dummies` has an argument `drop_first` which helps avoid this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "      <th>embark_town_Queenstown</th>\n",
       "      <th>embark_town_Southampton</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>54.0</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>58.0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     age     fare  embark_town_Queenstown  embark_town_Southampton\n",
       "1   38.0  71.2833                       0                        0\n",
       "3   35.0  53.1000                       0                        1\n",
       "6   54.0  51.8625                       0                        1\n",
       "10   4.0  16.7000                       0                        1\n",
       "11  58.0  26.5500                       0                        1"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic_small_dummy_safe = pd.get_dummies(titanic_small, drop_first=True)\n",
    "titanic_small_dummy_safe.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.058473805443555e-10"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the same experiment\n",
    "Xsafe = titanic_small_dummy_safe.values.astype('float')\n",
    "ysafe = Xsafe[:,0]\n",
    "Xsafe = Xsafe[:,1:]\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(Xsafe, ysafe)\n",
    "old_coef = lr.coef_\n",
    "\n",
    "Xsafe[0,-1] += 1e-8 # tiny perturbation\n",
    "lr.fit(Xsafe, y)\n",
    "new_coef = lr.coef_\n",
    "np.abs(new_coef[-1] - old_coef[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0244833 ,  0.71179309, -3.86809095])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_coef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0244833 ,  0.71179309, -3.86809095])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_coef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix conditioning\n",
    "\n",
    "The [condition number of a matrix](https://en.wikipedia.org/wiki/Condition_number#Matrices) tells us how worried we should be about this (see CPSC 302)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = titanic_small_dummy.values.astype('float')\n",
    "y = X[:,0]\n",
    "X = X[:,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1114660.8566703028\n"
     ]
    }
   ],
   "source": [
    "print(np.linalg.cond(X.T@X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.746115252852807e+21\n"
     ]
    }
   ],
   "source": [
    "print(np.linalg.cond(Xint.T@Xint))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, it's much bigger once we add the column of ones. And it's huge!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L2-regularization and the condition number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2193.696967851085"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "λ = 1000\n",
    "np.linalg.cond(Xint.T@Xint + λ*np.eye(Xint.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: nice! L2-regularization seems to reduce the condition number drastically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we explore the effects of regularization on a synthetic data set with collinear features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.30351544e+09, -4.90895417e+12,  4.90665066e+12])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "npr.seed(0)\n",
    "n = 1000\n",
    "d = 100\n",
    "X = np.random.randn(n,d) \n",
    "y = np.sum(X[:,:5],axis=1) + np.random.randn(n)\n",
    " \n",
    "# make the first 3 columns identical\n",
    "X[:,1] = X[:,0]\n",
    "X[:,2] = X[:,0]\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(X,y)\n",
    "lr.coef_[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: without regularization, the coefficients go crazy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.32834422, 0.32834422, 0.32834422])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge = Ridge()\n",
    "ridge.fit(X,y)\n",
    "ridge.coef_[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: with L2-regularization, they are all equal (and about $\\frac13$, which makes sense since the true total weight is 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.08798365e-02, 2.30688627e-16, 0.00000000e+00])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso = Lasso()\n",
    "lasso.fit(X,y)\n",
    "lasso.coef_[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: with L1-regularization, they are not going to be crazy, but they won't all be equal. \n",
    "\n",
    "The solution isn't unique: I could shuffle those 3 coefficients and I'd get another solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04167139954127175"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_score = lasso.score(X,y)\n",
    "old_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.30688627e-16, 1.08798365e-02, 0.00000000e+00])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso.coef_[0], lasso.coef_[1] = lasso.coef_[1], lasso.coef_[0] \n",
    "lasso.coef_[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.04167139954127175\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_score = lasso.score(X,y)\n",
    "print(new_score)\n",
    "new_score == old_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overfitting and $d>n$\n",
    "\n",
    "One question is what it means to overfit with a linear model: the model is so _simple_!\n",
    "\n",
    "One answer is that if we have a lot of features, then we have a lot of parameters (one per feature), which can make the model complex even if it's linear. For example, with $d=n$ (or $n-1$, if we're being careful about the intercept) and the features not collinear ($X$ is a square matrix of rank $n$), we can now achieve a training error of 0 on any data set. \n",
    "\n",
    "Intuition: for 0 loss we need $Xw=y$. So we have $n$ equations. But we also have $n$ unknowns since we have $d$ parameters and $d=n$. So we have an $n\\times n$ linear system, which should be solvable. Meaning we can get zero training error. Which sounds like overfitting.\n",
    "\n",
    "All this relates to the conditioning and invertibility issues discussed above. Because if $d=n$ then $X^TX$ then $X^TX$ has rank $n=d$, and so its invertible and we get a unique solution (the one with zero training error). And if $d>n$ then $X^TX$ can have rank at most $n$, but it's a $d\\times d$ matrix, and so it must not be invertible. In other words, the solution is not unique. There is a whole space of solutions that give zero training error.\n",
    "\n",
    "To summarize, all these things are related: the condition number of $X^TX$, the number of features (but also their collinearity), the rank of $X^TX$, the uniqueness of the solution, the sensitivity of $w$ to changes in the data.\n",
    "\n",
    "We can see the affects of overfitting by adding random features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEUCAYAAAASvPDLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4HNXV+PHvUZdsNau6yU2uFGNjqg02vQQMCQFCSEJJICGF9EIKISTkfQN50wOBQML7/iChhOYkhIDBNjYltlzB3ca9SbKK5SLJks7vjztrr9craWTvanel83mefVY7c2f27Oyuzt47d+4VVcUYY4yJlqRYB2CMMaZns0RjjDEmqizRGGOMiSpLNMYYY6LKEo0xxpioskRjjDEmqizRGGOMiSpLNMYYY6LKEo0xxpioSol1ALFWWFioQ4cOjXUYxhiTUBYuXFitqkV+yvb6RDN06FAqKipiHYYxxiQUEdnkt6w1nRljjIkqSzTGGGOiyhKNMcaYqLJEY4wxJqos0RhjjIkqSzTGGGOiyhKNMcaYqLJEE4fq9jfz4uJtsQ7DGGMiwhJNHHr87Y185ekl7Kg/EOtQjDHmuFmiiUMVG2sB2FnfGONIjDHm+FmiiTOtbcrizS7R7NpjicYYk/i6PdGIyCAR+a2IvCMi+0VERWRomHIZIvKAiOwQkQNe+XPDlEsSkbtEZKOINIrIUhG5pjteSzSs2rmHfc2tgNVojDE9QyxqNOXAdUAtMLeDco8BtwF3A1cAO4B/i8gpIeV+DNwD/A64DHgXeFZELo9s2N1j4abaQ3/vamiKYSTGGBMZsRi9+U1VLQEQkc8AF4cWEJHxwMeBW1X1z96yOcBy4F5guresGPgG8N+q+nNv81kiUg78N/BylF9LxFVsrKU0J4MkgV1WozHG9ADdXqNR1TYfxaYDB4Gng7ZrAZ4CLhGRdG/xJUAa8ETI9k8AJ4nIsOOPuHst3FTLqUPyKcnNYKedozHG9ADx2hngBGCDqu4PWb4cl1jKg8o1AevClAMYF7UIo2BnfSPb6g5w6pB8SnMs0RhjeoZ4TTT9cOdwQtUErQ/c16mqdlIu4uatrWb59vqI7rNikwt70tB8SnIyqNxj52iMMYkvXhONAKHJI7D8WModuVLkdhGpEJGKqqqqLgfX3NLGd194j5v/vIAtNaGVrmNXsbGWzNRkxvbPoSQng71NLextaonY/o0xJhbiNdHUEL42kh+0PnCfLyKhiSW03BFU9RFVnaSqk4qKfE15fYS0lCQevWkSTQdbuelP86nZ19zlfYSzaHMt4wfnkpqcRGmuOw1lXZyNMYkuXhPNcmCYiGSFLB8HNHP4nMxyIB0YEaYcwIpoBTiqJJvHbj6NbXUHuPXxBexvPr6ax/7mFpZv38OkIS6/luRkAFBp52mMMQkuXhPNDCAVuDawQERSgOuBV1U1cPLiFVziuTFk+08A76vqhmgGedrQfvzmhgks21rHF/+ymJZWPx3qwluypY7WNuXUoa4yFkg01iHAGJPoYnEdDSLyUe/PU737y0SkCqhS1TmqukREngZ+JSKpwAbgDmAYQUlFVStF5JfAXSLSACzCJaPzgau647VcckIp9151It9/8X2++8J7/Oyakzm6Ja9zi7wLNScOdomm1BKNMaaHiEmiAZ4Nefygdz8HmOb9fQtwH/ATIA9YClyqqotCtv0esBf4MlAKrAauU9W/Rz7s8D5x5hAqG5r4zetrKcnJ4OsXj+7yPio21TKqpC+5WakA9ElPITs9xXqeGWMSXkwSjap2+pNfVQ8AX/NuHZVrxSWjn0QmumPz1QtHUrmnkd++sY7i7HQ+edZQ39u2tSmLNtXyoZMHHLG8OCfdOgMYYxJerGo0PY6I8JOrT6R6bxN3z1hOUXY6l57Y39e2ayv3sqexhUlD8o9YXmqjAxhjeoB47QyQkFKSk/jtDROZMDiPO59acmi4/84EBtI8NSTRuIs2LdEYYxKbJZoIy0xL5rGbTqNfVhr/9a9Vvrap2FRDYd80hhQc2Zu7NCeDyoYm2trCXZNqjDGJwRJNFOT3SeOzU4czf0MN8zeEvWb0CIGBNEN7q5XkZNDSplTvsw4BxpjEZYkmSj52WhmFfdP43azQ8T6PVNXQxKbd+49qNoPgizYt0RhjEpclmijJTEvm01OG8+aaKpZuqWu33OHzM0ePuFOa611LYz3PjDEJzBJNFH3izDJyM1P5fQe1moWbakhLSeLEgTlHrSvJ8cY7sw4BxpgEZokmirIzUrn57KG8umIXq3buCVumYlMtJw/MJT0l+ah1RX3TSRIb78wYk9gs0UTZLZOH0ictmQdnrT9qXePBVt7fVn9ofLNQKclJFPZNtxqNMSahWaKJsrysND5x1hD+sWw7G6r3HbHuvW31HGzVQyM2h1OSk8FO6wxgjElglmi6wWemDCc1OYmHZh95rqZiY/gLNYPZRZvGmERniaYbFGWnc8PpZTy/aBtbaw/PyLlwUw3DC/vQr09au9uW5lrTmTEmsVmi6Sa3nzscEXjkzQ8AUNVDF2p2pCQ7g7r9B2k82NodYRpjTMRZoukmA/IyuWbiIJ5asIXKPY18UL2P2v0HmdROR4CAkly7aNMYk9gs0XSjO6aNoKW1jUfnbWChj/MzYBOgGWMSn00T0I2GFPRh+vgBPPHuJiaXF5KXlcrwwr4dbmNTOhtjEp3VaLrZF84rZ39zK6+t2MWpZfkkJXU8B1zpofHOLNEYYxKTJZpuNrIkm0tPKAVo90LNYDmZKWSkJtl4Z8aYhGWJJgbuvGAkhX3TOW90cadlRYTSHJtp0xiTuOwcTQyMG5BDxfcv9F2+OCfDep0ZYxKW1WgSgNVojDGJzBJNAijNdYlG1aZ0NsYkHks0CaA4O53mljbqDxyMdSjGGNNllmgSwKGZNq35zBiTgCzRJIBDowNYF2djTAKK20QjIpNF5FURqRSRPSKySERuDSmTISIPiMgOETkgIu+IyLmxijlaSnJsvDNjTOKKy0QjIicDM4FU4DbgGmAB8JiI3BFU9DFv/d3AFcAO4N8ickr3RhxdxTnpgDWdGWMSU7xeR/MxIBm4UlX3esteE5HxwKeAh7y/Pw7cqqp/BhCROcBy4F5geveHHR3pKcn065NmicYYk5DiskYDpAEHgQMhy+s4HPN0r8zTgZWq2gI8BVwiIundEGe3Kc5Ot/HOjDEJKV4TzePe/W9EZICI5InIbcAFwC+9dScAG1R1f8i2y3GJqrxbIu0mgWtpjDEm0cRlolHV94FpwFXANqAW+D3wOVV9yivWz1seqiZofY9RmpPBznrrDGCMSTxxeY5GREYCz+FqJ5/DNaFdBfxBRBpV9UlAgHCXync87r7b/+3A7QBlZWWRCjuqinMy2L2viYOtbaQmx+XvA2OMCSsuEw3wU9z5lytUNXA5/OsiUgD8WkT+iqu5hMsSgbH3a8KsA0BVHwEeAZg0aVJCjOtSmpOBKlQ1NDEgLzPW4RhjjG/x+tP4JGBpUJIJmA8UAMW42s4wEckKKTMOaAbWRT3KblSaa12cjTGJqdNE410U+aqITOuGeAJ2AqeISFrI8jOARlxtZQbuOptrAytFJAW4HnhVVXvUCY0Sm2nTGJOgOm06U9VGETkNd11Ld/kd8CzwdxF5EHeOZjpwA/BLVW0GlojI08CvRCQV2ADcAQwDbuzGWLtFiQ1DY4xJUH6bzmYAV0czkGCq+jfgciAdeBTXMWAK8AXgm0FFbwH+DPwE+CcwGLhUVRd1V6zdpV9WGqnJwk4bhsYYk2D8dgb4N/CAiPQHXgZ2EdLjS1VfjmRgqvov4F+dlDkAfM279WhJSUJxdkZEms6e/M8mJo8oZGhhnwhEZowxHfObaJ7w7j/i3UIp3du01iuV5KQfd2eADdX7+N4L73PtqYN44NrxEYrMGGPa5zfRDItqFMaX0twMVu1sOK59vL5yFwBz1lShqoh0etmRMcYcF1+JRlU3RTsQ07mSnAzmrK46rn3MXLkLEahsaGLFjj2cMCA3QtEZY0x4vq+jEZEUEbleRH4rIk9699d5XYpNNyjJyWBfcyt7m1qOafu6/c0s2FjLdacOBmD2cSYtY4zxw1eiEZFioAL4K/AhYLh3/xSwQESKohahOeR4Z9qcvbqK1jblY6cP5oQBOcxeXRnJ8IwxJiy/NZpf4K7IP0NVh6vqWao6HHcBZYG33kRZ4FqaXcfYIWDmyl0U9k1n/KA8zhtdzKLNddQfCB18wRhjIstvorkc+LaqLghe6D2+C1e7MVFW4s20eSyJprmljTmrq7hwbDFJScK00UW0tinz1lZHOkxjjDmC30STDrTX3akBN/+LibLSXK/p7BgSzfwNNTQ0tXDB2BIAThmcR05GijWfGWOizm+ieRf4togccYWf9/jb3noTZVlpKWRnpLDrGM7RzFy5i/SUJKaUFwKQkpzEOaOKDnVzNsaYaPHbY+zrwCxgi4i8ihsZoBi4BDf/y7SoRGeOUpKTwa4uDkOjqsxcuYtzRhaSmXb4utppo4r457Id1s3ZGBNVvmo0qroEGImbw6UIuAiXaP4AjFTVpVGL0ByhNKfrUzqv3tXA1toDh5rNAqaOdp0FrZuzMSaaOq3ReCMjnw5sUNXvRD8k05GSnAzWr+/aCfzXV7rzMBeMKT5ieXF2BicOdN2cv3BeecRiNMaYYH5qNK3AG8DYKMdifCjJSaeyoYm2Nv/nVV5bsYvxg/Mo9rpHB5s2yro5G2Oiq9NEo6ptwFqgpLOyJvpKczNobVOq9/k7T1PZ0MiSLXVcGFKbCbBuzsaYaPPb6+x7wN0iclI0gzGdO3TRZr2/RDNrlWs2u3Bc+N8J1s3ZGBNtfnudfR83AsASEdlG+PloTo9wbCaM0qDRAU6i855ir62oZGBeJmNKs8OuD3Rznm2jORtjosRvolkOvB/NQIw/h6Z09tHzrPFgK/PWVXH9pMEdJpBAN+fl2/dw4kDr5myMiSy/0wTcHOU4jE+FfdNIEn/D0Ly1rprGg23tNpsFBLo5z1lTZYnGGBNxnZ6jEZEMEWkSkau7IyDTsZTkJIqy030lmpkrd9E3PYUzhhV0WC64m7MxxkSan15njUAlcGyToJiIK8nJYGcnowO0tSkzV1YydVQRaSmd9/mwbs7GmGjx2+vsYeBO7+JNE2MlORls2r2PA82t7ZZ5b1s9VQ1NXDgufLfmUNbN2RgTLX47A+QBJwIbReR1ju51pqr67UgHZ8KbNrqI11bs4qJfzuFH0084amgZcM1myUnCeaP9JZpAN+dZqyv50Mn9Ix2yMaYX85torgECbTXnhFmvuFGcTTe48YwhjCjqyw9efJ9P/28FF40r4YdXjmNQftahMq+t2MWpQ/LJy/I3g0NKchLneqM5t7UpSUnWzdkYExl+B9Uc1slteLQDNUc6c3gB/7zzHL5z2Rjmra3mwl/M4cHZ62huaWNr7X5W7WzgojA1nY5MG11MVUMTK3bsiVLUxpjeyG+NxsShtJQkPjd1BFeOH8C9f1/O/a+s5rmFWzl1SD7Q/mgA7Zk6yro5G2Miz29nAETkZBF5WkTWe92dJ3rL7xORy6IVoIhcLiJvisheEdkjIhUicn7Q+nwReVREqkVkn4jM7G1D5QzMy+ThT07iTzdPorm1jWcqtjK8qA/DCvt0vnGQoux06+ZsjIk4X4nGSyQLgVLg/4Dg3mdNwJciHxqIyGeBl7zn/jBwLfAskOWtF2AGcKkXwzVebLNEZFA0Yopn548p4bWvTuWuy8bwgw+NO6Z9BLo5VzV0bXI1Y4xpj/iZxldElgALVPU2EUkBmoFJqrpIRKYDf1DVARENTGQosBK4S1V/1U6Zq4AXgfNVdZa3LBfYADyhqnd29jyTJk3SioqKSIWd8JZtreOq379FSpIwaUg/zhtTxLTRxYws7mvjoBljDhGRhao6yU9Zv+doxgDf8P4OzUx7gH4+99MVtwJtuFk82zMd2B5IMgCqWi8ifweuAjpNNOZIJw/K47k7zubfy3cyZ3UVP315FT99eRUD8zKZOrqIaaOKmFxeSJ90O71njPHH73+LSqC9nmUnAJsjE84RpgCrgI+JyA+AIcBG4Jeq+vug5w432Ody4FMi0ldV90Yhth5tYlk+E8vyueuysWyvO8CcNVXMWlXJS4u38Zf/bKY4O53XvjqV3Cy7ftcY0zm/ieYp4F4RWQG84y1TERmFu37msSjENsC7PQB8F1iPO0fzOxFJUdVf42pSG8NsW+Pd5wOWaI7DgLxMbji9jBtOL6O5pY0XFm/l28+9x8LNNZw/xubCM8Z0zm+vsx8AFcAcDtdeXsLVJpYBP418aCQB2cBnVfWPqvqGqt4BvALc5XUEEI5uysNb3i4Rud3rvVZRVVUV8cB7qrSUJK4cP4AkgcWb62IdjjEmQfidJqAJuEJELgAuAApxtYbXVfW1KMW2GxgJhO7/VVwvs/5eDOHOD+V797XhdqyqjwCPgOsMEIlge4ustBTGlOawaHPYQ2uMMUfp0hldVX0deD1KsYRaDpwZZnmgttLmlbk4TJlxwGY7PxMdE4fk8eLi7bS2Kck2VI0xphO+L9iMgRe8+0tCll8CbFXVnbhraAaKyNTAShHJAa701pkomDA4n71NLaytbIh1KMaYBBDPfVRfBmYBD4tIIfAB8FFcDeYWr8wMXOeEJ0Tkm7imsrtwtZ77uz3iXmKiN8TN4s11jCnN6dK2b6zaRWZqCmeN6HgyNmNMzxG3NRp1V5Jejevx9iPgH7imtBtV9XGvTBtwBe48zoO4WlArcJ6qbolB2L3C0IIs8rNSWdzF8zSqyrf+9h4/+vvyKEVmjIlH8VyjQVX3AF/wbu2VqcFd3Hlrd8XV24kIE8ryWdTFnmerdjZQvbeJ6r1NVDY0UpydEaUIjTHxJG5rNCa+TRicx7rKvV2a+jl49s631+2ORljGmDjkq0YjIp/qYHUbbhiapaq6KSJRmbg3ocydp1mype7Q9AKdmbuumuGFfajd38y8ddVcPWFgNEM0xsQJv01nj3P4wsjg/qzBy1RE/oE7h2Ldinu48YNzEYHFm2t9JZrGg63M37Cbj51WRlVDE2+tq0ZVbaBOY3oBv01nE4F1wHeAsbgLNsfienitBy4EbsJN8/yzyIdp4k12RiqjirN9jxCwaFMtjQfbmFJeyOTyQnbUN/JB9b4oR2mMiQd+azT/AzzojS8WUAPcLyLNwA9VdaqIlABfp4OT96bnmDgkj38u20Fbm5LUyYWbc9dVk5IknDmigJq9zQC8ta6aEUV9uyNUY0wM+a3RnIWbGyaclcBp3t8LAbtAopeYMDifPY0tvmom89ZWM6Esj77pKZQVZDG4X+YRnQOMMT2X30SzFbi5nXW3eOvBjTFm3Yl6iQlleQCdjntWu6+Z97fXM6X88LmcKeWFvLN+Ny2tbVGN0RgTe34TzfeA60RkmYjcJyJf8e6X4a7W/65X7iJgbjQCNfFnRFFfsjNSOj1P89b6alRhysjCQ8smlxfS0NTCsm310Q7TGBNjfkdvflZENuDmnvk4UArsBBYAt6jqQq/c56MVqIk/SUnCKYPzOh0hYN7aarIzUhg/KPfQsrNHFCICb62tZmJZfgdbG2MSne8LNlW1QlWvVdVhqprp3V8XSDKmd5pYls/qXQ3sbWoJu15Vmbu2mrOGF5CSfPjj1q9PGicMyGHeOjtPY0xPZyMDmOMyoSwPVVi2JXzz2cbd+9lWd4BzgprNAiaXF7Jocy37m8MnKWNMz+A70YjIR0XkLyLypojMD71FM0gTvyYMds1e7XUImLfWzWA6ZeTRF3VOKS/kYKsyf0PNUeuMMT2H3yFo7gHuBpYCK4DmKMZkEkhuViojivq02yFg7tpqBuZlMrQg66h1pw3tR1pKEm+tq2ba6OJoh2qMiRG/F2x+GvhvVf1upyVNrzOhLJ83VlUeNaRMS2sb76zfzYdO7h92qJmM1GQmDclnrl1PY0yP5rfpLJvum8LZJJiJZfnU7Gtm0+79Ryxftq2ehqaWI7o1h5pcXsiqnQ1UNTRFO0xjTIz4TTRPAZdGMxCTuAIXbi7ecuR5mnlrqxGBySPaTzRTyt26t9dbrcaYnspv09nrwM+8KZVfA45qkFfVlyMZmEkco0qy6ZOWzKJNdXx4wqBDy+etrebEAbnk90lrd9sTB+aSm5nKW+uqueoUmzbAmJ7Ib6J52rsfihulOZQCyZEIyCSe5CRh/OC8I2o0e5taWLS5ltvOHd7ptmePKGDeWps2wJieym/T2bBObh3/NzE93oSyPFbuaDh0Tcx/PthNS5tyTnn7zWYBk8sL2V7fyMaQczzGmJ7B7xA0NnOm6dDEsnxa25T3ttZzxvAC5q6tJiM1iVOHdj68TOA8zby1VQwr7BPtUI0x3azdGo2IZAX/3dmte8I18eqUwYGRnN3pu3nrqjl9WAHpKZ23qA4pyGJgXqYNR2NMD9VR01mDiJzu/b0XaOjkZnqxgr7pDC3IYvHmWnbUH2Bd5V5fzWYAIsKU8kLeXr+b1jbtfANjTELpqOnsVtw0zYG/7T+A6dCEsnzmras+NKFZR9fPhJo8spCnK7bw3rb6Q7UjY0zP0G6iUdX/Dfr78W6JxiS0CWV5vLB4G08v2EJh33TGlGb73vbsEW5i1rfWVVuiMaaHsdGbTcQE5pWp2FTLlPKCLnVVLuybztj+OTa9szE9kK9EIyKpIvINEXlbRDaLSGXoLdqBisgrIqIi8pOQ5fki8qiIVIvIPhGZKSInRTsec7TRpdlkpLqPVLjRmjszpbyAhZtqOdDcGunQjDEx5PeCzV8CnwX+Acyim0dvFpEbgPFhlgswA3ctz5eAWuAuYJaInKKqW7szzt4uNTmJkwflMX9DzaEuy10xZWQRf5y7gfkba5g6quuJyhgTn/wmmmuB76jq/0QzmHBEJA+X6L4K/CVk9XRgCnC+qs7yyr8DbAC+BdzZjaEa4NpTBzEoP5PS3Iwub3u6N23A6yt3WaIxpgfxe45GgGXRDKQD9wPLVfWvYdZNB7YHkgyAqtYDfweu6qb4TJBrJw3mF9edckzbZqYlc/G4El5asp3Gg9Z8ZkxP4TfR/BG4IZqBhCMiU4BPAZ9vp8gJwPthli8HykSkb7RiM9Fx3aTB1B84yMyVu2IdijEmQvw2ne0CbhSRWYQfvVlV9aFIBiYiqcDDwM9VdXU7xfoBG8MsD8wNnI+72DR037cDtwOUlZUdd6wmciaXFzIgN4NnKrZyxckDYh2OMSYC/CaaX3n3ZcDUMOsViGiiAb4NZAL3dVBGCH8haYf9alX1EeARgEmTJtmFqHEkOUm45tRB/G7WOrbXHWBAXmasQzLGHCdfTWeqmtTJLaJTBIhIGfA94AdAuojkeZ0CCHqcjKu59Auzi8BIjrVh1pk499FTB6EKzy+yToPG9ATxesHmcCADeAKXLAI3gG94f5+EOxdzQpjtxwGbVfWoZjMT/4YU9OHM4f14pmIrbTb2mTEJr6PRm8eJSHrQ3x3eIhzXEuC8MDdwyec8YB3uGpqBInKoOU9EcoArvXUmQV03aTCba/Yzf2NN54WNMXGto3M07wNnAvO9v9v7aRk4TxKx5jNVrQNmH/VEbkiTTao623s8A3gHeEJEvsnhCzYF1y3aJKjLTuzP3S8t55mKLZw5vCDW4RhjjkNHieY8YEXQ33FHVdtE5Arg58CDuOa2d4DzVHVLTIMzxyUzLZkrxw/ghcVb+dH0E8jOSI36c27avY8kEQb3s+mVjImkjkZvnhPu71hS1aN6k6lqDW4ag1u7PyITTddNGsRf52/mH8t2cMPp0e+G/sW/LKappZV/f+XcLg0IaozpWJc7A4hIks2wabrDKYPzGFncl2cqol85PdDcyoode1iza++hWUKNMZHhd/RmEZFvi8g64CA2w6bpBiLCdZMGs3hzHWt3Rfcjtnx7/aHZPZ+avzmqz2VMb+O3RnMn8B3gMdyJ9vuAe4E1uCvzb49GcMZcPWEgKUnCswuje03Nki2uFnPBmGL+sWwHDY0Ho/p8xvQmfhPNbcAPOdyT60VV/RHuGpZVwMgoxGYMRdnpnD+mmOcXbeVga1vUnmfp1noG5GbwxfPLOXCwlRlLt0ftuYzpbfwmmmHAElVtxTWd5YHr9YXr7XVTdMIzxo0IXb23mVmroje/3rKtdYwfnMcpg/MYU5rN0wus06IxkeI30ewGAiMhbwYmBK3Lx41JZkxUTBtdRGHfdJ6piE7zWe2+Zjbt3s/4wXmICNefNphlW+tZvr0+Ks9nTG/jN9G8BZzm/f0X4B4RuU9Efgj8Ang9GsEZA27mzmsmDmTW6koqGxojvv+lW935mZMH5QLw4QkDSUtJslqNMRHiN9HcA8z1/v4p8CfgZuDLuKmd74h0YMYEu3bSIFrblBcXbztieXNLG++s383PXlnFh34zl9Pum8nuvU1d2vfSLfWIwEkDXaLJy0rjshNLeWHxNg402wRsxhyvTqcJEJEk3Jwu8wFUtQmXYL4c3dCMOay8OJuJZXk8vWALF40r5c01Vby5pop3PtjN/uZWUpKEsf1zqGpoYu7aaq6eMND3vpdtraO8qO8Row987LQyXlqynX+9v4OPTBwUjZdkTK/hp0aThOvCPCW6oRjTsesmDWZ91T7O+/lsfjhjOWsr9/KRiQN55JOnsvjui3jpC5PJz0rlzbVVvvepqiz1OgIEO3N4P4YWZPHUfGs+M+Z4dVqjUdUWEdkE2NX/JqauOmUga3btZWhhFueOLGJoYZ+jykwuL2Te2mpU1dcwMtvqDlC9t5nx3vmZANcpoIyfvbKK9VV7GVFks4Ibc6z8nqP5GfA9ESmKZjDGdCQzLZm7rxzHp84aGjbJAJwzspDKhibWVvqbimjZVtezLLRGA3DNqe5iUesUYMzx8TuV88VAf2CjiCwEdnHktAGqqtdHOjhjumrKSPdb6M01VYwqye60/NItdaQlJzGmNOeodcXZGVwwtpjnFm7lGxePJi0lXucJNCa++f3mFAKrcR0CWr3HRUG34qhEZ0wXDczLZHhhH+atq/ZVfsmWOsYNyGk3iXzstDJ272tm5sqUdzhpAAAbAklEQVRdkQzTmF7FV41GVeNyPhpjwjlnZCHPVGylqaWV9JT25+NrbVPe21bPtae236vs3FFF9M/N4KkFW7j8pP7RCNeYHs/v6M13i8iAdtb1F5G7IxuWMcduysgiDhxsZdGmjof7X1+1l/3NrWHPzwQkJwnXThrM3LVVbKnZH+lQjekV/Dad/RBo72ffAG+9MXHhzOH9SE4S5nbSzTkwYnNHiQbcBGxA1EeQNqan8ptohCNP/gcbBNRGJhxjjl92RioTy/I6PU+zdEsd2ekpDCsI34MtYFB+FueMLOLZii2H5qwxxvjXbqIRkZtE5A0ReQOXZB4KPA66vQ08AcTFVM/GBEwpL+K9bfXU7mtut8yyrfWcPDiXpKTOr7e54bTB7KhvZM6a6I0gbUxP1VGNZj9u1ObduBpNfdDjwG0Dbo4am/jMxJUpIwtRhbfWh6/VNB5sZeWOPYwf1HGzWcAFY0vIy0rlH0t3RDJMY3qFdnudqeqzwLMAIvJn4F5V3dBdgRlzPMYPyiU7I4V5a6u54uSj+7Gs2LGHljbt9PxMQFpKEueOLOLNtdW0tamvWpAxxvF1jkZVb7EkYxJJSnISZ48oYK43HE2opYGOAD5rNABTRxVRvbeJFTv2RCxOY3oDu9TZ9FhTRhaxre4AG3cf3S152dZ6SnLSKc3N8L2/c0e5UQfmrPE/aKcxxhKN6cHOKS8ECNvNeemWui7VZgCKstM5cWAOc1ZbojGmK+I20YjIR0XkORHZJCIHRGS1iPyXiGSHlMsXkUdFpFpE9onITBE5KVZxm/gxpCCLQfmZzF17ZIeA+gMH+aB6n+/zM8Gmjipi4eZa6g8cjFSYxvR4cZtogG/gxlX7LnAp8BBuJs/XvMnYEDcO/Axv/ZeAa4BUYJaI2GxVvZyIcM7IIt5dv5uDrW2Hlr8XGLG5izUagKmjimltU972OZaaMSa+E82Vqnqdqj6pqnNU9VfAncAZwDSvzHTchGyfVNW/quor3rIk4FuxCNrEl3NGFtLQ1HLo5D/A0q3u75NC5qDxY2JZHtkZKXaexpguiNtEo6rhvskLvPvAPL3Tge2qOitou3rg78BV0Y3QJIKzRxQgwhHNZ0u21DG8qA+5makdbBleSnISU8oLmbOmKmxvNmPM0eI20bRjqne/0rs/AXg/TLnlQJmI2LSIvVxeVhonDzpyOJqlW+o45RiazQKmjipiR30ja3b5m1zNmN4uYRKNiAwE7gVmqmqFt7gf4cdZq/Hu87sjNhPfzikvZMmWOvY0HmRnfSOVDU2cfAzNZgFTRwe6OdtwNMb4kRCJxquZvAS0ALcEryL8YJ8dXrYtIreLSIWIVFRVWVt7TzdlZCGtbco763f7HrG5I/1zMxldkm3naYzxKe4TjYhk4HqWDQcuUdXgsdprcLWaUIGaTNhRpVX1EVWdpKqTioqKIhqviT8Ty/LJSktm3tpqlm6tIzVZGNv/6Kmbu2Lq6CIWbKhlX1NLhKI0pueK60QjIqnAc8DpwOWq+l5IkeW48zShxgGbVdUa0Q1pKUmcObyAeeuqWba1jjGlOWSktj/zph9TRxXR3NrGO+t3RyhKY3quuE003rUyTwIXAFep6rthis0ABorI1KDtcoArvXXGADClvJAN1ftYsKGW8YOP/fxMwKShrpZkzWfGdK7d0ZvjwO+Ba4H7gH0icmbQuq1eE9oM4B3gCRH5Jq6p7C7cOZr7uzleE8fOGemGo2lubTumCzVDpackc/aIAmavqURVcdcOG2PCidsaDXCZd/89XDIJvn0GQFXbgCuA14AHgRdwowmcp6pbujtgE7/Ki/tSkpMOwCnH0REg2NRRRWypCT9opzHmsLhNNKo6VFWlnds9QeVqVPVWVe2nqlmqeoGqLo1h6CYOiQjnjS4mNzOV4UWRubxq6qhiAGavtm7OxnQkbhONMZF21+Vjef7zZ5McoUnLygqyGF7Yx87TGNMJSzSm18jNTGVEhGozAeeOKuLdD3bTeLA1ovs1piexRGPMcZg6uojGg23M31DTeWFjeilLNMYchzOHFZCWksRsmwzNmHZZojHmOGSmJXPm8AIb98yYDliiMeY4TR1VxPqqfWypsW7OxoRjicaY4zR1lBsv78211nxmTDiWaIw5TiOK+jAoP9PO0xjTDks0xhwnEWHqqCLeXldNc0tbrMMxJu5YojEmAqaOKmJfc6t1czYmDEs0xkTAOSOLyMtK5fG3N8Q6FGPijiUaYyIgMy2Zm88eysyVlaze2RDrcIyJK5ZojImQm84aSlZaMn+Ysz7WoRgTVyzRGBMh+X3SuOH0MmYs3W7X1BgTxBKNMRH0mXOGkSTwx7kfxDoUY+KGJRpjIqh/biYfnjCQpxdsoXpvU6zDMSYuWKIxJsI+O3UEza1tPP7WxliHYkxcsERjTISNKOrLpSeU8r/vbKSh8WCswzEm5izRGBMFd0wbQUNjC3/5z+ZYh2JMzFmiMSYKTh6Ux5TyQh6dt8Fm3zS9niUaY6LkjmkjqGpo4vlF22IdijExZYnGmCg5e0QB4wfl8vCb62lptcE2Te9licaYKBER7pg2gk279/Ov93fGOhxjYsYSjTFRdPG4UoYX9eHB2etR1ViHY0xMWKIxJoqSkoTPTR3Byh17mLPGJkYzvVOPSDQiMlhE/iYi9SKyR0SeF5GyWMdlDMDVpwykf24GD85abz3QTK+U8IlGRLKAN4AxwE3AJ4GRwCwR6RPL2IwBSEtJ4rPnDmf+xhrG/+hVPvnYf3ho9nre21pPa5s1p5meLyXWAUTAbcBwYLSqrgMQkWXAWuCzwC9iGJsxANx09lCGFPThzbVVvLWump+9soqfAbmZqZw9ooCzyws5d2QhQwrst5HpeSTRT1CKyOtAhqpODlk+B0BVp3a0/aRJk7SioiKKERpztMo9jby9fjdvravmrXXVbK9vBKC8uC8XjC3morElTCjLJzlJYhypMeGJyEJVneSrbA9INDuBl1T1syHLHwSuVdWijra3RGNiTVXZuHs/c1ZXMnNlJf/ZsJuDrUq/PmmcN7qYC8cWc86oIvqm94QGCNNTdCXR9IRPbj+gNszyGiC/m2MxpstEhGGFfRhWOIybJw9jT+NB3lxTxesrK5m5chfPLdpKWnISZQVZWP3GRNI/7pxCekpy1J+nJyQagHDVsna/kyJyO3A7QFmZdU4z8SUnI5UrTh7AFScPoKW1jYWbanl9VSVba23WThNZ0k0/XXpCoqnF1WpC5RO+poOqPgI8Aq7pLHqhGXN8UpKTOGN4AWcML4h1KMYcs4Tv3gwsB04Is3wcsKKbYzHGGBOiJySaGcCZIjI8sEBEhgKTvXXGGGNiqCckmj8CG4GXROQqEZkOvARsAR6OZWDGGGN6QKJR1X3A+cAa4P8BTwIbgPNVdW8sYzPGGNMzOgOgqpuBa2IdhzHGmKMlfI3GGGNMfLNEY4wxJqos0RhjjImqhB/r7HiJSBWwqQubFALVUQqnJ7PjdmzsuB0bO27HpivHbUhnY0kG9PpE01UiUuF3IDlzmB23Y2PH7djYcTs20Tpu1nRmjDEmqizRGGOMiSpLNF33SKwDSFB23I6NHbdjY8ft2ETluNk5GmOMMVFlNRpjjDFRZYnGBxEZLCJ/E5F6EdkjIs+LiM2Y5hGRaSKiYW51IeXyReRREakWkX0iMlNETopV3N1JRAaJyG9F5B0R2e8dn6FhymWIyAMiskNEDnjlzw1TLklE7hKRjSLSKCJLRaTHDcPUheMW7vOnInJKSLnectw+KiLPicgm73O0WkT+S0SyQ8r5+k76/Vy2xxJNJ0QkC3gDGAPcBHwSGAnMEpE+sYwtDt0JnBV0uzCwQkQEN23DpcCXcGPTpeKO46DuD7XblQPX4Sbjm9tBuceA24C7gSuAHcC/Q/9hAj8G7gF+B1wGvAs8KyKXRzbsmPN73AAe58jP31m4wXaD9Zbj9g2gFfgu7jv3EHAH8JqIJEGXv5N+P5fhqardOrgBX/besPKgZcOAFuBrsY4vHm7ANNx02hd2UOYqr8x5QctygRrgN7F+Dd1wjJKC/v6MdyyGhpQZ7y2/JWhZCrAamBG0rBhoAn4Usv3rwLJYv9buPm7eOgV+0sm+etNxKwqz7FPecTrfe+zrO+n3c9nRzWo0nZsOvKuq6wILVHUD8BbujTL+TAe2q+qswAJVrQf+Ti84jqra5qPYdOAg8HTQdi3AU8AlIpLuLb4ESAOeCNn+CeAkERl2/BHHB5/Hza/edNyqwixe4N0P9O79fif9fi7bZYmmcycA74dZvhw3XbQ57EkRaRWR3SLyl5DzWB0dxzIR6ds9Ica1E4ANqro/ZPly3D/I8qByTcC6MOWg934u7xCRJu9czhsick7I+t5+3KZ69yu9e7/fSb+fy3ZZoulcP1z7cKgaIL+bY4lX9cD/4Jo2zse1g18IvCMixV6Zjo4j2LGEzo9Rv6D7OvXaMDoo15s8AXwe97m7HSgA3hCRaUFleu1xE5GBwL3ATFWt8Bb7/U76/Vy2q0dMfNYNwl1sJN0eRZxS1cXA4qBFc0TkTWA+roPA93HHy45jx/weIzuWIVT1k0EP54rIS7hf6z8BpnjLe+Vx82omL+HOK98SvIpu+rxZjaZztYTP2PmEz/IGUNVFuB4/p3mLamj/OIIdS+j8GNUE3ed7vYY6KtdrqWoD8E8Of/6gFx43EcnA9SwbDlyiqluDVvv9Tvr9XLbLEk3nluPaKEONA1Z0cyyJJviXUEfHcbOq7u22qOLXcmCY16U+2DigmcPnFpYD6cCIMOXAPpcBob/Ee9VxE5FU4DngdOByVX0vpIjf76Tfz2W7LNF0bgZwpogMDyzwLhib7K0zYYjIJGAU8B9v0QxgoIhMDSqTA1yJHceAGbjrGK4NLBCRFOB64FVVbfIWv4L7gt8Ysv0ngPe9XpG9mvfZ+hCHP3/Qi46bd63Mk8AFwFWq+m6YYn6/k34/l+2yczSd+yPwReAlEfk+7hfSj4EtwMOxDCxeiMiTwAZgEVAHTADuArYBv/WKzQDeAZ4QkW/iquV34X513t/dMceCiHzU+/NU7/4ycRPvVanqHFVdIiJPA7/yfo1uwF1kN4ygf46qWikivwTuEpEG3HG/HtcRo8d1Fe/suInIN4DRwCxgOzAEd8FiKb33uP0elxjuA/aJyJlB67Z6TWi+vpN+P5cdivWFRYlwA8pwVdA9QAPwImEuGuutN+/DuQzX++wgLgk/AvQPKdcP+BOuTXc/7kK58bGOvxuPk7Zzmx1UJhP4BbATaMT9Ip8WZl/JuE4Wm3BddpcBH431a4zFccP9An8LNzPkQWC390/09N563ICNHRy3e4LK+fpO+v1ctnez0ZuNMcZElZ2jMcYYE1WWaIwxxkSVJRpjjDFRZYnGGGNMVFmiMcYYE1WWaIwxxkSVJRrTIW/K25/HOo5wvGl5fy8iu7xpe++JdUzBRKSvF9fNMY5jsogs8qYujuj1DCJyerwd92gQkRO993JaJ+V+LiIbI/ScfxOR2ZHYV6zZyAAmkX0ENzT8p3HjVG3tuHiv9TBQiZv4q9PhQrrodOCHuOmRjQnLEo2JGRHJVNUDx7GLMUCtqv6pi8+bDCSravNxPHciGQM8oqpzYh1IZyLwmTBxyJrO4pSIPC4iFSJykYgsE5F9IjJPRE4IKjPUq85fEW7boMf3iEi1iJzh7fOAt69hIlIsIi+KyF4RWSki57cTzw9EZKdX7kkRyQ1Z309EHvaasRpF5G0ROSOkjIrI10TkV95YVaGjyQaXzRKR33jP2SgiC0Tk4qD1s3FjzuV7+1VvsNOOjuXVIrIcN4TGGSLSX0T+JCIfeMdkjYj8RETSwhzj67zXVy8iW0XkR97AhcHPc423jwPi5uMZEyaWZO/92CxuNsjlIvLxduL9kIisEDdj5D+9Y1wuIrO8z0OFiJzcwTGc5jWVJQO/9l7H40HrP+M9f5OIbBKRb4Vsf5aIzBCR7d7zLRGRG4PW34w3ll3QezA7+DWE7O+oz2tHnwkRucp7jY3e5+B+cWNtBdYPEpFnRKTSO+brReTH7R0Pb5tPeZ/9GhGp9Y7lpDDlPi8iW7zX/Xegf5gyeeJmkt0nIjtE5HvtPGeZiDzlPed+Efm3iIwOKTNYRF72XsdGEflMR68j4cR6TB67tTtW0eO45o4luIH/puPmd1kOh4YOGoobu+iKMNtWBD2+BzeO0VLcIHhXA5uBebixjb4BXAzMxI0TlRW07Ubc4JhzcGNK3Y4bOPPZoDLpuAEKPwA+BVyKm2ipASgNKqfADtzc45fihi5v7/U/6W3/JeAy4HncOFZTvPXjgEe9WM70bukdHMtq7/h9ArgIGAScBPzcOx5Tgdu81/pw0LaBY7wRN4voRcB/e8uuCyo3ETex1LNevN/wjocCNweVu897Hd/HNWU94pW5Icx7vxDXPPgJ3ICHfwMqgM96z7EE12Qo7bzuHO+4qPc6zwRGeOu+6cVxn/eavoNrVvti0PYfA74NXI4bePIHuNGPb/DWF3n71aD3YFy4z2B7n9f2PhPAdUAr8CDus3mH917/PGjbN3CDQl4NTANuBe7v5Ht1N+4zfIF3DP8f7rsxPKjMVV5cD3nv0U9x4/cpQeN7AS9478ttuO/GHFzz7cagMv1w37XF3mu6Ave92wJkemUE9/3ZDHzce8/fw30WZ8f6f1FE/p/FOgC7tfPGuC9qCzAyaNnV3od9jPf4qC9u0LahiUaBqUHLPu8tuzto2Thv2WVByzbiBtzrG7TsRqANGOs9/rT3Dyg41hRgPfBA0DIFFvt47WO9/d8UtCwJN2Piv0NeV7XPY6nAKZ2US/G+6I1AWsgx/r+QskuAp4IeP0PIP33gewQlGu+fzj7ghyH7ehlYHea9HxG07H5vX58KWna5t2xsJ69LOTKB5AB7w8RxL27QxOQw+xDv+DwMvBG0/IuAtnPM/SaaxWGeaxPw55DltwIHgALv8V7gyuP4jiV5r2kVR34P5gP/Cin7R4ISDW4eFwWuDyrTF/dd2Ri07Me4H2/9gpbl4wag/ULI+3hGUJkh3mdg9rG+vni6WdNZfNuoqmuDHgcmZhp0DPtqBuYGPQ5MVvRGmGUDQ7Z9TY+cmOx53D+DwOyFF+J+fW8QkRRxc1WA+4UX2izxTx+xnubt/9nAAlVt8x5PaW+jTmxT1SXBC8T5itc8dQD3C/9JXA2tLGT7V0Mer+DI9+F0YIZ6/yU8z4dscyKQRdDr8jwNjBKR4qBlG1V1fdDjrrxfnTkL6AM8G3i/vPfsDaAE73WJSL645stNuGNzEFcbGNXF5+tM6GdiFO74PxMmvgzccQSX7P9LRG4WkdD3KywRGSsiL4jILlyN6SBuioFR3vpk3DQXL4VsGvpeBj77h+Zt8b4jr4WUu9BbtifodTTgvi+B78bpwC5V/U/QvjZ5ZXoESzTxrS7kceDkdcYx7KvB+2cduq9Dz6GHT46H7r8y+IG6k7V7OdxuXYhrNjkYcrsFGByyr10+Yu0P7FXV/WG2zRKRdB/7CBXueb+Caw57AddccjrwBW9d6DEI914Elykl5DiFeRw4XqGxBB7nBy1r772vC7Osq5+HQu9+OUe+X7O85YH37HFcs+0DuOar03BDyh/L568joccjEN/LIfFtCInvelxT4i+BTd45pAvaexIRycb9YBgMfA04B/ealnL4NRXhajmdvZeluO9UaMeF0HKFXpyh343zgl5HuM9OuH0lLOt1ltgavfu0kOXh5vc+HsG/tBGRTFwzwQ5vUQ3uC39HmG1Du9P6uY5jB9BXRLJCkk0JsF99zOgXRrjnvRZ3runQSVwRGRemnB87CTlOYR7vCFq+O2h5iXffXXPWB57nCsIn4NXi5pr/EK7J7Q+BFaEdIDrQiP/PZeh7E4jvdty5jVAbAFR1G3CzF9PpuKbUGSJSpqq7w2x3Fq62dpGqrgoslCM7tlThmqw6ey93AtlydC+50HI1uFpPuE4KDUH7Ct0usK8e0QPPajSJrRL362hsYIGI9MV9oSLpIm+/AR/B/XMI9Cp6HSjHzTNeEXJrt2dZBxZ4+w/MrIiIiPd43jG9gvAyOToR+psx8GgLgOlenAEfCSnzPu7E87Uhy68D1qhq1TE+d1e9g/sHNiDM+1Whqg245sNkgo6PVyOYHrKvZm9daC1nKzA0ZPlFPuNbjTsRPrSd+I5IIqrapm6q4h/hmiaHtLPfTO8++DWdjTt3FNhXK65JLnTGzdD3coF3f+h4eN+R0Nf4Ou58zvIwr2N10L5KJKiXptcUOLGd15FwrEaTwFS1TUReAr7qtaPXAV8n8r+CDgD/FJEHcM0/DwAvqGrgnNH/AZ8DZosbReADoAD3K3Onqv6yK0+mqitF5K/A78TNYb4O17NnDOFrTcfqNeBOEfkPruPCjbiEeSx+hpt18BkReQx3HuHTwQVUtUZEfgV8X0RacIn6I7iTwTcc4/N2marWibua/9ciMgR4E/ejcxRwnqp+WFXrRWQBcLeI7MF1zvgO7iR2TtDuAjWDL4vIG8Ae7x/oi7jOBY+K61I9AdeU6ie+NhH5OvD/vPf/X7iENhzXIeajuDns/4377K3BJcav42oHK9vZ9bu4Jt8/isj9uNrNPbikFuynwPMi8hCuWXUqrkdccIzLRWQG8JAX4w5cT77Q5t5f4HoNviEiv/Weq8Tb5zxV/SuuiXAp7pzZt3G1wXvpQU1nMe+NYLfwN/z32inBnbjcg+upc3votoTpnYXrDqrAiSHLQ3sobcSdx7gH18yyD/grkBeyXS7wa1y3zWbcL9rngcnt7buT15+Fu0ZjF+4XaAVwSUiZo16X32PpLe8L/BnXvFGD6y59RfBxCXfMO3h/rsUlxUZczes0ju7enIz75R04TiuAG33s+2ZvX8G9/8LGFuZ1hj3uuH+AC3E/JGpxifJrQevLcSfg9+G63n4r9JhzeH757bhkNDsk5vW4f77/AM4OjbejzwSu+/Fc7/n34GoaP8H9QE7H9QRb7e2/2nuOkzo5FpfiapYHcNM4Xw7MBv4WUu6LuM/wflwiuJijuzfnA0958e3CdZ3+OUG9zrxyA7zPWeCzvBF4AjghqEwZ8IoX1yZcF/a/0UN6ndlUzsYYY6LKztEYY4yJKks0xhhjosoSjTHGmKiyRGOMMSaqLNEYY4yJKks0xhhjosoSjTHGmKiyRGOMMSaqLNEYY4yJqv8PETmuCJSZpgcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n = 100\n",
    "\n",
    "training_errors = []\n",
    "d_values = list(range(2,200,5))\n",
    "\n",
    "for d in d_values:\n",
    "    X = np.random.randn(n,d-1) \n",
    "    y = X[:,0] + np.random.randn(n) # y is just the first feature\n",
    "\n",
    "    lr = LinearRegression()\n",
    "    lr.fit(X,y)\n",
    "    training_error = np.sum((lr.predict(X)-y)**2)\n",
    "    training_errors.append(training_error)\n",
    "    \n",
    "plt.plot(d_values, training_errors);\n",
    "plt.xlabel('number of random features added', fontsize=15)\n",
    "plt.ylabel('training error', fontsize=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we get to 0 training error when $d=n=100$ ($\\pm 1$), as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying loss functions\n",
    "\n",
    "![](loss_venn2.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}