machine_learning_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- title: Machine Learning Metrics\n",
    "- author: Fabian Geiger\n",
    "- date: 2018-09-03\n",
    "- category: Machine Learning\n",
    "- tags: Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This blogpost is a growing collection of different metrics, translated from the math notation to code. I try to keep it as concise and efficient as possible.  \n",
    "Don't hesitate to contact me for any mistakes found or suggestions.  \n",
    "Inspiration and reference:  https://github.com/Jam3/math-as-code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-100,100,201)\n",
    "y = x + 2\n",
    "y_scatter = x + np.random.normal(0, 15, 201)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(201, 201, 201)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x), len(y), len(y_scatter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x10c6bb358>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFYVJREFUeJzt3X2wJXV95/H3V0QLE7OjAXEYuRlIYGqltnaAU4QqVzdZUB4qBkiigewKGyyvVEntZtelBLEMq2uVig+VB8vkmlDBBBVTyDBlyA5gbR62asdwhyEwOM4yg6DzsAOII1Y5RRzy3T9OX+i5nPtw7j3dffr0+1U1dfv8us/tL30O8znf7j7zi8xEktRdL2u6AElSswwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjXt50Actx/PHH5/r165suQ5JaZdu2bU9n5glLbdeKIFi/fj2zs7NNlyFJrRIRTyxnO08NSVLHGQSS1HEGgSR1nEEgSR1nEEhSx7XiriFJ6ppN2/dx85Zd7D90mJPWHMd1F2zg0jPXVbIvg0CSxsim7fu4afMjHDr8kxfG9h06zA1fexigkjAYyamhiLglIp6MiB2lsddGxL0R8Wjx8zXFeETE70fE7oh4KCLOGkUNktR2H9r0MP/l9gePCoE5h3/yPDdv2VXJfkd1jeDPgAvnjV0PfCMzTwO+UTwGuAg4rfgzDXx+RDVIUitt2r6Pjf/9Hv5i63dZbBb5/YcOV7L/kQRBZv4d8My84UuAW4vlW4FLS+NfzL6twJqIWDuKOiSpbRbrAuY7ac1xldRQ5TWCEzPzAEBmHoiI1xXj64DvlbbbW4wdKD85IqbpdwxMTU1VWKYk1W/QtYDFHHfsMVx3wYZKamniYnEMGHtJN5SZM8AMQK/XW6xbkqRW+dCmh7ltidNAZa951bH87tvPaOVdQwcjYm3RDawFnizG9wInl7Z7A7C/wjokaSwM2wUE8O/PneJ/XPqvKq2ryi+UbQauKpavAu4qjV9Z3D10LvDDuVNIkjSphrkWAP0u4LO/ubHyEIARdQQR8WXgl4DjI2Iv8LvAx4GvRsS7ge8C7yg2vxu4GNgN/Bj47VHUIEnjaFy7gLKRBEFmXrHAqvMGbJvA+0axX0kaZ+N2LWAhfrNYkkasDV1AmUEgSSPUli6gzCCQpBFoWxdQZhBI0iq1sQsoMwgkaYXa3AWUGQSStAJt7wLKDAJJGsKkdAFlBoEkLdMkdQFlBoEkLWESu4Ayg0CSFjGpXUCZQSBJA0x6F1BmEEjSPF3oAsoMAkkqdKkLKDMIJHXesAEA7e8CygwCSZ027GmgSekCyioNgojYANxeGjoV+DCwBngP8FQx/sHMvLvKWiSprOtdQFmlQZCZu4CNABFxDLAPuJP+rGSfzcxPVbl/SRrELuBodZ4aOg/Yk5lPRESNu5WkPruAweoMgsuBL5ceXxsRVwKzwPsz8wc11iKpY7p2S+gwoj+FcMU7iXgFsB84IzMPRsSJwNNAAh8F1mbm1fOeMw1MA0xNTZ39xBNPVF6npMnT1VtCASJiW2b2ltquro7gIuCBzDwIMPcTICK+AHx9/hMycwaYAej1etWnlaSJYxewPHUFwRWUTgtFxNrMPFA8vAzYUVMdkjqgy13ASlQeBBHxKuCtwHtLw5+MiI30Tw09Pm+dJK2YXcDwKg+CzPwx8LPzxt5V9X4ldYtdwMr5zWJJrWcXsDoGgaTWsgsYDYNAUivZBYyOQSCpVewCRs8gkNQadgHVMAgkjT27gGoZBJLGml1A9QwCSWPJLqA+BoGksWMXUC+DQNLYsAtohkEgaSzYBTTHIJDUKLuA5hkEkhrhtJHjwyCQVDsnjx8vBoGk2tgFjCeDQFIt7ALGVx0zlD0O/Ah4HjiSmb2IeC1wO7Ce/gxl78zMH1Rdi6T62QWMv7o6gl/OzKdLj68HvpGZH4+I64vHH6ipFkk18ZbQdmjq1NAlwC8Vy7cCf4NBIE0MbwltlzqCIIF7IiKBP87MGeDEzDwAkJkHIuJ1NdQhqQZ2Ae1TRxC8KTP3F3/Z3xsR317OkyJiGpgGmJqaqrI+SSNgF9BelQdBZu4vfj4ZEXcC5wAHI2Jt0Q2sBZ4c8LwZYAag1+st98OFpAbYBbRbpUEQET8FvCwzf1Qsvw34CLAZuAr4ePHzrirrkFQNu4DJUHVHcCJwZ0TM7etLmfk/I+J+4KsR8W7gu8A7Kq5D0ojZBUyOSoMgMx8D/vWA8e8D51W5b0nVsAuYPH6zWNKy2QVMJoNA0pLsAiabQSBpUXYBk88gkDSQXUB3GASSXrBp+z5u3rKLfYcOD/U8u4B2MwgkAcOfAgK7gElhEEgdt5J/JhrsAiaJQSB1mF2AwCCQOskuQGUGgdQxdgGazyCQOmKlXcC6Ncdx3QUb7AImmEEgTTjnDNZSDAJpgg17GshTQN1kEEgTyC5AwzAIpAljF6BhGQTShLAL0EpVFgQRcTLwReD1wD8DM5n5exFxE/Ae4Kli0w9m5t1V1SF1gf9CqFajyo7gCPD+zHwgIl4NbIuIe4t1n83MT1W4b6kT/BdCNQqVBUFmHgAOFMs/ioidgB8/pBGxC9Co1HKNICLWA2cC3wTeBFwbEVcCs/S7hh8MeM40MA0wNTVVR5lSK9gFaNQic5gvmq9gBxE/Dfwt8LHM/FpEnAg8DSTwUWBtZl692O/o9Xo5OztbaZ1SG9gFaBgRsS0ze0ttV2lHEBHHAncAt2Xm1wAy82Bp/ReAr1dZgzQJ7AJUpSrvGgrgT4GdmfmZ0vja4voBwGXAjqpqkCaBXYCqVmVH8CbgXcDDEfFgMfZB4IqI2Ej/1NDjwHsrrEFqLbsA1aXKu4b+N/335nx+Z0Bagl2A6uQ3i6UxYhegJhgE0piwC1BTDAKpYXYBappBIDXILkDjwCCQGmAXoHFiEEg1swvQuDEIpJrYBWhcGQRSDewCNM4MAqlCdgFqA4NAqoDTRqpNDAJpxJw8Xm1jEEgjYhegtjIIpBGwC1CbGQTSKtgFaBIYBNIKeUuoJkVjQRARFwK/BxwD/ElmfrypWqRheEuoJk0jQRARxwCfA94K7AXuj4jNmfmtJuqRlssuQJOoqY7gHGB3Zj4GEBFfAS4BDAKNJbsATbKmgmAd8L3S473ALzZUi7QouwBNuqaCYNBcxkf9fxYR08A0wNTUVB01SUexC1BXNBUEe4GTS4/fAOwvb5CZM8AMQK/XW+6HMWkk7ALUJU0Fwf3AaRFxCrAPuBz4rYZqkV5gF6AuaiQIMvNIRFwLbKF/++gtmflIE7VIc+wC1FWNfY8gM+8G7m5q/9IcuwB1nd8sVqfZBUgGgTrKLkB6kUGgzrELkI5mEKgz7AKkwQwCdYJdgLQwg0ATzS5AWppBoIllFyAtj0GgiWMXIA3HINDEcNpIaWUMAk0EJ4+XVs4gUKvZBUirZxCotewCpNEwCNQ6dgHSaBkEahVvCZVGzyBQK3hLqFQdg0Bjzy5AqlYlQRARNwNvB/4J2AP8dmYeioj1wE5gV7Hp1sy8pooa1H52AVI9quoI7gVuKKak/ARwA/CBYt2ezNxY0X41IewCpPpUEgSZeU/p4VbgN6rYjyaPXYBUvzquEVwN3F56fEpEbAeeBT6UmX9fQw1qAbsAqRkrDoKIuA94/YBVN2bmXcU2NwJHgNuKdQeAqcz8fkScDWyKiDMy89kBv38amAaYmppaaZlqAbsAqVkrDoLMPH+x9RFxFfArwHmZmcVzngOeK5a3RcQe4HRgdsDvnwFmAHq93nI/JKpl7AKk5lV119CF9C8O/9vM/HFp/ATgmcx8PiJOBU4DHquiBo03uwBpfFR1jeAPgVcC90YEvHib6FuAj0TEEeB54JrMfKaiGjSm7AKk8VLVXUO/sMD4HcAdVexT488uQBpPfrNYtbALkMaXQaBK2QVI488gUGXsAqR2MAg0cnYBUrsYBBopuwCpfQwCjYRdgNReBoFWxWkjpfYzCLRiTh4vTQaDQEOzC5Ami0GgodgFSJPHINCy2AVIk8sg0JK8JVSabAaBFuQtoVI3GAQayC5A6g6DQC9YyXUAuwCp/QwCrSgAwC5AmhSVBUFE3AS8B3iqGPpgZt5drLsBeDf9Wcr+U2ZuqaoOLW7YU0BgFyBNmqo7gs9m5qfKAxHxRuBy4AzgJOC+iDg9M5+vuBaV2AVImtPEqaFLgK9k5nPAdyJiN3AO8H8aqKWT7AIklVUdBNdGxJXALPD+zPwBsA7YWtpmbzF2lIiYBqYBpqamKi6zG+wCJA2yqiCIiPuA1w9YdSPweeCjQBY/Pw1cTf/D5Xwv+XCamTPADECv1xvmw6sGWEkXYABI3bCqIMjM85ezXUR8Afh68XAvcHJp9RuA/aupQwvzS2GSllLlXUNrM/NA8fAyYEexvBn4UkR8hv7F4tOAf6iqji7zS2GSlqPKawSfjIiN9E/7PA68FyAzH4mIrwLfAo4A7/OOodGyC5A0jMqCIDPftci6jwEfq2rfXWYXIGlYfrN4QtgFSFopg2AC2AVIWg2DoMXsAiSNgkHQQs4WJmmUDIKWcc5gSaNmELSEXYCkqhgELWAXIKlKBsEYswuQVAeDYEx5S6ikuhgEY8ZbQiXVzSAYI3YBkppgEIwBuwBJTTIIGmYXIKlpBkFD7AIkjQuDoAF2AZLGiUFQI7sASeOokiCIiNuBDcXDNcChzNwYEeuBncCuYt3WzLymihrGjV2ApHFVSRBk5m/OLUfEp4EfllbvycyNVex3HNkFSBp3lZ4aiogA3gn8uyr3M67sAiS1QdXXCN4MHMzMR0tjp0TEduBZ4EOZ+feDnhgR08A0wNTUVMVljpZdgKQ2WXEQRMR9wOsHrLoxM+8qlq8AvlxadwCYyszvR8TZwKaIOCMzn53/SzJzBpgB6PV6y/1Q3Ti7AElts+IgyMzzF1sfES8Hfg04u/Sc54DniuVtEbEHOB2YXWkd48IuQFJbVXlq6Hzg25m5d24gIk4AnsnM5yPiVOA04LEKa6iFXYCkNqsyCC7n6NNCAG8BPhIRR4DngWsy85kKa6iUXYCkSVBZEGTmfxwwdgdwR1X7rJNdgKRJ4TeLh2QXIGnSGATL5LSRkiaVQbAMTh4vaZIZBIuwC5DUBQbBAuwCJHWFQTCPXYCkrjEISrwlVFIXGQR4S6ikbut8ENgFSOq6zgaBXYAk9XUyCOwCJOlFnQoCuwBJeqnOBIFdgCQNNvFBYBcgSYub6CCwC5Ckpb1sNU+OiHdExCMR8c8R0Zu37oaI2B0RuyLigtL4hcXY7oi4fjX7X8ym7fuWHQIB/Idzp9j+4bcZApI6Z7UdwQ768xL/cXkwIt5If4ayM4CTgPsi4vRi9eeAtwJ7gfsjYnNmfmuVdbzEzVt2LSsE7AIkdd2qgiAzdwJExPxVlwBfKSar/05E7AbOKdbtzszHiud9pdh25EGw/9DhRdd7LUCS+qq6RrAO2Fp6vLcYA/jevPFfrKKAk9Ycx74FwsAuQJJetOQ1goi4LyJ2DPhzyWJPGzCWi4wP2u90RMxGxOxTTz21VJkvcd0FGzju2GNeUpTXAiTpaEt2BJl5/gp+717g5NLjNwD7i+WFxufvdwaYAej1esu98ecFc3/R37xlF/sPHeakNcdx3QUbDABJmqeqU0ObgS9FxGfoXyw+DfgH+h/KT4uIU4B99C8o/1ZFNXDpmev8i1+SlrCqIIiIy4A/AE4A/ioiHszMCzLzkYj4Kv2LwEeA92Xm88VzrgW2AMcAt2TmI6v6L5AkrUpkDn3WpXa9Xi9nZ2ebLkOSWiUitmVmb6ntVvWFMklS+xkEktRxBoEkdVwrrhFExFPAE6v4FccDT4+onFGyruFY13CsaziTWNfPZeYJS23UiiBYrYiYXc4Fk7pZ13CsazjWNZwu1+WpIUnqOINAkjquK0Ew03QBC7Cu4VjXcKxrOJ2tqxPXCCRJC+tKRyBJWsBEBcE4T51Z2t/tEfFg8efxiHiwGF8fEYdL6/6o6lrm1XVTROwr7f/i0rqBx66mum6OiG9HxEMRcWdErCnGGz1eRQ21vncWqePkiPhfEbGzeP//52J8wde0xtoej4iHi/3PFmOvjYh7I+LR4udraq5pQ+mYPBgRz0bE7zR1vCLiloh4MiJ2lMYGHqPo+/3iPfdQRJw1kiIyc2L+AP8S2AD8DdArjb8R+EfglcApwB76/+jdMcXyqcArim3eWGO9nwY+XCyvB3Y0eOxuAv7bgPGBx67Gut4GvLxY/gTwiTE5Xo2+d+bVshY4q1h+NfB/i9dt4Gtac22PA8fPG/skcH2xfP3ca9rg6/j/gJ9r6ngBbwHOKr+fFzpGwMXAX9P/l5zPBb45ihomqiPIzJ2ZuWvAqhemzszM7wBzU2eeQzF1Zmb+EzA3dWbloj+/5zuBL9exv1VY6NjVIjPvycwjxcOt9OewGAeNvXfmy8wDmflAsfwjYCcvzgg4ji4Bbi2WbwUubbCW84A9mbmaL6yuSmb+HfDMvOGFjtElwBezbyuwJiLWrraGiQqCRazjpVNkrltkvA5vBg5m5qOlsVMiYntE/G1EvLmmOsquLdrNW0rtepPHaL6r6X8amtPk8Rqn4/KCiFgPnAl8sxga9JrWKYF7ImJbREwXYydm5gHohxjwugbqmnM5R38Ya/p4zVnoGFXyvmtdEERDU2dWUOMVHP0GPABMZeaZwH+lP7HPz6y2liHq+jzw88DGopZPzz1twK8a6a1myzleEXEj/bktbiuGKj9eS5U9YKzRW/Ai4qeBO4DfycxnWfg1rdObMvMs4CLgfRHxlgZqGCgiXgH8KvCXxdA4HK+lVPK+q2qGsspkQ1NnDmOpGiPi5cCvAWeXnvMc8FyxvC0i9gCnAyObiGG5xy4ivgB8vXi42LGrpa6IuAr4FeC8LE6U1nG8llD5cRlGRBxLPwRuy8yvAWTmwdL68mtam8zcX/x8MiLupH9K7WBErM3MA8VpjSfrrqtwEfDA3HEah+NVstAxquR917qOYIU2A5dHxCujP03m3NSZ91NMnVl8Ori82LZq5wPfzsy9cwMRcUJEHFMsn1rU+FgNtcztv3ye8TJg7g6GhY5dXXVdCHwA+NXM/HFpvNHjRXPvnZcorjf9KbAzMz9TGl/oNa2rrp+KiFfPLdO/8L+D/nG6qtjsKuCuOusqOaorb/p4zbPQMdoMXFncPXQu8MO5U0irUvcV8oqvvl9GPzGfAw4CW0rrbqR/l8cu4KLS+MX077LYA9xYU51/Blwzb+zXgUfo333yAPD2mo/dnwMPAw8Vb7a1Sx27muraTf+c6IPFnz8ah+PV1HtngTr+Df3TAw+VjtPFi72mNdV1avH6/GPxWt1YjP8s8A3g0eLnaxs4Zq8Cvg/8i9JYI8eLfhgdAH5S/P317oWOEf1TQ58r3nMPU7o7cjV//GaxJHVcV04NSZIWYBBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR13P8HgTPfiqSK4+IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metrics which are chosen to evaluate ones machine learning algorithm vary wildly, depending on the usecase (or the Kaggle competition). A few of them are shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**RMSE**  \n",
    "Arguably on of the most frequently used measure of the differences between values predicted by a model or an estimator and the values observed. The RMSE, also called RMSD, is calculated by taking the square root of the mean of the squared delta between `y_pred` and `y_true`.\n",
    "\n",
    "$${\\displaystyle \\operatorname {RMSE} ={\\sqrt {\\frac {\\sum _{t=1}^{T}({\\hat {y}}_{t}-y_{t})^{2}}{T}}}}$$\n",
    "\n",
    "where $y$ is the actual value and $\\hat y$ is the predicted value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse(y_pred, y): return math.sqrt(((y_pred-y)**2).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**SMAPE**  \n",
    "Symmetric mean absolute percentage error (SMAPE or sMAPE) is an accuracy measure based on percentage (or relative) errors. It is usually defined as follows:\n",
    "\n",
    "$${\\displaystyle {\\text{SMAPE}}={\\frac {100\\%}{n}}\\sum _{t=1}^{n}{\\frac {\\left|F_{t}-A_{t}\\right|}{(|A_{t}|+|F_{t}|)/2}}}$$\n",
    "\n",
    "where $A_t$ is the actual value and $F_t$ is the predicted value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def SMAPE(y_pred, y): return ((200*np.abs(y_pred-y)/(y+y_pred)).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Confusion Matrix**\n",
    "<td>\n",
    "<table class=\"wikitable\" style=\"border:none; float:left; margin-top:0;\">\n",
    "<tbody><tr>\n",
    "<th style=\"background:white; border:none;\" colspan=\"2\" rowspan=\"2\">\n",
    "</th>\n",
    "<th colspan=\"3\" style=\"background:none;\">Actual class\n",
    "</th></tr>\n",
    "<tr>\n",
    "<th>Cat\n",
    "</th>\n",
    "<th>Dog\n",
    "</th>\n",
    "<th>Rabbit\n",
    "</th></tr>\n",
    "<tr>\n",
    "<th rowspan=\"3\" style=\"height:6em;\"><div style=\"display: inline-block; -ms-transform: rotate(-90deg); -webkit-transform: rotate(-90deg); transform: rotate(-90deg);;\">Predicted<br> class</div>\n",
    "</th>\n",
    "<th>Cat\n",
    "</th>\n",
    "<td><b>5</b>\n",
    "</td>\n",
    "<td>2\n",
    "</td>\n",
    "<td>0\n",
    "</td></tr>\n",
    "<tr>\n",
    "<th>Dog\n",
    "</th>\n",
    "<td>3\n",
    "</td>\n",
    "<td><b>3</b>\n",
    "</td>\n",
    "<td>2\n",
    "</td></tr>\n",
    "<tr>\n",
    "<th>Rabbit\n",
    "</th>\n",
    "<td>0\n",
    "</td>\n",
    "<td>1\n",
    "</td>\n",
    "<td><b>11</b>\n",
    "</td></tr></tbody></table>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Accuracy**  \n",
    "The accuracy of a test is its ability to differentiate the cases correctly. To estimate the accuracy of a test, we should calculate the proportion of true positive and true negative in all evaluated cases.\n",
    "\n",
    "$${\\displaystyle \\mathrm {ACC} ={\\frac {\\mathrm {TP} +\\mathrm {TN} }{P+N}}={\\frac {\\mathrm {TP} +\\mathrm {TN} }{\\mathrm {TP} +\\mathrm {TN} +\\mathrm {FP} +\\mathrm {FN} }}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def acc(y_pred, y): return np.mean(y_pred == y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Precision**  \n",
    "Precision is the fraction of relevant instances among the retrieved instances.\n",
    "\n",
    "$${\\displaystyle {\\text{Precision}}={\\frac {tp}{tp+fp}}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Recall**\n",
    "Recall is the fraction of relevant instances that have been retrieved over the total amount of relevant instances.\n",
    "\n",
    "$${\\displaystyle {\\text{Recall}}={\\frac {tp}{tp+fn}}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**F1-Measure**\n",
    "The F1-Measure is a measure that combines precision and recall by taking the harmonic mean of precision and recall:\n",
    "\n",
    "$${\\displaystyle F=2\\cdot {\\frac {\\mathrm {precision} \\cdot \\mathrm {recall} }{\\mathrm {precision} +\\mathrm {recall} }}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In mathematics, statistics, and computer science, particularly in the fields of machine learning and inverse problems, regularization is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting [(source)](https://en.wikipedia.org/wiki/Regularization_(mathematics))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**L1 Regularization or Lasso Regression**  \n",
    "\n",
    "$$\\sum _{i=1}^{n}(y_{i}-\\sum _{j=1}^{p}x_{ij}\\beta_{j})^{2}+\\lambda\\sum _{j=1}^{p}|\\beta_{j}|$$  \n",
    "\n",
    "where $\\lambda\\sum _{j=1}^{p}|\\beta_{j}|$ is the L1 regularization term. Lasso Regression (Least Absolute Shrinkage and Selection Operator) adds “absolute value of magnitude” of coefficient as penalty term to the loss function. If $\\lambda$ is zero the term is a simple ordinary least squares term whereas very large values will make coefficients zero hence it will under-fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**L2 Regularization or Ridge Regression**\n",
    "\n",
    "$$\\sum _{i=1}^{n}(y_{i}-\\sum _{j=1}^{p}x_{ij}\\beta_{j})^{2}+\\lambda\\sum _{j=1}^{p}\\beta_{j}^{2}$$  \n",
    "\n",
    "where $\\lambda\\sum _{j=1}^{p}\\beta_{j}^{2}$ is the L2 regularization term. Ridge Regression adds the “squared magnitude” of coefficient as penalty term to the loss function. Here, if $\\lambda$ is zero, the term is a simple ordinary least squares term. However, if $\\lambda$ is very large, the penalty term will add too much weight and it will lead to under-fitting.  \n",
    "Having said that it’s important how lambda is chosen. This technique works very well to avoid over-fitting issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Binary Crossentropy**\n",
    "\n",
    "$$−(ylog(p)+(1−y)log(1−p))$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_loss(y, p): return np.mean(-(y * np.log(p) +  (1-y)*np.log(1-p)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activation functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Softmax**\n",
    "\n",
    "$${\\displaystyle P(y=j\\mid \\mathbf {x} )={\\frac {e^{\\mathbf {x} ^{\\mathsf {T}}\\mathbf {w} _{j}}}{\\sum _{k=1}^{K}e^{\\mathbf {x} ^{\\mathsf {T}}\\mathbf {w} _{k}}}}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sigmoid**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x): return 1 / (1 + math.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10c897fd0>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFjFJREFUeJzt3X+Q3PV93/HnWzp0AiF+GMlAkLAAy6416Q/IDXFx47pjnCKmQbXzY2DSidt6QjsT2nqcdEqGDvXQv2xPmplMqF01cZ14EhOS1I2aKIOTGDcZUhyEDRiBqQ+ZFFkCZExPvkO3d3v77h/7PXm97OlWut3bu8/3+Zi50e73+73dt777ude977vf/X4iM5EklWXDqAuQJA2e4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kq0Nionnjbtm25a9euUT29JK1Ljz/++Lczc/ty240s3Hft2sWhQ4dG9fSStC5FxF/3s52HZSSpQIa7JBXIcJekAhnuklQgw12SCrRsuEfEpyPilYh4eon1ERG/EhGTEfFURNww+DIlSWejn879M8AtZ1i/F9hdfd0JfHLlZUmSVmLZ89wz888jYtcZNtkH/Ga25+t7NCIuiYgrM/P4gGqU1oRTcws8c3yKo6+dYrrRZKbRpDHfYnGiysUZK7NaksnpdTidpTq89x2X87d3XjLU5xjEh5iuAl7suH+0WvaGcI+IO2l391x99dUDeGpp+KZOzXP/w5N85i9fYK7ZOufHiRhgUVrX3nzR5nUR7r2GbM82JTP3A/sBJiYmbGW05s01W/z0rz3K4WMnef/1V7H3B6/kmm1b2Lp5jC3jY4yPbSCAqJJ78YdhMcjDRNeIDCLcjwI7O+7vAI4N4HGlkfvVL36Dp791kk/+9A3s/ZtXjrocqW+DOBXyAPAz1Vkz7wSmPN6uEhw5Mc39X3qeD9xwlcGudWfZzj0iPge8B9gWEUeB/wCcB5CZnwIOArcCk8DrwD8bVrHSavri119hoZX8wo++fdSlSGetn7Nl7lhmfQI/N7CKpDXikclvc+22LfzAJeePuhTprPkJVamH+YUWf/XN73DTWy8bdSnSOTHcpR6efPH/MTO3wE3XbRt1KdI5MdylHh6ZfJUI+LvX2rlrfTLcpR4ee+E7vOOKi7h0y6ZRlyKdE8Nd6uHoa69z7fYtoy5DOmeGu9QlMzk+NcsVF20edSnSOTPcpS5Tp+ZpNFtccbHhrvXLcJe6HJ+aBTDcta4Z7lKXl062w/1Kw13rmOEudXm56twv95i71jHDXepyfGqWCHjzVsNd65fhLnV5+eQsl20ZZ9OYPx5avxy9UpfjU7Meb9e6Z7hLXV4+Oevxdq17hrvUxc5dJTDcpQ6n5haYOjXvOe5a9wx3qcPiOe5eekDrneEudXjJT6eqEIa71GHq1DwAF59/3ogrkVbGcJc6zDSaAGzdvOz0wtKaZrhLHWbm2uG+Zdxw1/pmuEsdpqvO/ULDXeuc4S51mGk02bghGPfSA1rnHMFSh+nZJls2bSQiRl2KtCKGu9RhurHgIRkVwXCXOsw0mr6ZqiIY7lKHmbkmF3oapApguEsdphtND8uoCIa71GGm0WTLJsNd65/hLnWYnvWYu8rQV7hHxC0R8VxETEbE3T3WXx0RD0fEVyPiqYi4dfClSsPXPiyzcdRlSCu2bLhHxEbgfmAvsAe4IyL2dG3274EHM/N64HbgPw+6UGnYMpOZuQU7dxWhn879RmAyM49k5hzwALCva5sELqpuXwwcG1yJ0upoNFsstNKzZVSEfkbxVcCLHfePAj/ctc1HgS9ExL8CtgA3D6Q6aRV5XRmVpJ/OvdfnsLPr/h3AZzJzB3Ar8NmIeMNjR8SdEXEoIg6dOHHi7KuVhmjxcr+eLaMS9BPuR4GdHfd38MbDLh8CHgTIzP8NbAa2dT9QZu7PzInMnNi+ffu5VSwNyXdnvdyvytFPuD8G7I6IayJiE+03TA90bfN/gfcCRMQ7aIe7rbnWlRkPy6ggy4Z7ZjaBu4CHgGdpnxVzOCLui4jbqs1+HvjZiHgS+BzwTzOz+9CNtKZ9b6IOT4XU+tdXi5KZB4GDXcvu7bj9DPCuwZYmra7pxgLgFHsqg59QlSqn31D1sIwKYLhLFcNdJTHcpcrps2U8FVIFMNylykyjyfnnbWTjBqfY0/pnuEuVmTmvCKlyGO5SpT1/qqdBqgyGu1SZaTjFnsphuEuVaWdhUkEMd6ky0/CYu8phuEuVRrPF5vP8kVAZHMlSZXZ+gc1jvqGqMhjuUqXRbDFu565COJKlyuz8AuN27iqE4S5V7NxVEkeyBLRayVyz5TF3FcNwl2h37YCdu4rhSJaARrM9UYedu0phuEvA7Hy7c998nuGuMhjuEt/r3MfH/JFQGRzJEnbuKo/hLmHnrvI4kiXs3FUew12io3P3VEgVwpEs0dG5eyqkCmG4S9i5qzyOZAk7d5XHcJewc1d5HMkSdu4qj+EuYeeu8jiSJb7XufshJpXCkSzR7tw3jW0gIkZdijQQfYV7RNwSEc9FxGRE3L3ENj8VEc9ExOGI+O3BlikNV2O+xWa7dhVkbLkNImIjcD/wPuAo8FhEHMjMZzq22Q38IvCuzHwtIt48rIKlYWg0Fxj30gMqSD+tyo3AZGYeycw54AFgX9c2Pwvcn5mvAWTmK4MtUxqu2fkWm30zVQXpZzRfBbzYcf9otazT24C3RcQjEfFoRNzS64Ei4s6IOBQRh06cOHFuFUtDMDu/wLinQaog/YR7r3eYsuv+GLAbeA9wB/BrEXHJG74pc39mTmTmxPbt28+2VmloGk07d5Wln9F8FNjZcX8HcKzHNn+QmfOZ+U3gOdphL60Ls/MLfoBJRekn3B8DdkfENRGxCbgdONC1zf8A/gFARGyjfZjmyCALlYap0Wz5ASYVZdnRnJlN4C7gIeBZ4MHMPBwR90XEbdVmDwGvRsQzwMPAv83MV4dVtDRodu4qzbKnQgJk5kHgYNeyeztuJ/CR6ktad+zcVRpHs4Sdu8pjuEvYuas8jmYJz3NXeQx3CTt3lcfRrNprtZK5Zstj7iqK4a7am1uoruVu566COJpVe7Pz7VmY7NxVEsNdtddo2rmrPI5m1Z6du0pkuKv27NxVIkezas/OXSUy3FV7s/N27iqPo1m112hWnbtzqKoghrtqb7Fz97CMSmK4q/YWO3cPy6gkjmbVnp27SmS4q/bs3FUiR7Nqz85dJTLcVXt27iqRo1m1d/o89zF/HFQOR7Nqr9FcYNPYBiJi1KVIA2O4q/Ya8y0227WrMI5o1V6jucC4n05VYQx31d7sfIvNvpmqwjiiVXuN5gLjngapwhjuqj07d5XIEa3as3NXiQx31Z6du0rkiFbtzc7buas8hrtqr9G0c1d5+hrREXFLRDwXEZMRcfcZtvuJiMiImBhcidJw2bmrRMuGe0RsBO4H9gJ7gDsiYk+P7bYC/xr48qCLlIbJzl0l6mdE3whMZuaRzJwDHgD29djuPwIfB2YHWJ80dHbuKlE/4X4V8GLH/aPVstMi4npgZ2b+4QBrk1ZFo9nycr8qTj8jutel8vL0yogNwC8DP7/sA0XcGRGHIuLQiRMn+q9SGpJWK5lrtpyoQ8XpJ9yPAjs77u8AjnXc3wr8IPCliHgBeCdwoNebqpm5PzMnMnNi+/bt5161NCBzC9W13O3cVZh+RvRjwO6IuCYiNgG3AwcWV2bmVGZuy8xdmbkLeBS4LTMPDaViaYBm59uzMNm5qzTLhntmNoG7gIeAZ4EHM/NwRNwXEbcNu0BpmBpNO3eVaayfjTLzIHCwa9m9S2z7npWXJa0OO3eVynZFtWbnrlI5olVrdu4qleGuWrNzV6kc0aq10527c6iqMIa7aq0xX3XuY/4oqCyOaNXabNPOXWUy3FVrdu4qlSNatWbnrlIZ7qq1WTt3FcoRrVpr2LmrUIa7as3OXaVyRKvWGs0FNo1tIKLXtAXS+mW4q9Ya8y0227WrQI5q1VqjucC4x9tVIMNdtTY732Kz15VRgRzVqrVGc4FxrwipAhnuqjU7d5XKUa1as3NXqQx31Zqdu0rlqFat2bmrVIa7am12vuWnU1UkR7Vq7fVGkws2jY26DGngDHfV2nSjydbNhrvKY7irtjKTmbkFtox7zF3lMdxVW7PzLRZayZZxO3eVx3BXbU03mgBcaLirQIa7amumCvctvqGqAhnuqq3Fzt3DMiqR4a7aWuzcPVtGJTLcVVszc3buKpfhrtr67uziG6qeCqny9BXuEXFLRDwXEZMRcXeP9R+JiGci4qmI+LOIeMvgS5UGa6axANi5q0zLhntEbATuB/YCe4A7ImJP12ZfBSYy828Bvwd8fNCFSoM24xuqKlg/nfuNwGRmHsnMOeABYF/nBpn5cGa+Xt19FNgx2DKlwZv2VEgVrJ9wvwp4seP+0WrZUj4E/HGvFRFxZ0QciohDJ06c6L9KaQhmGk0u2LSRjRti1KVIA9dPuPca+dlzw4h/AkwAn+i1PjP3Z+ZEZk5s3769/yqlIZiZa3pIRsXqZ2QfBXZ23N8BHOveKCJuBu4B/n5mNgZTnjQ8040FLz2gYvXTuT8G7I6IayJiE3A7cKBzg4i4HvgvwG2Z+crgy5QGb3p23itCqljLhntmNoG7gIeAZ4EHM/NwRNwXEbdVm30CuBD43Yh4IiIOLPFw0pox01jwzVQVq6+RnZkHgYNdy+7tuH3zgOuShm660eTKizePugxpKPyEqmprZq7JhV5XRoUy3FVbMw3PllG5DHfV1nSj6dkyKpbhrlpqLrSYnW/5hqqKZbirlr530TBPhVSZDHfV0vSc86eqbIa7amnxipCeLaNSGe6qJedPVekMd9XS6c7dcFehDHfV0ozXclfhDHfV0tSpeQC2esxdhTLcVUsvTbWvSn35RV5bRmUy3FVLL52c5bItm9g05o+AyuTIVi29NHWKK7wipApmuKuWXjrZ4AoPyahghrtqyc5dpTPcVTuz8wu89vq8nbuKZrirdl4+OQtg566iGe6qnZemDHeVz3BX7bxUde7On6qSGe6qncXO3Q8wqWSGu2rn+NQsF46PsXXzeaMuRRoaw1218/LJWS6/aHzUZUhDZbirdo5PzfpmqopnuKtWMpMXv/M6P3Dx+aMuRRoqw121MvnKNK/OzDGx69JRlyINleGuWnlk8tsA3HTdthFXIg2X4a5aeeT5V9n5pvPZ+aYLRl2KNFSGu2qjudDi0SOv8i67dtWA4a7aOHzsJN+dbXLTWw13la+vcI+IWyLiuYiYjIi7e6wfj4jfqdZ/OSJ2DbpQaaX2/8URztsY3HTdZaMuRRq6ZcM9IjYC9wN7gT3AHRGxp2uzDwGvZeZbgV8GPjboQqWV+J9PHuOPnjrOh29+G9su9ANMKl8/U7/fCExm5hGAiHgA2Ac807HNPuCj1e3fA341IiIzc4C1SmeludBi8sQ0n//Kt/hvf/kC1199Cf/i3deOuixpVfQT7lcBL3bcPwr88FLbZGYzIqaAy4BvD6LITg8+9iL7/+LIkuuX+32y7G+bZTbo57fVSmtY7ldiLvMIy37/Cn/lDvv/136Mlf0f5xdaTJ2ap5UQAR+4fgd37/0bjG30bSbVQz/hHj2Wdf9o9bMNEXEncCfA1Vdf3cdTv9GlWzbx9su3nnmjXtX0v5qIM2+x3Pe3H2PENSz7/Ms8/orrX9nz9/UYZ1g/tmEDl1xwHtdtv5Afesulnvqo2ukn3I8COzvu7wCOLbHN0YgYAy4GvtP9QJm5H9gPMDExcU794/v2XM779lx+Lt8qSbXRz9+ojwG7I+KaiNgE3A4c6NrmAPDB6vZPAF/0eLskjc6ynXt1DP0u4CFgI/DpzDwcEfcBhzLzAPDrwGcjYpJ2x377MIuWJJ1ZP4dlyMyDwMGuZfd23J4FfnKwpUmSzpWnDkhSgQx3SSqQ4S5JBTLcJalAhrskFShGdTp6RJwA/vocv30bQ7i0wQBY19mxrrO3VmuzrrOzkrrekpnbl9toZOG+EhFxKDMnRl1HN+s6O9Z19tZqbdZ1dlajLg/LSFKBDHdJKtB6Dff9oy5gCdZ1dqzr7K3V2qzr7Ay9rnV5zF2SdGbrtXOXJJ3Bmg/3iPjJiDgcEa2ImOha94vVpNzPRcQ/7Fh+xgm9h1Dj70TEE9XXCxHxRLV8V0Sc6lj3qWHX0lXXRyPiWx3Pf2vHup77bpXq+kREfD0inoqIz0fEJdXyke6vqoZVHTtnqGNnRDwcEc9W4//fVMuXfE1XsbYXIuJr1fMfqpa9KSL+JCK+Uf176SrX9PaOffJERJyMiA+PYn9FxKcj4pWIeLpjWc/9E22/Uo23pyLihoEVkplr+gt4B/B24EvARMfyPcCTwDhwDfA87UsSb6xuXwtsqrbZs4r1/hJwb3V7F/D0CPfdR4Ff6LG8575bxbp+FBirbn8M+Nga2V8jHTtdtVwJ3FDd3gr8n+p16/marnJtLwDbupZ9HLi7un334ms6wtfxJeAto9hfwLuBGzrH8lL7B7gV+GPak5u9E/jyoOpY8517Zj6bmc/1WLUPeCAzG5n5TWCS9mTepyf0zsw5YHFC76GL9tx4PwV8bjWebwWW2nerIjO/kJnN6u6jtGf3WgtGNna6ZebxzPxKdfu7wLO05ypeq/YBv1Hd/g3gH4+wlvcCz2fmuX5IckUy889540x0S+2ffcBvZtujwCURceUg6ljz4X4GvSbuvuoMy1fDjwAvZ+Y3OpZdExFfjYj/FRE/skp1dLqr+nPv0x1/Ko9yH3X757Q7l0Wj3F9rab+cFhG7gOuBL1eLer2mqymBL0TE49GeFxng8sw8Du1fTMCbR1DXotv5/gZr1PsLlt4/QxtzayLcI+JPI+LpHl9n6pqWmpS7r8m6h1TjHXz/oDoOXJ2Z1wMfAX47Ii5aaS1nUdcngeuAv1PV8kuL39bjoQZ62lQ/+ysi7gGawG9Vi4a+v5Yru8eykZ5OFhEXAr8PfDgzT7L0a7qa3pWZNwB7gZ+LiHePoIaeoj0V6G3A71aL1sL+OpOhjbm+ZmIatsy8+Ry+7UwTdy83ofdZW67GaE8M/gHghzq+pwE0qtuPR8TzwNuAQyutp9+6Our7r8AfVnf7mfR8qHVFxAeBfwS8N6uDj6uxv5Yx9P1yNiLiPNrB/luZ+d8BMvPljvWdr+mqycxj1b+vRMTnaR/OejkirszM49VhhVdWu67KXuAri/tpLeyvylL7Z2hjbk107ufoAHB7RIxHxDXAbuCv6G9C72G4Gfh6Zh5dXBAR2yNiY3X72qrGI6tQy+Lzdx67ez+w+O79Uvtuteq6Bfh3wG2Z+XrH8pHuL0Y3dt6gev/m14FnM/M/dSxf6jVdrbq2RMTWxdu03xx/mvZ++mC12QeBP1jNujp831/Po95fHZbaPweAn6nOmnknMLV4+GbFVvNd5HN85/n9tH+7NYCXgYc61t1D++yG54C9HctvpX12wfPAPatU52eAf9m17MeBw7TPuvgK8GOrvO8+C3wNeKoaRFcut+9Wqa5J2scZn6i+PrUW9teoxs4Sdfw92n+eP9Wxn24902u6SnVdW70+T1av1T3V8suAPwO+Uf37phHsswuAV4GLO5at+v6i/cvlODBfZdeHlto/tA/L3F+Nt6/RcUbgSr/8hKokFWg9H5aRJC3BcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUD/H3QIC38+F0iOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, [sigmoid(num) for num in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**tanh**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tanh(x): return np.tanh(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10ccc5550>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGwJJREFUeJzt3X+QHOV95/H3R7srYYxtVkhgRT8scZGJSXEn8B7xxRcn5pcFdYdw/COi6sryHS7VXZm75FxOWRRX2EXiK0jK4S5VXGzZViw7PiAmR6G7yIX5GafOhrAQGSGIrEVgsycFFLSyHS3Wzo/v/TE9oneZ2R/TvTPah8+ramq6n366+6veZ/Sd5+nuaUUEZmZmTYt6HYCZmZ1anBjMzGwSJwYzM5vEicHMzCZxYjAzs0mcGMzMbBInBjMzm8SJwczMJnFiMDOzSfp7HUAnli1bFmvXru11GGZmC8oTTzzxDxGxfKZ6CzIxrF27luHh4V6HYWa2oEj60WzqeSjJzMwmcWIwM7NJnBjMzGwSJwYzM5vEicHMzCYpJTFI2iHpZUlPt1kuSX8saUTSU5Iuyi3bIulA9tpSRjxmZta5snoMXwM2TrP8SmB99toK/AmApKXAZ4FfAS4GPitpsKSYzMysA6XcxxAR35W0dpoqm4CvR+M5oo9KOlPSCuA3gPsj4iiApPtpJJg7yojLrBdOVGsMvzDGs4d/SrUe1OpBtRbUIsCP0rWCtvzqWs46Y8m87qNbN7itBF7MzY9mZe3KX0fSVhq9DdasWTM/UZoV9LOfV7jyv/81o2OvtlwudTkgS87VG1YmkxhafRximvLXF0ZsB7YDDA0N+WuXnZK+/v0fMTr2Kl/4yD/jkl86m9MG+li0CPoXLWKRQM4MtgB0KzGMAqtz86uAQ1n5b0wpf6RLMZmV6h9PVPnyXx/k/ect50PvXtXrcMw61q3LVXcBH8uuTnoP8JOIOAzcB1whaTA76XxFVma24Hxr+EWOjVf47cve2etQzAoppccg6Q4a3/yXSRqlcaXRAEBEfBHYDVwFjADjwL/Nlh2V9HvA49mmbm6eiDZbaJ478o8Mnj7AhtVn9joUs0LKuirp2hmWB/DJNst2ADvKiMOsl8bGKwyevrjXYZgV5jufzUpybHyCwTc7MdjC58RgVpKjxysMnj7Q6zDMCnNiMCvJsfEJDyVZEpwYzEpy9LiHkiwNTgxmJXh1osaJat09BkuCE4NZCY6OTwD4HIMlwYnBrARjx7PE4KEkS4ATg1kJxk72GJwYbOFzYjArwdh4BfBQkqXBicGsBB5KspQ4MZiVoDmUdOab3GOwhc+JwawEY8cneOtp/fT3+SNlC59bsVkJxsYrHkayZDgxmJVgzD+HYQlxYjArQSMx+PyCpaGUxCBpo6T9kkYkbWux/DZJe7LXDyUdyy2r5ZbtKiMes24bO+6hJEtH4Qf1SOoDbgcup/EM58cl7YqIZ5p1IuI/5+r/R+DC3CZejYgNReMw6yUPJVlKyugxXAyMRMTBiJgA7gQ2TVP/WuCOEvZrdko4Ua0xPlHzUJIlo4zEsBJ4MTc/mpW9jqR3AOuAh3LFp0kalvSopGtKiMesq34+UQfgtIG+HkdiVo4ynvmsFmXRpu5m4O6IqOXK1kTEIUnnAg9J2hsRz71uJ9JWYCvAmjVrisZsVppKvZEYFvf7Wg5LQxkteRRYnZtfBRxqU3czU4aRIuJQ9n4QeITJ5x/y9bZHxFBEDC1fvrxozGalqdYa34P6FzkxWBrKaMmPA+slrZO0mMZ//q+7ukjSecAg8P1c2aCkJdn0MuC9wDNT1zU7lVVqjR5Df1+rzrPZwlN4KCkiqpKuB+4D+oAdEbFP0s3AcEQ0k8S1wJ0RkR9mehfwJUl1GknqlvzVTGYLQbXeaNIDTgyWiDLOMRARu4HdU8pumjL/uRbrfQ+4oIwYzHrlZI/BQ0mWCLdks4KaiWHAP6BniXBLNiuoefLZQ0mWCicGs4Kq9ebJZ3+cLA1uyWYFVZo9hkXuMVganBjMCjp5H4N7DJYIt2Szgnwfg6XGicGsoGZiWOwegyXCLdmsoOYNbu4xWCqcGMwK8g1ulhq3ZLOCfB+DpcaJwawg38dgqXFLNitowvcxWGKcGMwKqvq3kiwxbslmBb12g5t7DJYGJwazgpqP9nSPwVLhlmxW0GuP9nSPwdJQSmKQtFHSfkkjkra1WP5xSUck7clen8gt2yLpQPbaUkY8Zt3UvI+hz4nBElH4CW6S+oDbgcuBUeBxSbtaPKLzroi4fsq6S4HPAkNAAE9k644VjcusWyq1YKBPSE4MloYyegwXAyMRcTAiJoA7gU2zXPcDwP0RcTRLBvcDG0uIyaxrqrW6zy9YUspozSuBF3Pzo1nZVB+S9JSkuyWtnuO6Zqesaj18fsGSUkZiaPWJiCnz/xtYGxH/FHgA2DmHdRsVpa2ShiUNHzlypONgzcpWcY/BElNGax4FVufmVwGH8hUi4pWIOJHNfhl492zXzW1je0QMRcTQ8uXLSwjbrBzVWvgeBktKGYnhcWC9pHWSFgObgV35CpJW5GavBp7Npu8DrpA0KGkQuCIrM1swKrW6f1nVklL4qqSIqEq6nsZ/6H3AjojYJ+lmYDgidgH/SdLVQBU4Cnw8W/eopN+jkVwAbo6Io0VjMuumSj1Y3O/EYOkonBgAImI3sHtK2U256RuAG9qsuwPYUUYcZr1QrdV98tmS4q85ZgVVauGf3LakuDWbFVSt1/2QHkuKE4NZQdWa72OwtDgxmBU0Uat7KMmS4tZsVlC1VmexE4MlxK3ZrKBq3Te4WVqcGMwKqtTCN7hZUtyazQpq/LqqewyWDicGs4IaQ0n+KFk63JrNCpqo1hnw5aqWECcGs4IaN7j5o2TpcGs2K8g/u22pcWIwK8gP6rHUuDWbFeRHe1pqnBjMCqr4JzEsMW7NZgVEBJVasNjnGCwhpSQGSRsl7Zc0Imlbi+WfkvSMpKckPSjpHbllNUl7steuqeuancpq9QBwj8GSUvgJbpL6gNuBy4FR4HFJuyLimVy1vwWGImJc0n8A/gD4rWzZqxGxoWgcZr1QPZkY3GOwdJTxNediYCQiDkbEBHAnsClfISIejojxbPZRYFUJ+zXruUqtDsCAfyvJElJGa14JvJibH83K2rkO+HZu/jRJw5IelXRNu5Ukbc3qDR85cqRYxGYlqdbcY7D0FB5KAlp9IqJlRenfAEPAr+eK10TEIUnnAg9J2hsRz71ugxHbge0AQ0NDLbdv1m3NHoPPMVhKymjNo8Dq3Pwq4NDUSpIuA24Ero6IE83yiDiUvR8EHgEuLCEms66oZOcYfFWSpaSMxPA4sF7SOkmLgc3ApKuLJF0IfIlGUng5Vz4oaUk2vQx4L5A/aW12Sqs2eww+x2AJKTyUFBFVSdcD9wF9wI6I2CfpZmA4InYBfwicAXxLEsCPI+Jq4F3AlyTVaSSpW6ZczWR2Sqv4HIMlqIxzDETEbmD3lLKbctOXtVnve8AFZcRg1gvVenZVks8xWELcms0KOHlVkn8ryRLixGBWwETNPQZLj1uzWQHNHoMTg6XErdmsgJNXJfnksyXEicGsgOZ9DANODJYQJwazAnwfg6XIrdmsgIqHkixBTgxmBVR88tkS5NZsVoBvcLMUuTWbFVDxDW6WICcGswJ8H4OlyK3ZrIDmUJJPPltKnBjMCpio+tGelh63ZrMCqs0b3PrdY7B0ODGYFeAb3CxFpbRmSRsl7Zc0Imlbi+VLJN2VLX9M0trcshuy8v2SPlBGPGbd8tp9DO4xWDoKJwZJfcDtwJXA+cC1ks6fUu06YCwifhG4Dbg1W/d8Go8C/WVgI/A/su2ZLQjVep2+RSJ7MqFZEsroMVwMjETEwYiYAO4ENk2pswnYmU3fDVyqxidpE3BnRJyIiOeBkWx7ZgtCtRa+h8GSU0ZiWAm8mJsfzcpa1omIKvAT4KxZrmt2ypqo1X0PgyWnjBbd6utSzLLObNZtbEDaKmlY0vCRI0fmGKLZ/KjWwucXLDllJIZRYHVufhVwqF0dSf3A24Cjs1wXgIjYHhFDETG0fPnyEsI2K65ar9PvHoMlpowW/TiwXtI6SYtpnEzeNaXOLmBLNv1h4KGIiKx8c3bV0jpgPfA3JcRk1hWVWjDgcwyWmP6iG4iIqqTrgfuAPmBHROyTdDMwHBG7gK8C35A0QqOnsDlbd5+kPweeAarAJyOiVjQms26p1txjsPQUTgwAEbEb2D2l7Kbc9M+Bj7RZ9/PA58uIw6zbKvXw7yRZcvxVx6yASrXu30my5LhFmxVQrYd/J8mS48RgVkClVvfvJFly3KLNCvB9DJYiJwazAqp19xgsPW7RZgVM1HxVkqXHicGsgKp/K8kS5BZtVoDPMViKnBjMCqj4t5IsQW7RZgVU/VtJliAnBrMC/FtJliK3aLMCJnyOwRLkxGBWQLXuq5IsPW7RZgU0nvnsj5GlxS3arIBKre6hJEuOE4NZAVU/j8ESVCgxSFoq6X5JB7L3wRZ1Nkj6vqR9kp6S9Fu5ZV+T9LykPdlrQ5F4zLqpXg9qdQ8lWXqKtuhtwIMRsR54MJufahz4WET8MrAR+G+Szswt/92I2JC99hSMx6xrKvU6gIeSLDlFE8MmYGc2vRO4ZmqFiPhhRBzIpg8BLwPLC+7XrOeqtQDwVUmWnKIt+pyIOAyQvZ89XWVJFwOLgedyxZ/Phphuk7SkYDxmXdNMDL7BzVLTP1MFSQ8Ab2+x6Ma57EjSCuAbwJaIqGfFNwB/TyNZbAc+A9zcZv2twFaANWvWzGXXZvPCQ0mWqhkTQ0Rc1m6ZpJckrYiIw9l//C+3qfdW4C+B/xIRj+a2fTibPCHpT4FPTxPHdhrJg6GhoZgpbrP5drLH4JPPlpiiLXoXsCWb3gLcO7WCpMXAPcDXI+JbU5atyN5F4/zE0wXjMeuaSq3RY/DlqpaaoonhFuBySQeAy7N5JA1J+kpW56PA+4CPt7gs9ZuS9gJ7gWXA7xeMx6xrmonBQ0mWmhmHkqYTEa8Al7YoHwY+kU3/GfBnbda/pMj+zXqpWvdVSZYmt2izDp0cSvI5BkuMW7RZh167j8FDSZYWJwazDlXrzZPP/hhZWtyizTo0Uc16DH60pyXGicGsQ+4xWKrcos065HMMlionBrMOvXYfgz9Glha3aLMONe9j8J3PlhonBrMO+T4GS5VbtFmHKj7HYIlyYjDrUNXnGCxRbtFmHar4HIMlyonBrEMneww+x2CJcYs269Brj/Z0j8HS4sRg1qHXHu3pj5GlxS3arEOVavPRnu4xWFoKJQZJSyXdL+lA9j7Ypl4t9/S2XbnydZIey9a/K3sMqNmCUK3XkaDPicESU7THsA14MCLWAw9m8628GhEbstfVufJbgduy9ceA6wrGY9Y1lVowsGgRjUeWm6WjaGLYBOzMpncC18x2RTU+TZcAd3eyvlmvVWt1n3i2JBVNDOdExGGA7P3sNvVOkzQs6VFJzf/8zwKORUQ1mx8FVrbbkaSt2TaGjxw5UjBss+Kq9fD5BUtS/0wVJD0AvL3FohvnsJ81EXFI0rnAQ5L2Aj9tUS/abSAitgPbAYaGhtrWM+uWiVrdVyRZkmZMDBFxWbtlkl6StCIiDktaAbzcZhuHsveDkh4BLgT+AjhTUn/Wa1gFHOrg32DWEx5KslQV/bqzC9iSTW8B7p1aQdKgpCXZ9DLgvcAzERHAw8CHp1vf7FRVrYV7DJakoq36FuBySQeAy7N5JA1J+kpW513AsKQf0EgEt0TEM9myzwCfkjRC45zDVwvGY9Y1lboTg6VpxqGk6UTEK8ClLcqHgU9k098DLmiz/kHg4iIxmPVKtVb3yWdLkr/umHWoUgv63WOwBLlVm3WoUqv7IT2WJCcGsw5V675c1dLkVm3WoUrNN7hZmpwYzDpU9Q1ulii3arMOVevhG9wsSU4MZh1qDCX5I2Tpcas265CvSrJUOTGYdcjnGCxVbtVmHWrc4OYeg6XHicGsQ9V6nQGfY7AEuVWbdajqHoMlyonBrEMVn2OwRLlVm3XIdz5bqpwYzDpUrdcZ6PdHyNLjVm3WgYigUgsG3GOwBBVKDJKWSrpf0oHsfbBFnfdL2pN7/VzSNdmyr0l6PrdsQ5F4zLqlVg8AP4/BklS0VW8DHoyI9cCD2fwkEfFwRGyIiA3AJcA48J1cld9tLo+IPQXjMeuK6snE4B6DpadoYtgE7MymdwLXzFD/w8C3I2K84H7NemqiVgfwfQyWpKKt+pyIOAyQvZ89Q/3NwB1Tyj4v6SlJt0la0m5FSVslDUsaPnLkSLGozQqq1txjsHTNmBgkPSDp6RavTXPZkaQVwAXAfbniG4BfAv45sBT4TLv1I2J7RAxFxNDy5cvnsmuz0p2o1gBY0t/X40jMytc/U4WIuKzdMkkvSVoREYez//hfnmZTHwXuiYhKbtuHs8kTkv4U+PQs4zbrqbHjjWY8ePpAjyMxK1/RoaRdwJZsegtw7zR1r2XKMFKWTJAkGucnni4Yj1lXHBufAODM0xf3OBKz8hVNDLcAl0s6AFyezSNpSNJXmpUkrQVWA381Zf1vStoL7AWWAb9fMB6zrjiaJYalb3ZisPTMOJQ0nYh4Bbi0Rfkw8Inc/AvAyhb1Limyf7NeGRv3UJKly9famXVg7LiHkixdTgxmHRgbn+CMJf0s9m8lWYLcqs06MHZ8gsE3exjJ0uTEYNaBsfEKgx5GskQ5MZh14Nj4hBODJcuJwawDR8cnfEWSJcuJwawDx45XGPQ9DJYoJwazOZqo1vnZiaqHkixZTgxmc3Ts1cY9DO4xWKqcGMzmyD+gZ6lzYjCbo7Hsd5I8lGSpcmIwm6Pmz2E4MViqnBjM5ujkD+j5zmdLlBOD2Rx5KMlS58RgNkdjxyd400Afpw34sZ6WpkKJQdJHJO2TVJc0NE29jZL2SxqRtC1Xvk7SY5IOSLpLkr+C2SktInjyx2OsHHxTr0MxmzdFewxPA78JfLddBUl9wO3AlcD5wLWSzs8W3wrcFhHrgTHguoLxmM2r/zvyCk/++BhbfnVtr0MxmzeFEkNEPBsR+2eodjEwEhEHI2ICuBPYlD3n+RLg7qzeThrPfTY7Jf3k1Qp/dP9+VrztND46tKrX4ZjNm0KP9pyllcCLuflR4FeAs4BjEVHNlb/u8Z9luvGevTz2/NG2yyNixm3MWGPmTcxYpYw4ZtpEzCLQGbcxi3/rzPuY/39rYxvTV5r5eMHR4xPU6sF//eAFLOn3+QVL14yJQdIDwNtbLLoxIu6dxT7UoiymKW8Xx1ZgK8CaNWtmsdvX+4Uz38R557xl+kqtoppjlUZnqOg25j+OWexixkqaxVZm+rfMJo6Zt1FCHDMsX3bGEn79ncsZWrt0xn2ZLWQzJoaIuKzgPkaB1bn5VcAh4B+AMyX1Z72GZnm7OLYD2wGGhoY6+q76yff/YiermZm9oXTjctXHgfXZFUiLgc3ArmiMITwMfDirtwWYTQ/EzMzmUdHLVT8oaRT4F8BfSrovK/8FSbsBst7A9cB9wLPAn0fEvmwTnwE+JWmExjmHrxaJx8zMitNsTv6daoaGhmJ4eLjXYZiZLSiSnoiItvecNfnOZzMzm8SJwczMJnFiMDOzSZwYzMxsEicGMzObZEFelSTpCPCjDldfRuPmulON45obxzU3jmtuUo3rHRGxfKZKCzIxFCFpeDaXa3Wb45obxzU3jmtu3uhxeSjJzMwmcWIwM7NJ3oiJYXuvA2jDcc2N45obxzU3b+i43nDnGMzMbHpvxB6DmZlNI+nEIOkjkvZJqksamrLsBkkjkvZL+kCufGNWNiJpWxdivEvSnuz1gqQ9WflaSa/mln1xvmOZEtfnJP2/3P6vyi1reey6FNcfSvo7SU9JukfSmVl5T49XFkNX2840cayW9LCkZ7P2/9tZedu/aRdje0HS3mz/w1nZUkn3SzqQvQ92Oabzcsdkj6SfSvqdXhwvSTskvSzp6VxZy+Ojhj/O2ttTki4qLZCISPYFvAs4D3gEGMqVnw/8AFgCrAOeA/qy13PAucDirM75XYz3C8BN2fRa4OkeHrvPAZ9uUd7y2HUxriuA/mz6VuDWU+R49bTtTIllBXBRNv0W4IfZ363l37TLsb0ALJtS9gfAtmx6W/Nv2sO/498D7+jF8QLeB1yUb8vtjg9wFfBtGg9BfA/wWFlxJN1jiIhnI2J/i0WbgDsj4kREPA+MABdnr5GIOBgRE8CdWd15p8ZzOD8K3NGN/RXQ7th1RUR8J157TvijNJ78dyroWduZKiIOR8ST2fTPaDwHZV6fp17QJmBnNr0TuKaHsVwKPBcRnd5AW0hEfBeY+mD6dsdnE/D1aHiUxhMxV5QRR9KJYRorgRdz86NZWbvybvg14KWIOJArWyfpbyX9laRf61IceddnXdQdue59L4/RVP+Oxjempl4er1PpuJwkaS1wIfBYVtTqb9pNAXxH0hNqPMcd4JyIOAyNpAac3YO4mjYz+ctZr48XtD8+89bmFnxikPSApKdbvKb7ttbqse8xTXk3YryWyQ3yMLAmIi4EPgX8T0lvLRrLHOL6E+CfABuyWL7QXK3Fpkq9tG02x0vSjUAV+GZWNO/Ha6awW5T19JI/SWcAfwH8TkT8lPZ/0256b0RcBFwJfFLS+3oQQ0tqPHr4auBbWdGpcLymM29trr+MjfRSRFzWwWqjwOrc/CrgUDbdrrxjM8UoqR/4TeDduXVOACey6SckPQe8Eyjt0XWzPXaSvgz8n2x2umPXlbgkbQH+FXBpZIOt3TheM5j34zIXkgZoJIVvRsT/AoiIl3LL83/TromIQ9n7y5LuoTEE95KkFRFxOBsKebnbcWWuBJ5sHqdT4Xhl2h2feWtzC77H0KFdwGZJSyStA9YDfwM8DqyXtC779rA5qzvfLgP+LiJGmwWSlkvqy6bPzWI82IVYmvvPj1V+EGheJdHu2HUrro00nhV+dUSM58p7erzoXdt5nex81VeBZyPij3Ll7f6m3YrrzZLe0pymcSHB0zSO05as2hbg3m7GlTOp197r45XT7vjsAj6WXZ30HuAnzSGnwrp5xr3bLxp/zFEa3yRfAu7LLbuRxlUk+4Erc+VX0biK4zngxi7F+TXg308p+xCwj8bVLU8C/7rLx+4bwF7gqawBrpjp2HUprhEa46p7stcXT4Xj1au20yaOf0ljSOGp3HG6arq/aZfiOjf7+/wg+1vdmJWfBTwIHMjel/bgmJ0OvAK8LVfW9eNFIzEdBirZ/13XtTs+NIaSbs/a215yV14WffnOZzMzm+SNOpRkZmZtODGYmdkkTgxmZjaJE4OZmU3ixGBmZpM4MZiZ2SRODGZmNokTg5mZTfL/ARPUcgQQbHEdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, tanh(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ReLU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(x): return max(0,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10cdbebe0>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHSxJREFUeJzt3Xl4VOX5//H3LYssshMW2QLK7gIhAmqrVagL9SvSuqCi1NIvi1K11lYt31Z/bW1rq3WpCtJqRYKAC4ot7ohbrShZWMO+b0nYEQIhyfP7Yw5ekSaQWc/Myed1Xblm5pmZnJtn5rrn8JzJ55hzDhERCa6T/C5ARETiS41eRCTg1OhFRAJOjV5EJODU6EVEAk6NXkQk4NToRUQCTo1eRCTg1OhFRAKutt8FALRs2dKlp6f7XYaISErJzs7e4ZxLO9HjkqLRp6ens2DBAr/LEBFJKWa2oTqP09KNiEjAqdGLiAScGr2ISMCp0YuIBJwavYhIwKnRi4gEnBq9iEjAqdGLiPigvNzx4JxlbNp1MO7bOmGjN7PnzKzQzJZUGGtuZu+Z2Srvspk3bmb2hJmtNrNFZpYRz+JFRFLV43NX8bdP1vHJqh1x31Z19uifBy47ZuxeYK5zrisw17sNcDnQ1fsZDUyMTZkiIsHxwfICHp+7ih9ktOf6/h3ivr0TNnrn3MfArmOGhwJTvOtTgKsqjL/gQj4HmppZ21gVKyKS6jbsPMCdM/LofWpjHhx2BmYW921Gukbf2jm3DcC7bOWNtwM2VXjcZm9MRKTGKy4pY8zUbMyMSSP6Ua9OrYRsN9YHYyv7aHKVPtBstJktMLMFRUVFMS5DRCS5OOe4b9YiVhTs54nr+9KheYOEbTvSRl9wdEnGuyz0xjcDFRec2gNbK/sFzrnJzrlM51xmWtoJUzZFRFLalM/W83reVu4a3I0LuyW250Xa6N8ARnrXRwKzK4zf7H37ZiCw9+gSj4hITfXl+l38bk4+g3u24raLTk/49k+YR29m04HvAC3NbDNwP/BH4CUzGwVsBK7xHv4mMARYDRwEbolDzSIiKaNw3yFunZZD+2b1eeTaPpx0UvwPvh7rhI3eOXd9FXcNquSxDrgt2qJERILgSFk5t72Yw1eHSskaNYAm9ev4UkdSnGFKRCSIHpyTz5frd/PE9X3p3qaRb3UoAkFEJA5m523h+c/W86PzO3Pl2af6WosavYhIjOVv28c9ry6if+fm3Dekh9/lqNGLiMTS3uIjjM3Kpkn9Ojx5Q1/q1PK/zWqNXkQkRsrLHXfNzGPrnmJmjB5Iq0b1/C4J0B69iEjM/PWD1cxdXsivruhFv07N/S7na2r0IiIxMG95IY/NXcn3+7bjpoGd/C7nG9ToRUSitGHnAe6YkUuPNo15cNiZCUmkDIcavYhIFIpLyhiblYOZ8cyIftSvm5hEynDoYKyISIScc/zytcUs376Pf/zwHDq2SFwiZTi0Ry8iEqEX/rOB13K38NPB3fhO91YnfoJP1OhFRCKwYP0ufvuvZQzu2YrxPiRShkONXkQkTIX7/U+kDIfW6EVEwnCkrJzx03LZf6iUF0b19y2RMhxq9CIiYfj9m/l8sX4Xjw/vQ482jf0up1q0dCMiUk2z87bwj3+v55bz0xnap53f5VSbGr2ISDUs376Pe19dTP/05vxySE+/ywmLGr2IyAnsLT7CmKnZNKpXmydvTI5EynBojV5E5DiOJlJu2V3MzDHJk0gZjtT6WBIRSbAn5yVnImU41OhFRKowb0Uhj76/kmF923HzucmVSBkONXoRkUps3HmQO2fk0aNNY36fhImU4VCjFxE5RiiRMhvnHJNGZCRlImU4dDBWRKQC5xwTXl9M/vZ9PDfyHDq1aOh3SVHTHr2ISAVZn29gVs4W7hjUlYt6JG8iZTjU6EVEPNkbdvObfy3j4h6tuP3irn6XEzNq9CIiHE2kzKZtk/o8mgKJlOHQGr2I1HhHysoZ/2Iue4uP8Nqt/WnSIPkTKcOhRi8iNd4f31rOF+t28dh1fejZNjUSKcOhpRsRqdHeWLiVZz9dxw/PS+eqvqmTSBmOqBq9mf3UzJaa2RIzm25m9cyss5nNN7NVZjbTzOrGqlgRkVhasX0/97yyiMxOzVIukTIcETd6M2sH3A5kOufOAGoBw4GHgEedc12B3cCoWBQqIhJL+w4dYWxWNqfUq83TN2ZQt3ZwFzii/ZfVBuqbWW2gAbANuBh4xbt/CnBVlNsQEYmpUCLlQjbtOsjTN2bQqnHqJVKGI+JG75zbAjwMbCTU4PcC2cAe51yp97DNQDAXvUQkZT394Wrezy9gwvd6ck56aiZShiOapZtmwFCgM3Aq0BC4vJKHuiqeP9rMFpjZgqKiokjLEBEJy0cri3jkvZUM7XMqPzwv3e9yEiKapZvBwDrnXJFz7ggwCzgPaOot5QC0B7ZW9mTn3GTnXKZzLjMtLS2KMkREqmfTroPcMSOX7q0b8Yfvp3YiZTiiafQbgYFm1sBCszUIWAbMA672HjMSmB1diSIi0Tt0JJRIWVbumDSiHw3q1pw/I4pmjX4+oYOuOcBi73dNBu4B7jKz1UAL4NkY1CkiEjHnHP/3+hKWbt3HY9f1Ib1l6idShiOqjzTn3P3A/ccMrwX6R/N7RURiadr8jbySvZnbB3VlUM/WfpeTcMH94qiICJCzcTf/759Luah7GncOCk4iZTjU6EUksIr2H+bWrBzaNqnPY9f1DVQiZThqztEIEalRSsvKGf9iDnuKS5g17vzAJVKGQ41eRALpj28tZ/66XTx63dn0OjV4iZTh0NKNiATOPxdu5e+frmPkuZ0Y1re93+X4To1eRAJlZcF+7nl1Ef06NWPC93r5XU5SUKMXkcDYd+gIY6Zm0/Dk4CdShkOzICKBUF7u+NlLoUTKp27IoHXAEynDoUYvIoEw8aM1vLesgF8O6Un/zsFPpAyHGr2IpLyPVxbx8LsrGNrnVG45P93vcpKOGr2IpLRNuw5yew1MpAyHGr2IpKxDR8oYN61mJlKGQ7MiIinJOcevXl/Cki37eHZkZo1LpAyH9uhFJCW9+MVGXs7ezO0Xn14jEynDoUYvIiknd+NuHnhjKRd2S+OOwd38LifpqdGLSErZ8dVhxmXl0KZJPR4f3odaNTSRMhxaoxeRlHE0kXL3wRJeHXceTRvU9buklKBGLyIp40/vrODztbt45JqzOaNdE7/LSRlauhGRlDBn0TYmf7yWmwZ24gf9lEgZDjV6EUl6qwr28/NXFpLRsSm/ukKJlOFSoxeRpLbfS6RsULcWT9/YT4mUEdAavYgkLeccd7+8kA27DjLtxwNo00SJlJHQR6OIJK2JH63hnaUF3Hd5DwZ2aeF3OSlLjV5EktKnq3bw8DsruOKstoz6Vme/y0lpavQiknQ27z7IT6bncHqrU3joB2cpkTJKavQiklQOHSljXFYOpWWOZ27KpOHJOpQYLc2giCSV+2cvZfGWvfzt5kw6K5EyJrRHLyJJY/oXG5m5YBPjLzqd7/ZSImWsqNGLSFLI27SH+2cv5YJuafz0u0qkjCU1ehHx3c6vDnNrVjatGp/M49cpkTLWomr0ZtbUzF4xs+Vmlm9m55pZczN7z8xWeZfNYlWsiARPaVk5P5mey84DJUwa0Y9mDZVIGWvR7tE/DrztnOsBnA3kA/cCc51zXYG53m0RkUr9+d0VfLZmJw8OO1OJlHEScaM3s8bABcCzAM65EufcHmAoMMV72BTgqmiLFJFgemvxNp75aC0jBnbkaiVSxk00e/RdgCLgH2aWa2Z/N7OGQGvn3DYA77JVDOoUkYBZXbifu19eSN+OTfn1Fb39LifQomn0tYEMYKJzri9wgDCWacxstJktMLMFRUVFUZQhIqnmq8OljJmaTf26tZioRMq4i2Z2NwObnXPzvduvEGr8BWbWFsC7LKzsyc65yc65TOdcZlpaWhRliEgqcc7x85cXsn7nQZ68IUOJlAkQcaN3zm0HNplZd29oELAMeAMY6Y2NBGZHVaGIBMozH6/lrSXblUiZQNFGIPwEmGZmdYG1wC2EPjxeMrNRwEbgmii3ISIB8e/VO/jT28uVSJlgUTV651wekFnJXYOi+b0iEjxb9hTzk+m5SqT0gY6AiEjchRIpszlSWs6kEf2USJlgmm0RibsH3ljKos17mXxTP7qkneJ3OTWO9uhFJK5mfLGRGV9u4raLTuOS3m38LqdGUqMXkbhZuGkPv569lG93bcld3+1+4idIXKjRi0hc7PzqMOOysklrdDJPDO+rREofaY1eRGKutKyc22fksuNACbPGnadESp9pj15EYu7hd1fy79U7+d1VZyiRMgmo0YtITL29ZBuTPlrDDQM6cm1mB7/LEdToRSSGVhd+xc9eWkifDk25/396+V2OeNToRSQmQomUC6hXpxYTR2Rwcu1afpckHh2MFZGoHU2kXLfjAFk/HkDbJvX9Lkkq0B69iERtspdIee/lPTjvtJZ+lyPHUKMXkah8tnoHD729nO+d2Zb//XYXv8uRSqjRi0jEtu4pZvz0XLqkncJDVyuRMlmp0YtIRA6XljFuWg4lpeU8c1M/TlEiZdLSKyMiEXngjWUs3LSHSSP6cZoSKZOa9uhFJGwvfbmJ6V9sZNx3TuOyM5RImezU6EUkLIs27+H/Zi/hW6e35O5LlEiZCtToRaTadh0oYVxWDmmnnMwT1yuRMlVojV5EqqWs3HH79FyKvjrMK2PPpbkSKVOG9uhFpFoeeXcFn67ewW+H9uas9k39LkfCoEYvIif0ztLtPP3hGq7v34HrzunodzkSJjV6ETmuNUWhRMqz2zfhgSt7+12ORECNXkSqdOBwKWOnZnNy7ZOYOKKfEilTlA7GikilnHP84pVFrCn6iqxRAzi1qRIpU5X26EWkUn//ZB1zFm/jnst6cN7pSqRMZWr0IvJf/rNmJ398ezlDzmzD6AuUSJnq1OhF5Bu27S1m/Is5dG7ZkD9dfbYSKQNAjV5Evna4tIxxWTkcLi1n0gglUgaFXkUR+dpv/rmMvE17mDQig9NbKZEyKKLeozezWmaWa2b/8m53NrP5ZrbKzGaamf5OWiQFvLxgE9Pmb2Tshadx2Rlt/S5HYigWSzd3APkVbj8EPOqc6wrsBkbFYBsiEkdLtuxlwutLOP/0Ftx9STe/y5EYi6rRm1l74HvA373bBlwMvOI9ZApwVTTbEJH42n2ghDFTs2nZsC5PDO9L7Vo6dBc00b6ijwG/AMq92y2APc65Uu/2ZqBdZU80s9FmtsDMFhQVFUVZhohEoqzccfuMXIr2H2biiH60OOVkv0uSOIi40ZvZFUChcy674nAlD3WVPd85N9k5l+mcy0xLS4u0DBGJwl/eW8Enq3bwm6G9ObuDEimDKppv3ZwPXGlmQ4B6QGNCe/hNzay2t1ffHtgafZkiEmvvLt3OU/PWMPycDgzvr0TKIIt4j945d59zrr1zLh0YDnzgnLsRmAdc7T1sJDA76ipFJKbWeomUZymRskaIx1GXe4C7zGw1oTX7Z+OwDRGJ0IHDpYyZmk0dL5GyXh0lUgZdTP5gyjn3IfChd30t0D8Wv1dEYss5xy9eDSVSTh01gHZKpKwR9D0qkRrk2U/XMWfRNn5+aQ/OVyJljaFGL1JD/GfNTv7w1nIu692GsRcqkbImUaMXqQG27S3mJ9NzSG/RgD9fc5YSKWsYhZqJBNzh0jJunZZDcUkZM0YPpFG9On6XJAmmRi8ScL/91zJyN+7h6RszOL1VI7/LER9o6UYkwF7J3kzW5xsZc0EXhpypRMqaSo1eJKCWbNnLhNcWc26XFvz80u5+lyM+UqMXCaDdB0oYm5VN84Z1+esNSqSs6bRGLxIwZeWOO2bmUbjvMC+NPZeWSqSs8dToRQLmsfdX8vHKIn4/7Ez6KJFS0NKNSKC8t6yAv36wmmsz23N9/w5+lyNJQo1eJCDW7TjAXTPzOLNdE34z9Az9UZR8TY1eJAAOlpQydmo2tWsZE0dkKJFSvkFr9CIpzjnHPa8uZlXhfqb8qD/tmzXwuyRJMtqjF0lxz/17Pf9cuJWfXdKdb3fVaTnlv6nRi6Sw+Wt38vs387m0d2tu/c5pfpcjSUqNXiRFFew7xG0v5tKpRQMevuZsHXyVKmmNXiQFlZSWMy4rm4MlpUz/3wFKpJTjUqMXSUG/m7OMnI17eOqGDLq2ViKlHJ+WbkRSzKyczbzwnw2MvqAL3ztLiZRyYmr0Iilk6da93DcrlEj5CyVSSjWp0YukiD0HlUgpkdEavUgKKC933Dkzj4K9h5k5ZqASKSUs2iUQSQGPzV3FhyuKuP/KXvTt2MzvciTFqNGLJLm5+QU8MXcV1/Rrzw39O/pdjqQgNXqRJLZ+xwHunJnHGe0a89urlEgpkVGjF0lSB0tKGZuVTa2TjIk39lMipURMB2NFkpBzjvtmLWZFwX6m3NKfDs2VSCmR0x69SBJ6/rP1zM7byt2XdOeCbkqklOhE3OjNrIOZzTOzfDNbamZ3eOPNzew9M1vlXeorAiJh+GLdLh6ck893e7Vm3IVKpJToRbNHXwr8zDnXExgI3GZmvYB7gbnOua7AXO+2iFRDwb5D3Doth47NG/DItWdz0kk6+CrRi7jRO+e2OedyvOv7gXygHTAUmOI9bApwVbRFitQEJaXl3Doth4MlpUy6qR+NlUgpMRKTNXozSwf6AvOB1s65bRD6MABaxWIbIkH34JxlZG/YzZ+uPotuSqSUGIq60ZvZKcCrwJ3OuX1hPG+0mS0wswVFRUXRliGS0l7L3cyU/2zgx9/qzBVnnep3ORIwUTV6M6tDqMlPc87N8oYLzKytd39boLCy5zrnJjvnMp1zmWlp+laB1FzLtu7jvlmLGdC5Ofde3sPvciSAovnWjQHPAvnOub9UuOsNYKR3fSQwO/LyRIJt78EjjM3Kpmn9ujx5Q4YSKSUuovmDqfOBm4DFZpbnjf0S+CPwkpmNAjYC10RXokgwhRIpc9m2t5iZY84lrZESKSU+Im70zrlPgaq++zUo0t8rUlM8PncV81YU8durziBDiZQSR/p/oogPPlhewONzV/GDjPaMGKBESokvNXqRBNuw8wB3zsij96mNeXCYEikl/tToRRKouKSMMVOzMTMmjVAipSSG0itFEiSUSLmIFQX7eV6JlJJA2qMXSZApn63n9byt3DW4GxcqkVISSI1eJAG+XL+L383JZ3DPVtx20el+lyM1jBq9SJwVeomU7ZvV55Fr+yiRUhJOa/QicXSkrJzbXszhq0OlZI0aQJP6SqSUxFOjF4mjB+fk8+X63TxxfV+6t1EipfhDSzcicTI7bwvPf7aeUd/qzJVnK5FS/KNGLxIH+dv2cc+ri+ivREpJAmr0IjG2tziUSNmkfh2euiGDOkqkFJ9pjV4khsrLHXfNzGPrnmJmjFYipSQH7WqIxNBfP1jN3OWF/PqKXvTrpERKSQ5q9CIxMm9FIY/NXcn3M9oxYmAnv8sR+ZoavUgMbNx5kDum59KzTWN+P+xMJVJKUlGjF4lScUkZY7JCiZTP3KRESkk+OhgrEgXnHBNeW8zy7fv4xw/PUSKlJCXt0YtEYernG5iVu4WfDu7Gd7q38rsckUqp0YtEKHvDLn7zz2UM7tmK8UqklCSmRi8SgcL9hxiXpURKSQ1aoxcJ05GycsZPy2X/oVJeGNVfiZSS9NToRcL0hzeX88X6XTw+vA892jT2uxyRE9LSjUgYZudt4bl/r+OW89MZ2qed3+WIVIsavUg1Ld++j3tfXUz/9Ob8ckhPv8sRqTY1epFq2Ft8hDFTs2lUrzZP3thXiZSSUrRGL3ICRxMpt+wuZuaYgbRqVM/vkkTCot0SkRN4cl4okfJXV/SiX6fmfpcjEjY1epHjmLeikEffX8mwvu24+VwlUkpqUqMXqcLGnQe5c0YePZRIKSkuLo3ezC4zsxVmttrM7o3HNkTiqbikjLFZ2TjnmDQig/p1lUgpqSvmjd7MagFPAZcDvYDrzaxXrLcjEi/OOSa8vpj87ft4fHhfOrVo6HdJIlGJxx59f2C1c26tc64EmAEMjcN2ROIi6/MNzMrZwh2DunJRDyVSSuqLx9cr2wGbKtzeDAyIw3Z46ctN/O2TtfH41VKDrdtxgIt7tOL2i7v6XYpITMSj0Vd2xMr914PMRgOjATp27BjRhpo2qEPX1qdE9FyRqvTv3JxfXNpDiZQSGPFo9JuBDhVutwe2Hvsg59xkYDJAZmbmf30QVMclvdtwSe82kTxVRKTGiMca/ZdAVzPrbGZ1geHAG3HYjoiIVEPM9+idc6VmNh54B6gFPOecWxrr7YiISPXEJevGOfcm8GY8freIiIRHfxkrIhJwavQiIgGnRi8iEnBq9CIiAadGLyIScOZcRH+rFNsizIqADRE+vSWwI4blxIrqCo/qCl+y1qa6whNNXZ2cc2knelBSNPpomNkC51ym33UcS3WFR3WFL1lrU13hSURdWroREQk4NXoRkYALQqOf7HcBVVBd4VFd4UvW2lRXeOJeV8qv0YuIyPEFYY9eRESOI6UavZldY2ZLzazczDKPue8+72TkK8zs0grjCT1RuZnNNLM872e9meV54+lmVlzhvknxruWYuh4wsy0Vtj+kwn2Vzl2C6vqzmS03s0Vm9pqZNfXGfZ0vr4akOMm9mXUws3lmlu+9/+/wxqt8TRNY23ozW+xtf4E31tzM3jOzVd5lswTX1L3CnOSZ2T4zu9OP+TKz58ys0MyWVBirdH4s5Anv/bbIzDJiVohzLmV+gJ5Ad+BDILPCeC9gIXAy0BlYQygiuZZ3vQtQ13tMrwTW+wjwa+96OrDEx7l7ALi7kvFK5y6BdV0C1PauPwQ8lCTz5et755ha2gIZ3vVGwErvdav0NU1wbeuBlseM/Qm417t+79HX1MfXcTvQyY/5Ai4AMiq+l6uaH2AI8Bahs/QNBObHqo6U2qN3zuU751ZUctdQYIZz7rBzbh2wmtBJyn07UbmZGXAtMD0R24tCVXOXEM65d51zpd7NzwmdkSwZJM1J7p1z25xzOd71/UA+oXMzJ6uhwBTv+hTgKh9rGQSscc5F+geZUXHOfQzsOma4qvkZCrzgQj4HmppZ21jUkVKN/jgqOyF5u+OMJ8K3gQLn3KoKY53NLNfMPjKzbyeojorGe/8lfK7Cf6f9nKNj/YjQHs1Rfs5XMs3L18wsHegLzPeGKntNE8kB75pZtoXOAw3Q2jm3DUIfUkArH+o6ajjf3Nnye76g6vmJ23su6Rq9mb1vZksq+Tne3lRVJySv1onK41Tj9XzzDbYN6Oic6wvcBbxoZo2jrSWMuiYCpwF9vFoeOfq0Sn5VTL+KVZ35MrMJQCkwzRuK+3ydqOxKxnz9ipqZnQK8CtzpnNtH1a9pIp3vnMsALgduM7MLfKihUhY6lemVwMveUDLM1/HE7T0XlzNMRcM5NziCpx3vhOQnPFF5uE5Uo5nVBr4P9KvwnMPAYe96tpmtAboBC6Ktp7p1Vajvb8C/vJvVOpl7POsys5HAFcAg5y1WJmK+TiDu8xIOM6tDqMlPc87NAnDOFVS4v+JrmjDOua3eZaGZvUZoyavAzNo657Z5Sw+Fia7LczmQc3SekmG+PFXNT9zec0m3Rx+hN4DhZnaymXUGugJf4N+JygcDy51zm48OmFmamdXyrnfxalybgFqObr/iWt8w4Oi3AKqau0TVdRlwD3Clc+5ghXFf54skOsm9d7znWSDfOfeXCuNVvaaJqquhmTU6ep3QgfUlhOZppPewkcDsRNZVwTf+V+33fFVQ1fy8AdzsfftmILD36BJP1BJ5BDoGR7CHEfrUOwwUAO9UuG8CoW9JrAAurzA+hNC3FNYAExJU5/PA2GPGfgAsJfTtjRzgfxI8d1OBxcAi7w3V9kRzl6C6VhNal8zzfiYlw3z59d6poo5vEfov/KIK8zTkeK9pgurq4r0+C73XaoI33gKYC6zyLpv7MGcNgJ1AkwpjCZ8vQh8024AjXu8aVdX8EFq6ecp7vy2mwjcLo/3RX8aKiARcUJZuRESkCmr0IiIBp0YvIhJwavQiIgGnRi8iEnBq9CIiAadGLyIScGr0IiIB9/8BC0kpYv14GdgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, [relu(num) for num in y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other statistical measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Standard deviation**  \n",
    "\n",
    "$$\\sigma ={\\sqrt {{\\frac {1}{N}}\\sum _{i=1}^{N}(x_{i}-\\mu )^{2}}},{\\rm {\\ \\ where\\ \\ }}\\mu ={\\frac {1}{N}}\\sum _{i=1}^{N}x_{i}$$\n",
    "\n",
    "$n$ = numer of oberservations; $x_{i}$ = observation value; $\\mu$ = population mean\n",
    "\n",
    "**Sample standard deviation**  \n",
    "\n",
    "$${\\displaystyle s={\\sqrt {\\frac {1}{n-1}{\\sum _{i=1}^{N}(x_{i}-{\\overline {x}})^{2}}}}}$$\n",
    "\n",
    "$n-1$ = Bessel's correction; $x_{i}$ = observation value; $\\overline x$ = sample mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Probability concepts explained**\n",
    "* https://towardsdatascience.com/probability-concepts-explained-introduction-a7c0316de465\n",
    "* https://towardsdatascience.com/probability-concepts-explained-maximum-likelihood-estimation-c7b4342fdbb1\n",
    "\n",
    ">Maximum likelihood estimation is a method that determines values for the parameters of a model. The parameter values are found such that they maximise the likelihood that the process described by the model produced the data that were actually observed.\n",
    "\n",
    ">We want to know which curve was most likely responsible for creating the data points that we observed? Maximum likelihood estimation is a method that will find the values of μ and σ that result in the curve that best fits the data.\n",
    "\n",
    "> Likelihood and the probability density are fundamentally asking different questions — one is asking about the data and the other is asking about the parameter values.\n",
    "\n",
    "* https://towardsdatascience.com/probability-concepts-explained-bayesian-inference-for-parameter-estimation-90e8930e5348"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}