34_cross_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ICS 434: DATA SCIENCE FUNDAMENTALS\n",
    "\n",
    "## Cross-Validation\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "\n",
    "# from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Overfitting and RMSE\n",
    "\n",
    "* in regression or classification, sespite the fact that RMSE $\\approx$ 0, or erro rate $\\approx$ 0 it is very unlikely that the model is perfect\n",
    "\n",
    "* The results can be due to the model overfitting the training data\n",
    "  * The model \"Remembers\" the data perfectly and predicts observation as it saw them\n",
    "  \n",
    "* You will remember that models that overfit the data tend to have poor generalizaiton power\n",
    "  * Computing the RMSE on the test data shows that the model is overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overfitting and RMSE -- Cont'd\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/d6ob11stfawcddhdsi3ox/overfitting.png?rlkey=ypaf0zepwh6s1n166u13oaue7&dl=1\" alt=\"drawing\" width=\"900\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Assessing a Model's Generalization Power\n",
    "\n",
    "* In real modeling/ML applications, we are interested in how models generalize to unseen data\n",
    "  * There is no merit in being able to \"regurgitate\" previously seen data\n",
    "\n",
    "* Therefore, the most appropriate statistical learning method or parameters are selected based on the results observed with previously unseen test data\n",
    "  * You cannot use the test set for mitigating overfitting since the model will learn to fit the test set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Non-linear Regression and Train-Test Split\n",
    "\n",
    "\n",
    "* Non-linear regression or classification can be complex models\n",
    "  * Have many parameters that can impact the RMSE\n",
    "\n",
    "* The test error rate can be highly variable, depending which observations are in which set (train and test)\n",
    "  * Different train-test splits will yield different results\n",
    "  \n",
    "* How do we test the performance of different parameters while assessing the random nature of the splits in the data?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Train-Test Split\n",
    "\n",
    "* The test error rate can be highly variable, depending which observations are in which set (train and test)\n",
    "  * Different train-test splits will yield different results\n",
    "  \n",
    "* How do we test the performance of different parameters while assessing the random nature of the splits in the data?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Assessing a Model's Generalization Power -- Cont'd\n",
    "\n",
    "* You don't want to use the test data until you have completed all your tests on parameterizing the model\n",
    "  * You cannot use the test set for minimizing overfit since the model will learn to fit the test set\n",
    "\n",
    "* For instance, to explore which degree is best, we can split the training set into:\n",
    "  * a smaller training set: used to train the model\n",
    "  * validation set: after the training (before the testing) to minimize overfitting "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Train / Validation / Test  Approach\n",
    "\n",
    "* Train model on the new, smaller training set\n",
    "\n",
    "* Explore different parameters if needed (eg. the polynomial degree)\n",
    "  * Choose those perform best on the validation set\n",
    "    \n",
    "* Only use the test set to compare the decision tree regressor against other models\n",
    "  * Choose the model with the smallest generalization error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Shortcomings of the Train / Validation / Test  Approach\n",
    "\n",
    "\n",
    "* Statistical methods tend to perform worse when trained to learn complex model using fewer observations\n",
    "  * Less data to learn the model \n",
    "  * Validation set error rate may tend to overestimate the test error rate\n",
    "\n",
    "* Wastes another chunk of data (validation set), which cannot be used in training\n",
    "\n",
    "* A good alternative to the training/validation/test is a method called K-fold cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### K-Fold Cross-Validation\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/re1v0br0vvmgb5gxpw2p1/cross_validation.png?rlkey=axb8gr9i0z0x2sx1z3rdhi4u3&dl=1\" alt=\"drawing\" style=\"width:800px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### K-Fold Cross-Validation -- Cont'd\n",
    "\n",
    "* Cross-validation is applied on the training set\n",
    "\n",
    "* We use the following algorithm to train/validate using $K$-fold cross-validation\n",
    "  * The training set is split into $K$ complementary chunks of data\n",
    "  * We consider $K-1$ chunks as training and $1$ chunk as testing\n",
    "  * We repeat the training/testing $K$ times and average the estimates (ex. RMSE) into a single cross-validation estimate\n",
    "\n",
    "* Once we find the best model parameters, we then use them to train on the full training set\n",
    "\n",
    "* It's common to use $K=10$ for cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Testing\n",
    "\n",
    "* Once we have tweaked all the parameters for all models, we can use the test data to compare the models' generalization performance\n",
    "  * For each model, we run the prediction on test data and compare the predicted with the observed values \n",
    "  * We select the model that makes the smallest error\n",
    "* We determine whether the generalized performance is sufficient for our application\n",
    "  * Ex. The performance requirements for a patient-facing application are perhaps not the same as those for an application for predicting algae blooms\n",
    "* Note that in this approach, we are rewarding models that generalize well even if they are more complex \n",
    " * We don't penalize complex models if they result in high generalization performance\n",
    " * Between models with relatively similar generalization performance, we need to choose the one with fewer parameters"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}