Gaussian Discriminant Analyses.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gaussian Descriminant Analysis Implementation using numpy\n",
    "\n",
    "When we have a classification problem in which the input features x are\n",
    "continuous-valued random variables, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution.\n",
    "\n",
    "GDA is a family of generative algorithm that try to model p(x|y) (and p(y)). After modeling p(y) (called the class priors) and p(x|y), our algorithm can then use Bayes rule to derive the posterior distribution on y given x:\n",
    "\n",
    "\n",
    "![title](figures/bayes-theorem.png)\n",
    "\n",
    "GDA makes stronger modeling assumptions, and is more data efficient (i.e., requires less training data to learn “well”) when the modeling assumptions are correct or at least approximately correct. Comparing to Logistic regression that makes weaker assumptions, and is significantly more robust to deviations from modeling assumptions\n",
    "\n",
    "### Our motivation\n",
    "To gain better understand on how the algorithm works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load gda.py\n",
    "import numpy as np\n",
    "\n",
    "class GDABinaryClassifier:\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        self.fi = y.mean()\n",
    "        self.u = np.array([ X[y==k].mean(axis=0) for k in [0,1]])\n",
    "        X_u = X.copy()\n",
    "        for k in [0,1]: X_u[y==k] -= self.u[k]\n",
    "        self.E = X_u.T.dot(X_u) / len(y)\n",
    "        self.invE = np.linalg.pinv(self.E)\n",
    "        return self\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return np.argmax([self.compute_prob(X, i) for i in range(len(self.u))], axis=0)\n",
    "    \n",
    "    def compute_prob(self, X, i):\n",
    "        u, phi = self.u[i], ((self.fi)**i * (1 - self.fi)**(1 - i))\n",
    "        return np.exp(-1.0 * np.sum((X-u).dot(self.invE)*(X-u), axis=1)) * phi\n",
    "    \n",
    "    def score(self, X, y):\n",
    "        return (self.predict(X) == y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9666080843585237"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "X,y = load_breast_cancer(return_X_y=True)\n",
    "model = GDABinaryClassifier().fit(X,y)\n",
    "pre = model.predict(X)\n",
    "model.score(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}