Jean_Ndah_FML_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $~~~~~~~~~~~~~~~~~~~~\\color{black}{\\text{Jean N'dah Kouagou}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "###  $~~~~~~~~~~~~~~~~~\\color{blue}{\\text{Foundation of Machine Learning}} $\n",
    "\n",
    "#### $~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\color{green}{\\underline{\\text{Project 1}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{black}{\\text{Introduction}}$\n",
    "\n",
    "In this project, we implement three machine learning algorithms: Naive Bayes, Softmax Logistic Regression and Gaussian Discriminant Analysis. The three algorithms are used for classificaction and are tested on the same data set for comparison. Some nice observations have been made which we summarise at the bottom of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip3 install progress_bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np, pandas as pd, seaborn as sns, progress_bar as pb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder as le\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Naive_Bayes(object):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def Phi_ki(self, i, k):\n",
    "        \"\"\"This function is a method to compute the probabilities P(X^(i)=1/y=k)\"\"\"\n",
    "        Res1=np.sum((self.features.iloc[:, i] == 1) & (self.target==k))+1 # adding 1 for \n",
    "        #Laplace smoothing\n",
    "        Res2=np.sum(self.target==k)\n",
    "        Res3=self.target.nunique()# considering Res3 in the denominator still for\n",
    "        #Laplace smoothing\n",
    "        return Res1/(Res2+Res3)\n",
    "    \n",
    "    def train(self, features, target): \n",
    "        # Providing data for training\n",
    "        self.features=features\n",
    "        self.target=target\n",
    "        # Computing the probabilities P(y=k)\n",
    "        self.p_y_s=np.array([np.sum(self.target==j)/len(self.target) for j in \\\n",
    "                          np.unique(self.target)])\n",
    "        #Here we compute the probabilities P(X^(i)=1/y=k)\n",
    "        phi_i_k_s=np.array([[self.Phi_ki(i, k) for k in np.unique(self.target)]\\\n",
    "                        for i in range(self.features.shape[1])])\n",
    "        self.phi_i_k_s=phi_i_k_s\n",
    "\n",
    "    def P_Xi_given_y(self, i, k, X):\n",
    "        \"\"\"This function computes P(X^(i)/y=k). Since X^(i)/y=k follows Bernouli distribution\n",
    "        phi_k^(i), we then return the following:\"\"\"\n",
    "        return self.phi_i_k_s[i][k]**(X[i])*(1-self.phi_i_k_s[i][k])**\\\n",
    "                    (1-X[i])\n",
    "        \n",
    "    \n",
    "    def P_y_equal_k_given_X(self, k, X):\n",
    "        \"\"\"This function computes P(y=k/X) and will be used to predict a class\"\"\"\n",
    "        Res1=np.prod([self.P_Xi_given_y(i, k, X) for i in range(len(X))])\n",
    "        Res2=self.p_y_s[k]\n",
    "        return Res1*Res2\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"This function uses the above function for the predictions\"\"\"\n",
    "        predictions=[]\n",
    "        bar = pb.ProgressBar(\"predictions\", offset=2, total_width=100, min_value=0, \\\n",
    "                             max_value=X.shape[0]-1)\n",
    "        for j in range(X.shape[0]):\n",
    "            bar(j)\n",
    "            List=[]\n",
    "            for k in np.unique(self.target):\n",
    "                List.append(self.P_y_equal_k_given_X(k, X.iloc[j, :]))\n",
    "            predictions.append(np.argmax(List))\n",
    "        return np.array(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gaussian Discriminant Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GDAMulticlass(object):\n",
    "    \n",
    "    def __init__(self, regularizer=0.1):\n",
    "        # The regularizer will allow us avoid singular matrices during the computations\n",
    "        self.regularizer=regularizer\n",
    "        \n",
    "    def train(self, features, target):\n",
    "        self.features=features\n",
    "        self.target=target\n",
    "        self.number_features=self.features.shape[1]\n",
    "        combined=pd.concat([features, target], axis=1)\n",
    "        # We compute the parameters mu_k associated with each class\n",
    "        self.mu_s=[combined[combined['target']==j].drop('target', axis=1).mean(axis=0)\\\n",
    "                   for j in np.unique(self.target)]\n",
    "        # And here are the parameters phi_k associated with each class\n",
    "        self.phi_s=(1.0/len(self.target))*np.array([self.target[self.target==j].count() \\\n",
    "                                    for j in np.unique(target)])\n",
    "        \n",
    "        # Finally, we compute the matrix sigma\n",
    "        sigma=np.matrix(np.zeros([self.number_features, self.number_features]))\n",
    "        \n",
    "        bar = pb.ProgressBar(\"training\", offset=2, total_width=100, min_value=0,\\\n",
    "        max_value=self.target.shape[0]-1) # This tracks the progression of the computation\n",
    "        #of sigma\n",
    "        for i in range(self.target.shape[0]):\n",
    "            bar(i)\n",
    "            sigma += np.dot(np.matrix(self.features.iloc[i, :]-\\\n",
    "            self.mu_s[int(self.target.iloc[i])]).T, \\\n",
    "            np.matrix(self.features.iloc[i, :]-self.mu_s[int(self.target.iloc[i])]))\n",
    "        \n",
    "        self.sigma=(1.0/self.target.shape[0])*sigma\n",
    "    \n",
    "    def P_x_given_y(self, sigma, x, mu):\n",
    "        # The following if condition helps to avoid overflow in the computation of the \n",
    "        #exponential when its parameter is too large\n",
    "        if self.number_features>380:\n",
    "            #The following is to avoid singular matrices\n",
    "            if np.linalg.det(sigma)==0:\n",
    "                sigma=sigma+self.regularizer*np.eye(self.number_features)\n",
    "                comp1 = 1.0/(np.sqrt((2*np.pi)**380)*np.sqrt(np.linalg.det(sigma)))\n",
    "                comp2 = float(np.exp(np.dot(-0.5*np.dot(x-mu, np.linalg.inv(sigma)), x-mu)))\n",
    "            else:\n",
    "                comp1 = 1.0/(np.sqrt((2*np.pi)**380)*np.sqrt(np.linalg.det(sigma)))\n",
    "                comp2 = float(np.exp(np.dot(-0.5*np.dot(x-mu, np.linalg.inv(sigma)), x-mu)))\n",
    "        else:\n",
    "            if np.linalg.det(sigma)==0:\n",
    "                sigma=sigma+self.regularizer*np.eye(self.number_features)\n",
    "                comp1 = 1.0/(np.sqrt((2*np.pi)**self.features.shape[1])*\\\n",
    "                             np.sqrt(np.linalg.det(sigma)))\n",
    "                comp2 = float(np.exp(np.dot(-0.5*np.dot(x-mu, np.linalg.inv(sigma)), x-mu)))\n",
    "            else:\n",
    "                comp1 = 1.0/(np.sqrt((2*np.pi)**self.features.shape[1])*\\\n",
    "                             np.sqrt(np.linalg.det(sigma)))\n",
    "                comp2 = float(np.exp(np.dot(-0.5*np.dot(x-mu, np.linalg.inv(sigma)), x-mu)))\n",
    "            \n",
    "        return comp1*comp2\n",
    "        \n",
    "    def predict(self, X):\n",
    "        predictions=[]\n",
    "        bar = pb.ProgressBar(\"predictions\", offset=2, total_width=100, min_value=0, \\\n",
    "                                                        max_value=X.shape[0]-1)\n",
    "        for i in range(X.shape[0]):\n",
    "            bar(i)\n",
    "            Prob=[self.P_x_given_y(self.sigma, X.iloc[i, :], self.mu_s[j])*self.phi_s[j]\\\n",
    "                  for j in np.unique(self.target)]\n",
    "            \n",
    "            predictions.append(np.argmax(Prob))\n",
    "            \n",
    "        return np.array(predictions)\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiclass Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MulticlassLogReg(object):\n",
    "    def __init__(self, lr=0.2, Max_iter=30):\n",
    "        self.lr=lr\n",
    "        self.Max_iter=Max_iter\n",
    "       \n",
    "    #Definition of the softmax function\n",
    "    def softmax(self, k, X):\n",
    "        return np.exp(X[k])/np.sum([np.exp(X[j]) for j in range(len(X))])\n",
    "    \n",
    "    #Function to compute the probability of (y=k/X_i; weights)\n",
    "    def prob_yequal_k_given_Xi_weights(self, X_i, k, Weights):\n",
    "        return self.softmax(k, [np.dot(Weights[l], X_i) for l in range(len(Weights))])\n",
    "    \n",
    "    # Function to compute the gradient associated to the target k\n",
    "    def get_gradient(self, k, Weights, X, y):\n",
    "        K=len(Weights)\n",
    "        grad_k=-np.array([(float(y.iloc[i]==k)-self.prob_yequal_k_given_Xi_weights(X.iloc[i, :], k, Weights))\\\n",
    "                 *X.iloc[i, :] for i in range(X.shape[0])]).mean(axis=0)\n",
    "        return grad_k\n",
    "    \n",
    "    def fit(self, features, target, weights):\n",
    "        self.target=target\n",
    "        self.weights=weights\n",
    "        Intercept=np.ones_like(self.target).reshape(-1,1)\n",
    "        features=pd.DataFrame(np.concatenate([Intercept, features], axis=1))\n",
    "        \n",
    "        num_iter=0\n",
    "        #To track the progress of the training process\n",
    "        bar=pb.ProgressBar(\"training\", offset=2, total_width=100,\\\n",
    "                            min_value=0, max_value=self.Max_iter-1)\n",
    "        \n",
    "        while (num_iter<self.Max_iter):\n",
    "            bar(num_iter)\n",
    "            previous_weights=self.weights.copy()\n",
    "            for j in range(self.weights.shape[0]):\n",
    "                #We update each weight\n",
    "                self.weights[j]=self.weights[j]-self.lr*self.get_gradient(j, self.weights,\\\n",
    "                                    features, target)\n",
    "            num_iter += 1\n",
    "            \n",
    "    def predict(self, X_):\n",
    "         #We first add the intercept column\n",
    "        Intercept=np.ones(X_.shape[0]).reshape(-1,1)\n",
    "        X=pd.DataFrame(np.concatenate([Intercept, X_], axis=1))\n",
    "        \n",
    "        predictions=[]\n",
    "        # To track the prediction progress\n",
    "        bar = pb.ProgressBar(\"predictions\", offset=2, total_width=100,\\\n",
    "                            min_value=0, max_value=X.shape[0]-1)\n",
    "        for i in range(X.shape[0]):\n",
    "            bar(i)\n",
    "            \n",
    "            probs=[]\n",
    "            # Here we compute the probabilties for y to be a given target\n",
    "            for k in np.unique(self.target):\n",
    "                probs.append(self.prob_yequal_k_given_Xi_weights(X.iloc[i, :], k, self.weights))\n",
    "            # Now we choose the target with the highest probability\n",
    "            predictions.append(np.argmax(probs))\n",
    "        return np.array(predictions)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Data to use for the project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "Data_train=pd.read_table('drugLibTrain_raw.tsv')\n",
    "Data_test=pd.read_table('drugLibTest_raw.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Totaldata=pd.concat([Data_train, Data_test], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>urlDrugName</th>\n",
       "      <th>rating</th>\n",
       "      <th>effectiveness</th>\n",
       "      <th>sideEffects</th>\n",
       "      <th>condition</th>\n",
       "      <th>benefitsReview</th>\n",
       "      <th>sideEffectsReview</th>\n",
       "      <th>commentsReview</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2202</td>\n",
       "      <td>enalapril</td>\n",
       "      <td>4</td>\n",
       "      <td>Highly Effective</td>\n",
       "      <td>Mild Side Effects</td>\n",
       "      <td>management of congestive heart failure</td>\n",
       "      <td>slowed the progression of left ventricular dys...</td>\n",
       "      <td>cough, hypotension , proteinuria, impotence , ...</td>\n",
       "      <td>monitor blood pressure , weight and asses for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3117</td>\n",
       "      <td>ortho-tri-cyclen</td>\n",
       "      <td>1</td>\n",
       "      <td>Highly Effective</td>\n",
       "      <td>Severe Side Effects</td>\n",
       "      <td>birth prevention</td>\n",
       "      <td>Although this type of birth control has more c...</td>\n",
       "      <td>Heavy Cycle, Cramps, Hot Flashes, Fatigue, Lon...</td>\n",
       "      <td>I Hate This Birth Control, I Would Not Suggest...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1146</td>\n",
       "      <td>ponstel</td>\n",
       "      <td>10</td>\n",
       "      <td>Highly Effective</td>\n",
       "      <td>No Side Effects</td>\n",
       "      <td>menstrual cramps</td>\n",
       "      <td>I was used to having cramps so badly that they...</td>\n",
       "      <td>Heavier bleeding and clotting than normal.</td>\n",
       "      <td>I took 2 pills at the onset of my menstrual cr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0       urlDrugName  rating     effectiveness  \\\n",
       "0        2202         enalapril       4  Highly Effective   \n",
       "1        3117  ortho-tri-cyclen       1  Highly Effective   \n",
       "2        1146           ponstel      10  Highly Effective   \n",
       "\n",
       "           sideEffects                               condition  \\\n",
       "0    Mild Side Effects  management of congestive heart failure   \n",
       "1  Severe Side Effects                        birth prevention   \n",
       "2      No Side Effects                        menstrual cramps   \n",
       "\n",
       "                                      benefitsReview  \\\n",
       "0  slowed the progression of left ventricular dys...   \n",
       "1  Although this type of birth control has more c...   \n",
       "2  I was used to having cramps so badly that they...   \n",
       "\n",
       "                                   sideEffectsReview  \\\n",
       "0  cough, hypotension , proteinuria, impotence , ...   \n",
       "1  Heavy Cycle, Cramps, Hot Flashes, Fatigue, Lon...   \n",
       "2         Heavier bleeding and clotting than normal.   \n",
       "\n",
       "                                      commentsReview  \n",
       "0  monitor blood pressure , weight and asses for ...  \n",
       "1  I Hate This Birth Control, I Would Not Suggest...  \n",
       "2  I took 2 pills at the onset of my menstrual cr...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Totaldata.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd38ac40d10>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEGCAYAAABrQF4qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyV9Z33/9cn+wokJGEJgQQICLigBnADtVrF2kpnrOu0ox071rm1de52OtXf9G7ndh69p8tMZzpTpy13tZsLrvWmLRZtXVFBAohsImFLwpqQAFnIej6/P3J0jiHASUhycq68n49HHjnXdX2/53wO5Lxz5Vq+X3N3REQkuBJiXYCIiAwsBb2ISMAp6EVEAk5BLyIScAp6EZGAS4p1Ad3l5eV5cXFxrMsQEYkra9asqXX3/J62DbmgLy4upry8PNZliIjEFTPbfaJtUR26MbOFZrbVzCrM7L6TtPuMmbmZlUWsuz/cb6uZXd270kVE5HSdco/ezBKBB4GPA9XAajNb6u6bu7XLBr4MrIpYNxO4GZgFjAf+aGbT3L2z/96CiIicTDR79HOBCnff4e5twBJgUQ/t/gn4HtASsW4RsMTdW919J1ARfj4RERkk0QR9IVAVsVwdXvchMzsXKHL33/W2b7j/nWZWbmblNTU1URUuIiLRiSborYd1Hw6QY2YJwL8BX+1t3w9XuC929zJ3L8vP7/GksYiI9FE0V91UA0URyxOAvRHL2cCZwCtmBjAWWGpm10XRV0REBlg0e/SrgVIzKzGzFLpOri79YKO7H3H3PHcvdvdiYCVwnbuXh9vdbGapZlYClAJv9/u7EBGREzrlHr27d5jZPcByIBF42N03mdkDQLm7Lz1J301m9iSwGegA7tYVNyIig8uG2nj0ZWVlrhumRER6x8zWuHtZT9uG3J2xIiLdPbaqss99b503sR8riU8a1ExEJOAU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4KIKejNbaGZbzazCzO7rYftdZrbBzN4xsxVmNjO8vtjMjoXXv2NmP+nvNyAiIid3yqkEzSwReBD4OFANrDazpe6+OaLZY+7+k3D764AfAAvD27a7++z+LVtERKIVzR79XKDC3Xe4exuwBFgU2cDdj0YsZgJDa8ZxEZFhLJqgLwSqIparw+s+wszuNrPtwPeAL0dsKjGzdWb2qpnN7+kFzOxOMys3s/KamppelC8iIqcSTdBbD+uO22N39wfdfQrwdeAb4dX7gInufi7wFeAxMxvRQ9/F7l7m7mX5+fnRVy8iIqcUTdBXA0URyxOAvSdpvwT4NIC7t7r7ofDjNcB2YFrfShURkb6IJuhXA6VmVmJmKcDNwNLIBmZWGrF4LbAtvD4/fDIXM5sMlAI7+qNwERGJzimvunH3DjO7B1gOJAIPu/smM3sAKHf3pcA9ZnYl0A7UA7eFuy8AHjCzDqATuMvd6wbijYiISM9OGfQA7r4MWNZt3TcjHt97gn7PAM+cToEiInJ6dGesiEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4BT0IiIBp6AXEQk4Bb2ISMBFFfRmttDMtppZhZnd18P2u8xsg5m9Y2YrzGxmxLb7w/22mtnV/Vm8iIic2imDPjy594PANcBM4JbIIA97zN3PcvfZwPeAH4T7zqRrMvFZwELgvz6YLFxERAZHNHv0c4EKd9/h7m3AEmBRZAN3PxqxmAl4+PEiYIm7t7r7TqAi/HwiIjJIopkcvBCoiliuBuZ1b2RmdwNfAVKAj0X0Xdmtb2EPfe8E7gSYOHFiNHWLyGl6bFVln/rdOk+f0XgTzR699bDOj1vh/qC7TwG+Dnyjl30Xu3uZu5fl5+dHUZKIiEQrmqCvBooilicAe0/Sfgnw6T72FRGRfhZN0K8GSs2sxMxS6Dq5ujSygZmVRixeC2wLP14K3GxmqWZWApQCb59+2SIiEq1THqN39w4zuwdYDiQCD7v7JjN7ACh396XAPWZ2JdAO1AO3hftuMrMngc1AB3C3u3cO0HsREZEeRHMyFndfBizrtu6bEY/vPUnfbwPf7muBIiJyenRnrIhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScAp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJOAW9iEjAKehFRAIuqqA3s4VmttXMKszsvh62f8XMNpvZu2b2JzObFLGt08zeCX8t7d5XREQG1innjDWzROBB4ONANbDazJa6++aIZuuAMndvNrO/Ab4H3BTedszdZ/dz3SIiEqVo9ujnAhXuvsPd24AlwKLIBu7+srs3hxdXAhP6t0wREemraIK+EKiKWK4OrzuRO4DnI5bTzKzczFaa2ad76mBmd4bblNfU1ERRkoiIROuUh24A62Gd99jQ7LNAGXBpxOqJ7r7XzCYDL5nZBnff/pEnc18MLAYoKyvr8blFRKRvogn6aqAoYnkCsLd7IzO7EvgH4FJ3b/1gvbvvDX/fYWavAOcC27v3F5Hge2xVZaxLGJaiOXSzGig1sxIzSwFuBj5y9YyZnQv8FLjO3Q9GrM8xs9Tw4zzgYiDyJK6IiAywU+7Ru3uHmd0DLAcSgYfdfZOZPQCUu/tS4PtAFvCUmQFUuvt1wAzgp2YWouuXyne6Xa0jIiIDLJpDN7j7MmBZt3XfjHh85Qn6vQmcdToFiojI6dGdsSIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4BT0IiIBF1XQm9lCM9tqZhVmdl8P279iZpvN7F0z+5OZTYrYdpuZbQt/3dafxYuIyKmdMujNLBF4ELgGmAncYmYzuzVbB5S5+9nA08D3wn1zgW8B84C5wLfMLKf/yhcRkVOJZo9+LlDh7jvcvQ1YAiyKbODuL7t7c3hxJTAh/Phq4EV3r3P3euBFYGH/lC4iItGIJugLgaqI5erwuhO5A3i+N33N7E4zKzez8pqamihKEhGRaEUT9NbDOu+xodlngTLg+73p6+6L3b3M3cvy8/OjKElERKIVTdBXA0URyxOAvd0bmdmVwD8A17l7a2/6iojIwIkm6FcDpWZWYmYpwM3A0sgGZnYu8FO6Qv5gxKblwFVmlhM+CXtVeJ2IiAySpFM1cPcOM7uHroBOBB52901m9gBQ7u5L6TpUkwU8ZWYAle5+nbvXmdk/0fXLAuABd68bkHciIiI9OmXQA7j7MmBZt3XfjHh85Un6Pgw83NcCRUR60tTawbrKet6pPkxre4iEBKMgO5UrZ4xhzIi0WJc3pEQV9CIyND22qjLWJQy6jlCIl947yOvbaukMOUU56RTmpNMZcrbXNLJ571HmFOfyibPGkZKkm/9BQS8icWT/0RaeKq9i35EWZheNYkFpPmNH/vfee3NrBy9tPchb2w/R2NrBrfMmxrDaoUNBLyJxYfPeozxRXklKUiKfu2ASM8aNOK5NRmoSnzx7PDkZKfx+wz5e2LSfz14wqYdnG14U9CIypLk7b2w/xPMb9lGYk87nLphEdlrySftcNGU0tY2tvLatlmfWVHP9+RNO2j7odABLRIaszpCzdP1elm3Yx8zxI/jCJZNPGfIAZsYnzx5P8egMvr1sC0db2geh2qFLQS8iQ1JLeye/XrmLVTvrmF+axy1zJ/bq5GpignHtWeOpa2rjp69uH8BKhz4FvYgMOYeb21j82g4qDjbyZ7MLuebMcSRYTyOqnFxhTjqLZo/noRU72X+kZQAqjQ8KehEZUvbUH+PHr26nvrmN2y4qZk5J7mk9399dNZ1QCP7txff7qcL4o6AXkSHB3Vm9q47Fr28nMcG469IplBZkn/bzFuVm8NkLJvHUmir2HD7WD5XGHwW9iMTcsbZOniiv4jfr9jAxN4O/uXRKv97d+leXFOPAE28PvxvMQJdXikgMhdxZu7ue5Zv2c6y9k6tmjmHBtPw+HY8/mQk5GVw2LZ8lq6v40hWlJCcOr31cBX0M9fX2dd3tJ4OtIxSitqGN5rYOXti0n/SURHIyUijITiU/OxXrZTC3d4bYUH2EN7bXsu9ICxNzM/jUOeMpHJU+QO8Abp03ib/+VTl/2nKAhWeOG7DXGYoU9CLSo7aOEGsq69my7yi7DzXR3tk1Z9DPVuz8SLvMlERK8jOZnJdFSV4mk8OPi3LTSUtOJDkxgfrmNvYfaWHN7nq21zTy/oEGmts6KchO5YbzJ3BO0ah+34vv7vLp+YwbmcajqyoV9CIyvLV3hnh9Wy1vbq/9MIzLJuUycXQGWalJ/Nm5hbS0d1LX1Ma+Iy3srG1ie00jayvr+e27e/Ee55/7b5mpSZQWZHH+pFym5Gf2+q+BvkpKTOCmOUX8+x+3sftQE5NGZw7K6w4FCnoR+dCe+mM8uaaKmoZWzhibzaXT8o8LxDMLR56wf0t7J7sPNbOjppE9h4/R2hGivTPEqPRkxo5MY/O+Bsb04VBPf7l5zkT+40/beGZNNV+5anpMaogFBb2I4O68uf0Qz2/cR1ZqEp+/qJjSMb2/tDEtOZHpY7OZPrbnvnVNsR2KYOzINOaW5LJs4/5hFfTD69SziBwn5M7vN+zj9xv2ccbYEdx7xbQ+hXy8+MRZ46g42Mi2Aw2xLmXQKOhFhrHOkPPE6ire3H6Ii6eM5tZ5E0lPSYx1WQNq4ayxmMHvN+yLdSmDJqqgN7OFZrbVzCrM7L4eti8ws7Vm1mFmn+m2rdPM3gl/Le3eV0RiI+TOU2uq2LDnCAtnjeXas8cP+JUvQ0HBiDTmTMplmYL+v5lZIvAgcA0wE7jFzGZ2a1YJ3A481sNTHHP32eGv606zXhHpB+7Oc+v28G71Ea4O36Q0nHzirLG8f6CRioPD4/BNNHv0c4EKd9/h7m3AEmBRZAN33+Xu7wKhAahRRPrZH7ccoHx3PZdNz+fS6QWxLmfQfXAd/bIN+2NcyeCIJugLgaqI5erwumilmVm5ma00s0/31MDM7gy3Ka+pqenFU4tIb62vOszLW2s4f1IOH58xJtblxMTYkWmUTcrh+Y0K+g/0dNDuFLdEfMREdy8DbgX+3cymHPdk7ovdvczdy/Lzh9efkCKDqaqumWfWVlM8OoNFs8fH7Hr2oeDKmWPYsu/osBinPpqgrwaKIpYnAHujfQF33xv+vgN4BTi3F/WJSD85cqydR1btJjstiVvnTSIpYXhfdHd5+JDVK1sPxriSgRfN//RqoNTMSswsBbgZiOrqGTPLMbPU8OM84GJgc1+LFZG+aesI8cjK3bR2hPjchcVkpepeyWljshg/Mo2XFfTg7h3APcByYAvwpLtvMrMHzOw6ADObY2bVwA3AT81sU7j7DKDczNYDLwPfcXcFvcggcneeWVvN3sPHuKmsiLH9OM57PDMzLjujgBXbamnrCPZ1JFH9Wnf3ZcCybuu+GfF4NV2HdLr3exM46zRrFJHTsKKilg17jnD1rLHMGDci1uUMKZdPL+CxVZWU76rjoql5sS5nwAzvg3QiAbf7UBPLN+1n1vgRLCgNbpD11UVTRpOSmBD4wzcKepGAamzt4PG3KxmVkcL1500Y1lfYnEhmahLzJufy8tZgX9atoBcJoJA7T5VX0dzWya1zJ5KWHOzxa07HZdMLqDjYSFVdc6xLGTAKepEAemXrQbYdbOSTZ49n/ABOzxcEl03vunfntW3B3atX0IsETMXBRv605SCzi0Yxpzgn1uUMeZPzMhk/Mo3X36+NdSkDRkEvEiBHW9p5oryKvOzUYX/na7TMjPml+by5vZaOzmBeZqmgFwmIzpCz5O0q2jq6jsunJum4fLQuKc3jaEsH7+45EutSBoSCXiQg/rjlALsONfHp2YWM0U1RvXLx1DzMCOzhGwW9SABs3X+UV9+vYU5xDudO1HH53srNTOHM8SNZURHME7IKepE4d7i5jSfLqxk3Mo1Pnj0+1uXErfmleaytPExDS2wnMB8ICnqRONYRCvH425WE3Lll7kSSE/WR7qv5pfl0hpyVO+piXUq/00+FSBxbvnE/VfXH+PPzJpCXlRrrcuLaeZNGkZ6cyOsBvJ5eQS8Sp/6wcR9vbD/EhVNGc1bhyFiXE/dSkxK5YHIuK7YF74Ssgl4kDu2qbeJrT73LhJx0rjlzbKzLCYxLSvPZUdtEdX2whkNQ0IvEmZb2Tv7Ho2tJSDBumTtx2M8U1Z8+GOEzaHv1+gkRiTMP/G4zm/cd5Qc3nkNORkqsywmUqQVZjBmRyusKehGJlefW7eGxVZXcdekUrpgxJtblBM4HwyG8sb2WzpDHupx+E1XQm9lCM9tqZhVmdl8P2xeY2Voz6zCzz3TbdpuZbQt/3dZfhYsMN9sONHD/sxuYW5zL3101LdblBNb80jwON7ezMUDDIZwy6M0sEXgQuAaYCdxiZjO7NasEbgce69Y3F/gWMA+YC3zLzHTbnkgvNbd18D8eXUtGSiL/eeu5JOl6+QFzcXhKwRUVwTl8E81Py1ygwt13uHsbsARYFNnA3Xe5+7tA96HfrgZedPc6d68HXgQW9kPdIsOGu/ON32ykoqaRH958rsaxGWB5WanMGj+C194PzvX00QR9IVAVsVwdXheN0+krIsATq6t4dt0e7r2ilEs07+uguKQ0j7WV9TS1dsS6lH4RTdD3NKB1tGcpouprZneaWbmZldfUBOe3qMjp2rT3CN9cuon5pXl86WOlsS5n2FhQmk97p7Nq56FYl9Ivogn6aqAoYnkCsDfK54+qr7svdvcydy/Lz8+P8qlFgq2hpZ27H11LTkYy/3bTbBITNInIYDl/Ug6pSQm8FpBhi6MJ+tVAqZmVmFkKcDOwNMrnXw5cZWY54ZOwV4XXichJuDtff+ZdquqP8Z+3nKdxbAZZWnIi8yaPDswJ2VMGvbt3APfQFdBbgCfdfZOZPWBm1wGY2RwzqwZuAH5qZpvCfeuAf6Lrl8Vq4IHwOhE5iYdW7GTZhv187erpzC3JjXU5w9KC0jwqDjay78ixWJdy2pKiaeTuy4Bl3dZ9M+LxaroOy/TU92Hg4dOoUaTXHltV2ee+t86b2I+V9N6bFbX88/PvcdXMMdw5f3JMaxnOPjjx/fq2Wm4sKzpF66FNF+OKDCHV9c3c8/g6SvIy+cFNs0nQcfmYmT4mm/zsYAyHoKAXGSKOtXXyxV+vob0zxOLPnU9WalR/cMsAMTPmT83jjYpaQnE+HIJ+kkT6yekcLrplbhH3P/sum/cd5aHbypicn9WPlUlfzZ+Wx7Pr9rB531HOjOMx/7VHLzIEPLRiJ8+9s5evXDmNj52hwcqGig+GQ4j3wzcKepEY27r/KP/8/HtcPWsMd18+NdblSISC7DTOGJsd99MLKuhFYqi6vpnH3q7kjLHZ/OuNOvk6FC2Ylk/5rnqOtXXGupQ+0zF6iUo8X644VB1qbOWXb+4iKzWJn39+jk6+DlGXTM1j8Ws7WLXzEJdNL4h1OX2in6wYaWhpp/JQE7WNbbR2hhgzIpVxI9JJT0mMdWkyCOqa2vjZip04cPtFJRRka0TKoWpuSS4pSQm8vq1WQS/ROdjQwuJXd/DIqt20tHcf1RlmjM1mwbR8Jo3OjEF1Mhjqm9t4aMUO2jpC3HFJCfnZGt5gKEtLTmRucW5czyOroB9ET6yu5FtLN9HWEeLTswvJTE0iPyuV5KQEDhxtYVdtE6t21rHltR2UFmTxZ+cWMkpzggZKTUMrv3hzJ8faO7nj4smMH5Ue65IkCvNL8/jn59/jwNGWuJwPQEE/CDpDzrd/v4WH39jJ/NI8Hlh0JiV5mR857j0yPZlpY7K5bHoBq3Ye4o9bDvAfL23jU2eP59yJmpQrCCoPNfGrlbsx4K8uLqEwRyEfLy4pzYPnYcW2Wq4/v8fRXoY0XXUzwDpDzt2PruXhN3Zy+0XF/Pz2OZTknfiwTEpSAvNL8/nyx0opyE7jqTXVLF2/J1ATFQ9H66sO89AbO0lLTuSuS6cwIScj1iVJL8wYO4K8rJS4vcxSe/QDyN351tKN/GHTfr5x7Qy+0IsBqkZnpXLngsn8YeN+VlTUUtvQxi1zJ+pkbZxp7wzx+w37eHtnHZNyM/iLCybp6po4lJBgzC/N55WtB+noDMXdnL3xVW2c+a9XtvPIykq+eOnkXoX8BxLM+MRZ4/jzcwvZWdvET17bzuHmtgGoVAZC5aEm/uuVCt7eWceC0jy+MH+yQj6OXT1rDPXN7by9K/5GWlfQD5CX3jvA95dv5dOzx/P1q884recqK87l8xcX09DSzo9f3c6mvUf6qUoZCI2tHfy/d/bw09d20NIe4rYLJ7HwzHGaISrOLZiWT1pyAss37o91Kb2m3YsBsP9IC199cj0zx43gO9ef3S93O07Oz+KLC6bwizd3cdNPV/Ljz57H/NJgT7t4OjdpxUJjawdvVNTy5vZaOjqdCyaP5qqZY0hN1uG2IMhISWJBaT7LNx3gW5+aFVd3MWuPvp91hpx7l6yjtSPEf956Lmn9+CEfMyItfCIvnc//fDXPrKnut+eWvmnrCPHq+zXc/ehavvv8e7z2fg0zxo3gb6+cxqfOGa+QD5iFZ45l/9EW3t0TX39Va4++n/3k1e2s2lnHv95wDlMGYKjZkenJPHnXhfzNI2v46lPr2XfkGHdfPhWz+Nm7iGcdnSHe29/Ausp6Vu6o49X3a2hs7WBURjIXTM5lbslo3QAVYFecMYakBOMPG/czu2hUrMuJWlRBb2YLgR8CicDP3P073banAr8CzgcOATe5+y4zK6Zrntmt4aYr3f2u/il96Kk42MAP/7iNa88aN6DX2o5IS+bnt8/l759ez7+88D57j7TwwHWz4u5KgKHqSHM71Yeb2Xu4hT31zew90kJ1fTPbDzaxo7aR9s6uS13HjEjlU+eM48oZY7h4ah7Prt0T48ploI3MSObCKaP5w8Z9fH3h9LjZwTpl0JtZIvAg8HGgGlhtZkvdfXNEszuAenefamY3A98Fbgpv2+7us/u57iEnFHLue2YD6SmJ/ON1swb0tT44dj2nOJf65nYeW1XJ6p113FBWdMqrOjTAWJf2zhAHjraw93ALNQ0t1De3U9/cRn1zG//fbzZ8pG1KUgKFo9KZkp/J5WcUMGNcNudPyqFwVHrcfNCl/1w9ayzfeG4j7+1vYMa4EbEuJyrR7NHPBSrcfQeAmS0BFgGRQb8I+Mfw46eBH9kw+wQ8smo35bvr+Zcbzhm0P93NjKtnjSU3M4Xfrt/Lj17axo1zipicp9mJujvc3EbFwUZ2HWpi7+EWDja08ME9aMmJRk5GCjkZKUwanckVZxQwISed8aPSKcxJZ3RmigJdPnTNmWP5x6WbeG7dnkAFfSFQFbFcDcw7URt37zCzI8Do8LYSM1sHHAW+4e6vd38BM7sTuBNg4sT42+Pcc/gY333+PeaX5nH9eYWD/vpzinMpHJXO429X8tDrO5k3OZerZo7t1xPB8aalvZOdtU1sO9hAxcFGahu77j/ITE2icFQaZ4zLZ/zIrjDPyUj+SJDrrx45mdFZqVw2vYDfrNvD3y88Iy4um40m6Ht6F93vxz9Rm33ARHc/ZGbnA8+Z2Sx3P/qRhu6LgcUAZWVlcXWvv7vzjd9sIOTwf/7srJjt+Y0flc49l0/lhc0HWLnjEJv2HOWKGWM4b+KoYXHsvjPk7KlvZltNIxUHGqmqbybkXXvrk/OymFcymqkFWRRkp2rvXE7b9ecV8sctB1hRUcul04b+Zc7RBH01UBSxPAHYe4I21WaWBIwE6tzdgVYAd19jZtuBaUD56RY+VCxdv5eXt9bwvz45k6Lc2I5fkpqcyKfOGc95E3NYun4Pz72zh5feO8CFk0dzdtEocgI0EmZHKMSe+mPsqm1i56Emdh9qprUjhNH1S29BaT5TC7KYmJsxLH7RyeD62IwCRqYn8+za6sAE/Wqg1MxKgD3AzcCt3dosBW4D3gI+A7zk7m5m+XQFfqeZTQZKgR39Vn2M1TW18b9/u5nZRaO4/aLiWJfzocKcdO66dArba5p4ZetBlm8+wPLNB5iQk87O2kbOnjCKqQVZ5GWlkpuZwgd/eTa0dlDf1NZ1YrKpjUNNbdQ2tlLb0MrqXXU0tnbQ3NZJe6fTEQrREf4OkJqUSEpSAmlJCaQmJZKekkhWWhLZqUkkJRh52SnkZaWSn53K6MxUUpKiC9+QO0ePtXOwoZWDDa3UNLRw8Ggrew4foyN8kL0gO5VzJoxiSkEWU/IyydAwAzLAUpMS+dQ543h6TTUNLe1kpyXHuqSTOuUnInzM/R5gOV2XVz7s7pvM7AGg3N2XAg8BvzazCqCOrl8GAAuAB8ysA+gE7nL3+Bso4gT+6XebaWhp57vXnz3kjtOZGVMLsphakEVdUxsb9hxhy76j/PKt3bR17OzVc6Unh4M7NYmR6ckkJyaQnGgkJSSQlGg40NYeorWjk9aOEK0dIQ4faaPhQAetHSFe2HzguOcclZFMbmYKaUmJJCclkJJoJCcmYAbH2jppbutkT/0xmto6iBy4MyMlkYLsVOaW5FKSl8mk0ZkaP0Zi4s/Pm8AjKyt5fsN+bpxTdOoOMRTVJ8TdlwHLuq37ZsTjFuCGHvo9AzxzmjUOSa9sPchv1u3hy1eUMn1sdqzLOanczBQunZbPpdPy+cz5E3j/QANVdc3UNLZS39ROyB0HslOTyMlMITcz+cOrUPKyU8lMSeTxt6tO+To9ae8M8bEzCqhtbKWmoZXaxrbw91bqmtpo7ej6C6G9M0R7Z4iQd91qPjorlazUpK5fLhnJ5GenUpCdplCXIePcolFMyc/k0VW7uaFswpA+96NPTR80tnbwD7/ZyNSCLO6+fEqsy+mVlKQEziwcyZmFIwfl9ZITEyjKzejT+Yt4G+cx5oYAAApwSURBVOtGhhcz4/aLS/hfz21kze56yopzY13SCeksVR98/w/vsffIMb57/dmkJg3fSxhFhrvrzytkZHoyD63o3eHQwaag76U3t9fyy7d2c9uFxZw/SVP8iQxnGSlJ3DpvIss37aeqrjnW5ZyQgr4XGls7+NpT71KSl8nXF57eGPMiEgx/eeEkEsz4xZu7Yl3KCSnoe+Hbv9/MviPH+JcbztaUfiICwLiR6XzirHE8sbqKuqahOQOcgj5Kyzft5/G3q/jr+ZM5f9LQPekiIoPvno9Npbmtgx+9VBHrUnqkoI9CVV0zX3tqPWcVjuQrV02LdTkiMsRMG5PNjWVF/HrlLioPDb1j9Qr6U2jrCHHP4+twhwdvPU9X2YhIj/7nx6eRmGB8/4Wtp248yBT0J+HuPPC7TayvOsz3PnM2E0fHdiwbERm6xoxI46/nT+a36/eyrrI+1uV8hIL+JB5asZNHVlbyxQWTueascbEuR0SGuDsXTGbsiDT+7qn1HGvrjHU5H1LQn8DyTfv59rItXHPmWF1KKSJRyU5L5l9vPIftNU388/NbYl3OhxT0PXh560G+9Pg6zpkwin+7aTYJQ2zAMhEZui6emscdl5Twq7d28/J7B2NdDqCgP86Lmw/wxV+tobQgi5/fPmdYz9IkIn3ztaunM31MNn/7xDts3d8Q63IU9JGWvF3J3zyyhhnjR/DYFy4gJzM4E3WIyOBJS07kZ7eVkZqUwOceWhXzSy4V9HTNL/r1p9/lvmc3cOGU0fz6jrmMzBjaEwmIyNBWlJvBI1+YR1tniM8+tIqdtU0xq2XYB/3qXXUs+tEbPFFexd2XT+EXn5/LiCE+W4yIxIdpY7L5xefn0tDSznX/uYLlm/bHpI5hG/Q7a5v46pPrueEnb9HY2sHDt5fxtavjY0Z3EYkfs4tG8dsvXUJJfiZf/PUa7n92AwcbWga1hmE18UhrRydvVhxiyepKXth8gOSEBO66dApfvmIqGSnD6p9CRAbRhJwMnrrrQr77/FZ+9dYunlu3h9svLubGsiJK8jIH/PWjSjczWwj8kK45Y3/m7t/ptj0V+BVwPnAIuMndd4W33Q/cQdecsV929+X9Vv1JHGvr5MDRFqrrj7FhzxHWVx1mRUUtja0djExP5u7LpvKXF02iIDttMMoRkWEuNSmRb35qJn954SS+/8JWfvLqdn78ynbOmTCSi6bmcc6EkZxTNIpxI9P7/bVPGfRmlgg8CHwcqAZWm9lSd98c0ewOoN7dp5rZzcB3gZvMbCZdE4XPAsYDfzSzae7e77eMHWps5cLvvIS7E3LojJxRGpiYm8G1Z41j4VljuWjKaI1ZIyIxUZyXyYO3nse+I8f47fq9/H7Dfn72+g7aO52Z40aw7N75/f6a0ezRzwUq3H0HgJktARYBkUG/CPjH8OOngR9Z10y5i4Al7t4K7DSzivDzvdU/5f+39JRE/uriEswgwbpmfhkzIo3xI9OYOX4EozJ0qaSIDB3jRqZz54Ip3LlgCi3tnWzZd3TAhk2IJugLgaqI5Wpg3onauHuHmR0BRofXr+zWt7D7C5jZncCd4cVGMxsKw7/lAbWxLuI0Hfce/iIGRZzGa8bk/6Cf/42ieg+x+H/phY+8hyFe63H+IqCf5R5MOtGGaIK+p8tQPMo20fTF3RcDi6OoZdCYWbm7l8W6jtMR7+8h3usHvYehIN7rh9N/D9FcXlkNFEUsTwD2nqiNmSUBI4G6KPuKiMgAiiboVwOlZlZiZil0nVxd2q3NUuC28OPPAC+5u4fX32xmqWZWApQCb/dP6SIiEo1THroJH3O/B1hO1+WVD7v7JjN7ACh396XAQ8Cvwydb6+j6ZUC43ZN0nbjtAO4eiCtuBsiQOpTUR/H+HuK9ftB7GArivX44zfdgXTveIiISVMN2CAQRkeFCQS8iEnAK+h6Y2UIz22pmFWZ2X6zr6Q0zKzKzl81si5ltMrN7Y11TX5lZopmtM7PfxbqWvjCzUWb2tJm9F/7/uDDWNfWGmf3P8M/QRjN73MyG/HghZvawmR00s40R63LN7EUz2xb+nhPLGk/mBPV/P/wz9K6Z/cbMRvX2eRX03UQM+XANMBO4JTyUQ7zoAL7q7jOAC4C746z+SPcCQ2fizd77IfAHdz8DOIc4ei9mVgh8GShz9zPpuhDj5thWFZVfAAu7rbsP+JO7lwJ/Ci8PVb/g+PpfBM5097OB94H7e/ukCvrjfTjkg7u3AR8M+RAX3H2fu68NP26gK1yOuxt5qDOzCcC1wM9iXUtfmNkIYAFdV6Th7m3ufji2VfVaEpAevjcmgzi4B8bdX6Pryr9Ii4Bfhh//Evj0oBbVCz3V7+4vuHtHeHElXfcj9YqC/ng9DfkQd0EJYGbFwLnAqthW0if/Dvw9EIp1IX00GagBfh4+/PQzMxv48Wj7ibvvAf4FqAT2AUfc/YXYVtVnY9x9H3TtCAEFMa7ndPwV8HxvOynojxfVsA1DnZllAc8Af+vuR2NdT2+Y2SeBg+6+Jta1nIYk4Dzgx+5+LtDE0D5k8BHh49iLgBK6Rp7NNLPPxraq4c3M/oGuQ7OP9ravgv54cT9sg5kl0xXyj7r7s7Gupw8uBq4zs110HTr7mJk9EtuSeq0aqHb3D/6aepqu4I8XVwI73b3G3duBZ4GLYlxTXx0ws3EA4e8HY1xPr5nZbcAngb/wPtz8pKA/XjRDPgxZ4eGhHwK2uPsPYl1PX7j7/e4+wd2L6fr3f8nd42pv0t33A1VmNj286go+OrT3UFcJXGBmGeGfqSuIo5PJ3UQO0XIb8P9iWEuvhSd++jpwnbs39+U5FPTdhE96fDDkwxbgSXffFNuqeuVi4HN07QW/E/76RKyLGqa+BDxqZu8Cs4H/E+N6ohb+S+RpYC2wga6sGPJDCZjZ43TNdzHdzKrN7A7gO8DHzWwbXRMofedkzxFLJ6j/R0A28GL48/yTXj+vhkAQEQk27dGLiAScgl5EJOAU9CIiAaegFxEJOAW9iEjAKehFTsLM/tbMMiKWl/Vl9ECRWNLllTLshW8IMnc/blyd8N25Ze5eO+iFifQT7dHLsGRmxeEx4v+LrpuCHjKz8vD46/873ObLdI3z8rKZvRxet8vM8iL6/99wnxfMLD3cZk547PC3wmOJbzxRHSKDQUEvw9l04FfhQce+6u5lwNnApWZ2trv/B13jHF3u7pf30L8UeNDdZwGHgevD638O3OXuFwKdA/4uRE5BQS/D2W53Xxl+fKOZrQXWAbPomnTmVHa6+zvhx2uA4vDx+2x3fzO8/rF+rVikD5JiXYBIDDUBmFkJ8HfAHHevN7NfANFMm9ca8bgTSKfnYa5FYkp79CIwgq7QP2JmY+iaRvIDDXQNKBUVd68HGszsgvCqeJh+TwJOe/Qy7Ln7ejNbB2wCdgBvRGxeDDxvZvtOcJy+J3cA/9fMmoBXgCP9Wa9Ib+nySpF+ZmZZ7t4YfnwfMM7d741xWTKMaY9epP9da2b30/X52g3cHttyZLjTHr2ISMDpZKyISMAp6EVEAk5BLyIScAp6EZGAU9CLiATc/w9FEvd2QhFxpQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(Totaldata['rating'], bins=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We choose 'efficiency' to be our target and we divide the 'rating' column into two classes: $0$ if the rating is below $7$ and $1$ otherwise, which we use as one of our features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "Totaldata['rating'] = (Totaldata['rating'] >6).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{blue}{\\text{10% of the data}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(data, percentage):\n",
    "    Random=list(range(data.shape[0]))\n",
    "    np.random.shuffle(Random)\n",
    "    data=data.iloc[Random]\n",
    "    return data.iloc[[i for i in range(int(percentage*len(data)))]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "ten_percent_data=get_data(Totaldata, 0.1)\n",
    "\n",
    "ten_percent_data=ten_percent_data[['sideEffects', 'sideEffectsReview',\\\n",
    "                                   'commentsReview', 'rating', 'effectiveness']]\n",
    "rating=ten_percent_data['rating']\n",
    "ten_percent_data=ten_percent_data.drop('rating', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "Features=ten_percent_data.drop(['effectiveness'], axis=1)\n",
    "Target=ten_percent_data['effectiveness']\n",
    "Target=le().fit_transform(Target)\n",
    "\n",
    "Dat=''\n",
    "for col in Features.columns: \n",
    "    Dat += Features[col].map(str) + ' '"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "881     Moderate Side Effects Basically, I only notice...\n",
       "2356    Mild Side Effects Upset stomach, headache. Nau...\n",
       "1044    Moderate Side Effects Made me nasuous, dizzy, ...\n",
       "1004    Mild Side Effects I experienced drowsiness in ...\n",
       "1189    Moderate Side Effects Chronic constipation, dr...\n",
       "Name: sideEffects, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Dat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "Text_tranform=CountVectorizer(analyzer='word', max_features=100, stop_words='english', binary=True).fit(Dat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "Text_tranform.vocabulary_;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "Numerical_features=Text_tranform.transform(Dat)\n",
    "Numerical_features=Numerical_features.toarray()\n",
    "Data=pd.DataFrame(Numerical_features) \n",
    "\n",
    "Data=pd.DataFrame(np.concatenate([rating.values.reshape(-1,1), Data], axis=1))\n",
    "Data=pd.DataFrame(np.concatenate([Data, Target.reshape(-1,1)], axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>92</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "      <th>101</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0    1    2    3    4    5    6    7    8    9    ...  92   93   94   95   \\\n",
       "0    0    0    0    0    0    0    0    1    0    0  ...    0    0    0    1   \n",
       "1    0    0    0    0    1    0    0    0    0    0  ...    0    0    0    0   \n",
       "2    0    0    0    0    0    0    0    0    0    0  ...    0    0    0    0   \n",
       "3    0    0    1    0    0    1    0    0    0    0  ...    0    1    1    0   \n",
       "4    0    0    0    0    0    1    0    0    1    0  ...    0    1    0    0   \n",
       "\n",
       "   96   97   98   99   100  101  \n",
       "0    0    0    0    0    0    3  \n",
       "1    0    0    0    0    1    2  \n",
       "2    0    0    0    0    0    2  \n",
       "3    1    0    0    0    0    3  \n",
       "4    1    0    1    0    0    3  \n",
       "\n",
       "[5 rows x 102 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_features, Test_features, Train_target, Test_target = train_test_split(Data.iloc[:, :-1],\\\n",
    "        Data.iloc[:, -1], test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights=np.ones((len(np.unique(Train_target)), 1+Train_features.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=100, lr=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "MultiLogReg.fit(Train_features, pd.Series(Train_target), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "Predict=MultiLogReg.predict(Test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(Predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Multiclass Logistic Regression: 28.915662650602407%\n"
     ]
    }
   ],
   "source": [
    "accuracy=(Test_target==Predict).sum()/len(Test_target)\n",
    "print(\"Accuracy of Multiclass Logistic Regression: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB=Naive_Bayes()\n",
    "\n",
    "NB.train(Train_features, pd.Series(Train_target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "Pred=NB.predict(Test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of my Bernoulli Naive Bayes: 43.373493975903614%\n"
     ]
    }
   ],
   "source": [
    "accuracy=(Test_target==Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of my Bernoulli Naive Bayes: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA_Model=GDAMulticlass(regularizer=0.1)\n",
    "GDA_Model.train(pd.DataFrame(Train_features), pd.Series(Train_target, name='target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA_Pred=GDA_Model.predict(pd.DataFrame(Test_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of GDA: 45.78313253012048%\n"
     ]
    }
   ],
   "source": [
    "accuracy=(Test_target==GDA_Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of GDA: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{blue}{\\text{30% of the data}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "thirty_percent_data=get_data(Totaldata, 0.3)\n",
    "\n",
    "thirty_percent_data=thirty_percent_data[['sideEffects', 'sideEffectsReview', \\\n",
    "                                      'commentsReview', 'rating', 'effectiveness']]\n",
    "rating=thirty_percent_data['rating']\n",
    "thirty_percent_data=thirty_percent_data.drop('rating', axis=1)\n",
    "\n",
    "Features=thirty_percent_data.drop(['effectiveness'], axis=1)\n",
    "Target=thirty_percent_data['effectiveness']\n",
    "Target=le().fit_transform(Target)\n",
    "\n",
    "Dat=''\n",
    "for col in Features.columns: \n",
    "    Dat += Features[col].map(str) + ' '"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>92</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "      <th>101</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0    1    2    3    4    5    6    7    8    9   ...   92   93   94   95   \\\n",
       "0    1    0    0    0    0    0    0    0    0    0 ...     0    0    0    0   \n",
       "1    0    0    0    0    0    0    0    0    0    1 ...     0    0    0    0   \n",
       "2    1    0    0    0    0    0    0    1    0    1 ...     0    0    1    0   \n",
       "3    1    0    1    0    0    0    0    0    0    0 ...     0    0    1    1   \n",
       "4    1    0    0    0    0    0    0    0    0    0 ...     0    0    0    0   \n",
       "\n",
       "   96   97   98   99   100  101  \n",
       "0    0    0    0    0    0    4  \n",
       "1    0    0    0    0    0    3  \n",
       "2    1    0    1    1    0    1  \n",
       "3    0    0    0    0    0    1  \n",
       "4    0    0    0    0    0    1  \n",
       "\n",
       "[5 rows x 102 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Text_tranform=CountVectorizer(analyzer='word', max_features=100, stop_words='english', binary=True).fit(Dat)\n",
    "\n",
    "Text_tranform.vocabulary_;\n",
    "\n",
    "Numerical_features=Text_tranform.transform(Dat)\n",
    "Numerical_features=Numerical_features.toarray()\n",
    "Data=pd.DataFrame(Numerical_features) \n",
    "\n",
    "Data=pd.DataFrame(np.concatenate([rating.values.reshape(-1,1), Data], axis=1))\n",
    "Data=pd.DataFrame(np.concatenate([Data, Target.reshape(-1,1)], axis=1))\n",
    "\n",
    "Data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_features, Test_features, Train_target, Test_target = train_test_split(Data.iloc[:, :-1],\\\n",
    "        Data.iloc[:, -1], test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "weights=np.ones((len(np.unique(Train_target)), 1+Train_features.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=100, lr=0.3)\n",
    "\n",
    "MultiLogReg.fit(Train_features, pd.Series(Train_target), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Multiclass Logistic Regression: 44.46680080482898%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Predict=MultiLogReg.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Predict).sum()/len(Test_target)\n",
    "print(\"Accuracy of Multiclass Logistic Regression: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB=Naive_Bayes()\n",
    "\n",
    "NB.train(Train_features, pd.Series(Train_target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of my Bernoulli Naive Bayes: 43.05835010060362%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Pred=NB.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of my Bernoulli Naive Bayes: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA_Model=GDAMulticlass(regularizer=0.01)\n",
    "GDA_Model.train(pd.DataFrame(Train_features), pd.Series(Train_target, name='target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of GDA: 44.668008048289735%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "GDA_Pred=GDA_Model.predict(pd.DataFrame(Test_features))\n",
    "\n",
    "accuracy=(Test_target==GDA_Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of GDA: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{blue}{\\text{60% of the data}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "sixty_percent_data=get_data(Totaldata, 0.6)\n",
    "\n",
    "sixty_percent_data=sixty_percent_data[['sideEffects', 'sideEffectsReview', 'commentsReview', 'rating', 'effectiveness']]\n",
    "rating=sixty_percent_data['rating']\n",
    "sixty_percent_data=sixty_percent_data.drop('rating', axis=1)\n",
    "\n",
    "\n",
    "\n",
    "Features=sixty_percent_data.drop(['effectiveness'], axis=1)\n",
    "Target=sixty_percent_data['effectiveness']\n",
    "Target=le().fit_transform(Target)\n",
    "\n",
    "Dat=''\n",
    "for col in Features.columns: \n",
    "    Dat += Features[col].map(str) + ' '"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "Text_tranform=CountVectorizer(analyzer='word', max_features=100, stop_words='english', binary=True).fit(Dat)\n",
    "\n",
    "Text_tranform.vocabulary_;\n",
    "\n",
    "Numerical_features=Text_tranform.transform(Dat)\n",
    "Numerical_features=Numerical_features.toarray()\n",
    "Data=pd.DataFrame(Numerical_features) \n",
    "\n",
    "Data=pd.DataFrame(np.concatenate([rating.values.reshape(-1,1), Data], axis=1))\n",
    "Data=pd.DataFrame(np.concatenate([Data, Target.reshape(-1,1)], axis=1))\n",
    "\n",
    "Data.head();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_features, Test_features, Train_target, Test_target = train_test_split(Data.iloc[:, :-1],\\\n",
    "        Data.iloc[:, -1], test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "weights=np.ones((len(np.unique(Train_target)), 1+Train_features.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=100, lr=0.3)\n",
    "\n",
    "MultiLogReg.fit(Train_features, pd.Series(Train_target), weights)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Multiclass Logistic Regression: 46.17706237424547%\n"
     ]
    }
   ],
   "source": [
    "Predict=MultiLogReg.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Predict).sum()/len(Test_target)\n",
    "print(\"Accuracy of Multiclass Logistic Regression: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB=Naive_Bayes()\n",
    "\n",
    "NB.train(Train_features, pd.Series(Train_target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of my Bernoulli Naive Bayes: 45.87525150905433%\n"
     ]
    }
   ],
   "source": [
    "Pred=NB.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of my Bernoulli Naive Bayes: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA_Model=GDAMulticlass(regularizer=0.1)\n",
    "GDA_Model.train(pd.DataFrame(Train_features), pd.Series(Train_target, name='target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of GDA: 48.08853118712273%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "GDA_Pred=GDA_Model.predict(pd.DataFrame(Test_features))\n",
    "\n",
    "accuracy=(Test_target==GDA_Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of GDA: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{blue}{\\text{100% of the data}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "whole_data=get_data(Totaldata, 1.0)\n",
    "\n",
    "whole_data=whole_data[['sideEffects', 'sideEffectsReview', 'commentsReview', 'rating', 'effectiveness']]\n",
    "rating=whole_data['rating']\n",
    "whole_data=whole_data.drop('rating', axis=1)\n",
    "\n",
    "\n",
    "\n",
    "Features=whole_data.drop(['effectiveness'], axis=1)\n",
    "Target=whole_data['effectiveness']\n",
    "Target=le().fit_transform(Target)\n",
    "\n",
    "Dat=''\n",
    "for col in Features.columns: \n",
    "    Dat += Features[col].map(str) + ' '\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "Text_tranform=CountVectorizer(analyzer='word', max_features=100, stop_words='english', binary=True).fit(Dat)\n",
    "\n",
    "Text_tranform.vocabulary_;\n",
    "\n",
    "Numerical_features=Text_tranform.transform(Dat)\n",
    "Numerical_features=Numerical_features.toarray()\n",
    "Data=pd.DataFrame(Numerical_features) \n",
    "\n",
    "#Data=pd.DataFrame(np.concatenate([rating.values.reshape(-1,1), Data], axis=1))\n",
    "Data=pd.DataFrame(np.concatenate([Data, Target.reshape(-1,1)], axis=1))\n",
    "\n",
    "Data.head();\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_features, Test_features, Train_target, Test_target = train_test_split(Data.iloc[:, :-1],\\\n",
    "        Data.iloc[:, -1], test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "weights=np.ones((len(np.unique(Train_target)), 1+Train_features.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=100, lr=0.3)\n",
    "\n",
    "MultiLogReg.fit(Train_features, pd.Series(Train_target), weights)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Multiclass Logistic Regression: 41.61640530759952%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Predict=MultiLogReg.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Predict).sum()/len(Test_target)\n",
    "print(\"Accuracy of Multiclass Logistic Regression: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB=Naive_Bayes()\n",
    "\n",
    "NB.train(Train_features, pd.Series(Train_target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of my Bernoulli Naive Bayes: 39.56574185765983%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Pred=NB.predict(Test_features)\n",
    "\n",
    "accuracy=(Test_target==Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of my Bernoulli Naive Bayes: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA_Model=GDAMulticlass(regularizer=0.1)\n",
    "GDA_Model.train(pd.DataFrame(Train_features), pd.Series(Train_target, name='target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of GDA: 42.5211097708082%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "GDA_Pred=GDA_Model.predict(pd.DataFrame(Test_features))\n",
    "\n",
    "accuracy=(Test_target==GDA_Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of GDA: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison using the Iris data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "iris=datasets.load_iris()\n",
    "\n",
    "x=iris.data[:,:4]\n",
    "y=(iris.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(x,\\\n",
    "        y, test_size=0.8, random_state=101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "weights=np.zeros((len(np.unique(y)), 1+x.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=400, lr=0.2)\n",
    "\n",
    "MultiLogReg.fit(pd.DataFrame(x_train), pd.Series(y_train), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "Ypred=MultiLogReg.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Accuracy of Logistic regression: 97.5%'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\"Accuracy of Logistic regression: {}%\" .format(100*(Ypred==y_test).sum()/len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "NaiveB=Naive_Bayes()\n",
    "NaiveB.train(pd.DataFrame(x_train), pd.Series(y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "Y1pred=NaiveB.predict(pd.DataFrame(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Naive Bayes: 35.833333333333336%\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy of Naive Bayes: {}%' .format (100*(Y1pred==y_test).sum()/len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "GDA2=GDAMulticlass()\n",
    "GDA2.train(pd.DataFrame(x_train), pd.Series(y_train, name='target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]\n"
     ]
    }
   ],
   "source": [
    "Y2Pred=GDA2.predict(pd.DataFrame(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of GDA: 98.33333333333333%\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy of GDA: {}%\" .format(100*(Y2Pred==y_test).sum()/len(y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We download another data to compare the three models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://storage.googleapis.com/dataset-uploader/bbc/bbc-text.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tech</td>\n",
       "      <td>tv future in the hands of viewers with home th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>business</td>\n",
       "      <td>worldcom boss  left books alone  former worldc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sport</td>\n",
       "      <td>tigers wary of farrell  gamble  leicester say ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sport</td>\n",
       "      <td>yeading face newcastle in fa cup premiership s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>entertainment</td>\n",
       "      <td>ocean s twelve raids box office ocean s twelve...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        category                                               text\n",
       "0           tech  tv future in the hands of viewers with home th...\n",
       "1       business  worldcom boss  left books alone  former worldc...\n",
       "2          sport  tigers wary of farrell  gamble  leicester say ...\n",
       "3          sport  yeading face newcastle in fa cup premiership s...\n",
       "4  entertainment  ocean s twelve raids box office ocean s twelve..."
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Text=pd.read_csv(\"bbc-text.csv\")\n",
    "Text.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tv future in the hands of viewers with home theatre systems  plasma high-definition tvs  and digital video recorders moving into the living room  the way people watch tv will be radically different in five years  time.  that is according to an expert panel which gathered at the annual consumer electronics show in las vegas to discuss how these new technologies will impact one of our favourite pastimes. with the us leading the trend  programmes and other content will be delivered to viewers via home networks  through cable  satellite  telecoms companies  and broadband service providers to front rooms and portable devices.  one of the most talked-about technologies of ces has been digital and personal video recorders (dvr and pvr). these set-top boxes  like the us s tivo and the uk s sky+ system  allow people to record  store  play  pause and forward wind tv programmes when they want.  essentially  the technology allows for much more personalised tv. they are also being built-in to high-definition tv sets  which are big business in japan and the us  but slower to take off in europe because of the lack of high-definition programming. not only can people forward wind through adverts  they can also forget about abiding by network and channel schedules  putting together their own a-la-carte entertainment. but some us networks and cable and satellite companies are worried about what it means for them in terms of advertising revenues as well as  brand identity  and viewer loyalty to channels. although the us leads in this technology at the moment  it is also a concern that is being raised in europe  particularly with the growing uptake of services like sky+.  what happens here today  we will see in nine months to a years  time in the uk   adam hume  the bbc broadcast s futurologist told the bbc news website. for the likes of the bbc  there are no issues of lost advertising revenue yet. it is a more pressing issue at the moment for commercial uk broadcasters  but brand loyalty is important for everyone.  we will be talking more about content brands rather than network brands   said tim hanlon  from brand communications firm starcom mediavest.  the reality is that with broadband connections  anybody can be the producer of content.  he added:  the challenge now is that it is hard to promote a programme with so much choice.   what this means  said stacey jolna  senior vice president of tv guide tv group  is that the way people find the content they want to watch has to be simplified for tv viewers. it means that networks  in us terms  or channels could take a leaf out of google s book and be the search engine of the future  instead of the scheduler to help people find what they want to watch. this kind of channel model might work for the younger ipod generation which is used to taking control of their gadgets and what they play on them. but it might not suit everyone  the panel recognised. older generations are more comfortable with familiar schedules and channel brands because they know what they are getting. they perhaps do not want so much of the choice put into their hands  mr hanlon suggested.  on the other end  you have the kids just out of diapers who are pushing buttons already - everything is possible and available to them   said mr hanlon.  ultimately  the consumer will tell the market they want.   of the 50 000 new gadgets and technologies being showcased at ces  many of them are about enhancing the tv-watching experience. high-definition tv sets are everywhere and many new models of lcd (liquid crystal display) tvs have been launched with dvr capability built into them  instead of being external boxes. one such example launched at the show is humax s 26-inch lcd tv with an 80-hour tivo dvr and dvd recorder. one of the us s biggest satellite tv companies  directtv  has even launched its own branded dvr at the show with 100-hours of recording capability  instant replay  and a search function. the set can pause and rewind tv for up to 90 hours. and microsoft chief bill gates announced in his pre-show keynote speech a partnership with tivo  called tivotogo  which means people can play recorded programmes on windows pcs and mobile devices. all these reflect the increasing trend of freeing up multimedia so that people can watch what they want  when they want.'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Text.text[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "Text_tranformer=CountVectorizer(analyzer='word', max_features=400, binary=True).fit(Text['text'])\n",
    "\n",
    "bow1=Text_tranformer.transform(Text['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "Data=pd.DataFrame(data=bow1.toarray())\n",
    "Data.head()\n",
    "Text['category']=le().fit_transform(Text['category'])\n",
    "Data['target']=Text['category']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_features, Test_features, Train_target, Test_target = train_test_split(Data.iloc[:, :-1],\\\n",
    "        Data.iloc[:, -1], test_size=0.4, random_state=101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Multiclass\\ Logistic\\ Regression}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "weights=np.ones((len(np.unique(Train_target)), 1+Train_features.shape[1]))\n",
    "\n",
    "MultiLogReg=MulticlassLogReg(Max_iter=100, lr=0.2)\n",
    "\n",
    "MultiLogReg.fit(Train_features, Train_target, weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the Logistic-Regression:  93.14606741573034\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Predict=MultiLogReg.predict(Test_features)\n",
    "AccuracyLogRegression=(Predict==Test_target).sum()/(len(Test_target))\n",
    "print(\"Accuracy of the Logistic-Regression: \", AccuracyLogRegression*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Bernoulli\\ Naive\\ Bayes}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB=Naive_Bayes()\n",
    "\n",
    "NB.train(Train_features, Train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Naive Bayes: 86.62921348314607%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "Pred=NB.predict(Test_features)\n",
    "\n",
    "accuracy=(np.array(Test_target)==Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy Naive Bayes: {}%\".format(100*accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{blue}{Gaussian\\ Discriminant\\ Analysis}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  training: [=======================================100.0%=======================================]\n"
     ]
    }
   ],
   "source": [
    "MulticlassGDA=GDAMulticlass()\n",
    "\n",
    "MulticlassGDA.train(Train_features, Train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  predictions: [=====================================100.0%======================================]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the GDA:  93.37078651685393\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "GDA_Pred=MulticlassGDA.predict(Test_features)\n",
    "\n",
    "AccuracyGDA=(np.array(Test_target)==GDA_Pred).sum()/len(Test_target)\n",
    "print(\"Accuracy of the GDA: \", AccuracyGDA*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\color{green}{\\text{Report}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three algorithms GDA, Naive Bayes and Softmax Logistic Regression were compared on three different data sets: 'drugLib_raw.tsv', Iris data set and 'bbc-text.csv'.\n",
    "The comparison on the first data set 'drugLib_raw.tsv' was done in four steps: comparison on $10\\%$, $30\\%$, $60\\%$ and finally on $100\\%$ of the data. On these four comparisons, GDA appeared to be the most accurate (even though the accuracy was around $45\\%$ in average), followed very closely by Softmax Logistic Rgression. But it is important to underline that Naive Bayes and GDA are very fast (during the training process) as compared to Softmax Logistic Regression. This is due to the fact that GDA and Naive Bayes are both generative models and don't require any learning algorithm, their parameters are directly computed, no update required. On the other side, the low accuracy observed on the three algorithms is probably due to the fact that the features used are not predictive enough for the target chosen.\n",
    "\n",
    "\n",
    "\n",
    "On Iris data set (small data set), very good accuracy was observed but only on two of the three algorithms. Again, GDA and Softmax Logistic Regression were the best ($98.33\\%$ vs $97.5\\%$), whereas Naive Bayes gave $35.83\\%$ accuracy. The failure of Naive Bayes (actually Bernoulli Naive Bayes) on this example is due to the fact that features were not binary.\n",
    "\n",
    "On the third data set, which also is as big as the first one, we observed good accuracy for the three algorithms. On this particular data set, Softmax was slightly less accurate as compared to GDA ($93.15\\%$ vs $93.37\\%$) and Naive Bayes scored $86.63\\%$.\n",
    "\n",
    "\n",
    "In conclusion, each of the three algorithms is unique and reliable depending on which data we are investigating. The two generative models GDA and Naive Bayes are fast but can give bad results if the (strong) assumptions they are made of are not met. On the other side, Softmax Logistic Regression takes time to train, but is more stable as it uses fewer assumptions and can handle both binary and non-binary features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}