Spam_Or_Ham.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Being able to identify spam messages is a binary classification problem as messages are classified as either 'Spam' or 'Not Spam' and nothing else. Also, this is a supervised learning problem, as we will be feeding a labelled dataset into the model, that it can learn from, to make future predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### In the dataset. the first column takes two values, 'ham' which signifies that the message is not spam, and 'spam' which signifies that the message is spam. The second column is the text content of the SMS message that is being classified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PREPARE DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No.of rows: 5572\n",
      "No.of columns: (5572, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>sms_message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                        sms_message\n",
       "0   ham  Go until jurong point, crazy.. Available only ...\n",
       "1   ham                      Ok lar... Joking wif u oni...\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3   ham  U dun say so early hor... U c already then say...\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_table('Data/SMSSpamCollection', \n",
    "                   sep='\\t', \n",
    "                   header=None, \n",
    "                   names=['label', 'sms_message'])\n",
    "\n",
    "print('No.of rows:', df.shape[0])\n",
    "print('No.of columns:', df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preptocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert string label spam and ham into numerical value (Spam:1, Ham:0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5572, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>sms_message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label                                        sms_message\n",
       "0      0  Go until jurong point, crazy.. Available only ...\n",
       "1      0                      Ok lar... Joking wif u oni...\n",
       "2      1  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      0  U dun say so early hor... U c already then say...\n",
       "4      0  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['label'] = df.label.map({'ham':0, 'spam':1})\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we'd like to introduce the Bag of Words(BoW) concept which is a term used to specify the problems that have a 'bag of words' or a collection of text data that needs to be worked with. \n",
    "\n",
    "The basic idea of BoW is to take a piece of text and count the frequency of the words in that text. It is important to note that the BoW concept treats each word individually and the order in which the words occur does not matter. \n",
    "\n",
    "Using a process which we will go through now, we can covert a collection of documents to a matrix, with each document being a row and each word(token) being the column, and the corresponding (row,column) values being the frequency of occurrance of each word or token in that document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here as we can see, the documents are numbered in the rows, and each word is a column name, with the corresponding value being the frequency of that word in the document.\n",
    "\n",
    "Lets break this down and see how we can do this conversion using a small set of documents.\n",
    "\n",
    "To handle this, we will be using sklearns COUNT VECTORIZER method which does the following:\n",
    "\n",
    "* It tokenizes the string(separates the string into individual words) and gives an integer ID to each token.\n",
    "* It counts the occurrance of each of those tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Please Note: ** \n",
    "\n",
    "* The CountVectorizer method automatically converts all tokenized words to their lower case form so that it does not treat words like 'He' and 'he' differently. It does this using the lowercase parameter which is by default set to True.\n",
    "\n",
    "* It also ignores all punctuation so that words followed by a punctuation mark (for example: 'hello!') are not treated differently than the same words not prefixed or suffixed by a punctuation mark (for example: 'hello'). It does this using the token_pattern parameter which has a default regular expression which selects tokens of 2 or more alphanumeric characters.\n",
    "\n",
    "* The third parameter to take note of is the stop_words parameter. Stop words refer to the most commonly used words in a language. They include words like 'am', 'an', 'and', 'the' etc. By setting this parameter value to english, CountVectorizer will automatically ignore all words(from our input text) that are found in the built in list of english stop words in scikit-learn. This is extremely helpful as stop words can skew our calculations when we are trying to find certain key words that are indicative of spam."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing Bag of Words from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ** Step 1: Convert all strings to their lower case form. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = ['Hello, how are you!',\n",
    "             'Win money, win from home.',\n",
    "             'Call me now.',\n",
    "             'Hello, Call hello you tomorrow?']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello, how are you!', 'win money, win from home.', 'call me now.', 'hello, call hello you tomorrow?']\n"
     ]
    }
   ],
   "source": [
    "lower_case_documents = []\n",
    "for i in documents:\n",
    "    lower_case_documents.append(i.lower())\n",
    "print(lower_case_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ** Step 2: Removing all punctuations **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello how are you', 'win money win from home', 'call me now', 'hello call hello you tomorrow']\n"
     ]
    }
   ],
   "source": [
    "sans_punctuation_documents = []\n",
    "import string\n",
    "\n",
    "for i in lower_case_documents:\n",
    "    sans_punctuation_documents.append(i.translate(str.maketrans('', '', string.punctuation)))\n",
    "print(sans_punctuation_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ** Step 3: Tokenization **\n",
    "\n",
    "* Tokenizing a sentence in a document set means splitting up a sentence into individual words using a delimiter. The delimiter specifies what character we will use to identify the beginning and the end of a word(for example we could use a single space as the delimiter for identifying words in our document set.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['hello', 'how', 'are', 'you'], ['win', 'money', 'win', 'from', 'home'], ['call', 'me', 'now'], ['hello', 'call', 'hello', 'you', 'tomorrow']]\n"
     ]
    }
   ],
   "source": [
    "preprocessed_documents = []\n",
    "\n",
    "for i in sans_punctuation_documents:\n",
    "    preprocessed_documents.append(i.split(' '))\n",
    "print(preprocessed_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ** Step 4: Count frequencies **\n",
    "\n",
    "Now that we have our document set in the required format, we can proceed to counting the occurrence of each word in each document of the document set (or counting the occurence of each word of a sms message). We will use the counter() method from the Python collections library for this purpose. \n",
    "\n",
    "* counter() counts the occurrence of each item in the list and returns a dictionary with the key as the item being counted and the corresponding value being the count of that item in the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Counter({'hello': 1, 'how': 1, 'are': 1, 'you': 1}),\n",
      " Counter({'win': 2, 'money': 1, 'from': 1, 'home': 1}),\n",
      " Counter({'call': 1, 'me': 1, 'now': 1}),\n",
      " Counter({'hello': 2, 'call': 1, 'you': 1, 'tomorrow': 1})]\n"
     ]
    }
   ],
   "source": [
    "frequency_list = []\n",
    "import pprint\n",
    "from collections import Counter\n",
    "\n",
    "for i in preprocessed_documents:\n",
    "    frequency_counts = Counter(i)\n",
    "    frequency_list.append(frequency_counts)\n",
    "pprint.pprint(frequency_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have implemented the Bag of Words process from scratch! As we can see in our previous output, we have a frequency distribution dictionary which gives a clear view of the text that we are dealing with.\n",
    "\n",
    " \n",
    "#### We will now implement sklearn.feature_extraction.text.CountVectorizer method in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing Bag of Words in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have implemented the BoW concept from scratch, let's go ahead and use scikit-learn to do this process in a clean and succinct way. We will use the same document set as we used in the previous step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = ['Hello, how are you!', 'Win money, win from home.', 'Call me now.', 'Hello, Call hello you tomorrow?']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vector = CountVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we implemented a version of the CountVectorizer() method from scratch that entailed cleaning our data first. This cleaning involved converting all of our data to lower case and removing all punctuation marks. CountVectorizer() has certain parameters which take care of these steps for us. They are:\n",
    "\n",
    "* lowercase = True\n",
    "\n",
    "The lowercase parameter has a DEFAULT value of True which converts all of our text to its lower case form.\n",
    "\n",
    "* token_pattern = (?u)\\\\b\\\\w\\\\w+\\\\b\n",
    "\n",
    "The token_pattern parameter has a DEFAULT regular expression value of (?u)\\\\b\\\\w\\\\w+\\\\b which ignores all punctuation marks and treats them as delimiters, while accepting alphanumeric strings of length greater than or equal to 2, as individual tokens or words.\n",
    "\n",
    "* stop_words\n",
    "\n",
    "The stop_words parameter, if set to english will remove all words from our document set that match a list of English stop words which is defined in scikit-learn. Considering the size of our dataset and the fact that we are dealing with SMS messages and not larger text sources like e-mail, we will not be setting this parameter value.\n",
    "\n",
    "##### We can have a look at all the parameter values of our count_vector object by simply printing out the object as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
      "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
      "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
      "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
      "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
      "        tokenizer=None, vocabulary=None)\n"
     ]
    }
   ],
   "source": [
    "print(count_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will fit our document dataset to the CountVectorizer object we have created using fit(), and get the list of words which have been categorized as features using the get_feature_names() method.\n",
    "\n",
    "The get_feature_names() method returns our feature names for this dataset, which is the set of words that make up our vocabulary for 'documents'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['are',\n",
       " 'call',\n",
       " 'from',\n",
       " 'hello',\n",
       " 'home',\n",
       " 'how',\n",
       " 'me',\n",
       " 'money',\n",
       " 'now',\n",
       " 'tomorrow',\n",
       " 'win',\n",
       " 'you']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vector.fit(documents)\n",
    "count_vector.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will create a matrix with the rows being each of the 4 documents (or messages), and the columns being each word (of message). The corresponding (row, column) value is the frequency of occurrance of that word(in the column) in a particular document(in the row). We can do this using the transform() method and passing in the document data set as the argument. \n",
    "\n",
    "* The transform() method returns a matrix of numpy integers, you can convert this to an array using toarray(). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1],\n",
       "       [0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 2, 0],\n",
       "       [0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0],\n",
       "       [0, 1, 0, 2, 0, 0, 0, 0, 0, 1, 0, 1]], dtype=int64)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_array = count_vector.transform(documents).toarray()\n",
    "doc_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a clean representation of the documents in terms of the frequency distribution of the words in them. To make it easier to understand our next step is to convert this array into a dataframe and name the columns appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>are</th>\n",
       "      <th>call</th>\n",
       "      <th>from</th>\n",
       "      <th>hello</th>\n",
       "      <th>home</th>\n",
       "      <th>how</th>\n",
       "      <th>me</th>\n",
       "      <th>money</th>\n",
       "      <th>now</th>\n",
       "      <th>tomorrow</th>\n",
       "      <th>win</th>\n",
       "      <th>you</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   are  call  from  hello  home  how  me  money  now  tomorrow  win  you\n",
       "0    1     0     0      1     0    1   0      0    0         0    0    1\n",
       "1    0     0     1      0     1    0   0      1    0         0    2    0\n",
       "2    0     1     0      0     0    0   1      0    1         0    0    0\n",
       "3    0     1     0      2     0    0   0      0    0         1    0    1"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frequency_matrix = pd.DataFrame(doc_array, columns = count_vector.get_feature_names())\n",
    "\n",
    "frequency_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### We have successfully implemented a Bag of Words problem for a document dataset that we created. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Train and test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total no. of data points: 5572\n",
      "No. of data points in training dataset: 3900\n",
      "No. of data points in testing dataset: 1672\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['sms_message'], \n",
    "                                                    df['label'], \n",
    "                                                    random_state=1,\n",
    "                                                   test_size=0.3)\n",
    "\n",
    "print('Total no. of data points: {}'.format(df.shape[0]))\n",
    "print('No. of data points in training dataset: {}'.format(X_train.shape[0]))\n",
    "print('No. of data points in testing dataset: {}'.format(X_test.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Applying Bag of Words to our dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have split the data, our next objective is to follow the steps from Bag of words implemented earlier and convert our data into the desired matrix format. To do this we will be using CountVectorizer() as we did before. There are two steps to consider here:\n",
    "\n",
    "* Firstly, we have to fit our training data (X_train) into CountVectorizer() and return the matrix.\n",
    "\n",
    "* Secondly, we have to transform our testing data (X_test) to return the matrix. \n",
    "\n",
    "Note that X_train is our training data for the 'sms_message' column in our dataset and we will be using this to train our model. \n",
    "\n",
    "X_test is our testing data for the 'sms_message' column and this is the data we will be using(after transformation to a matrix) to make predictions on. We will then compare those predictions with y_test in a later step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Instantiate the CountVectorizer method\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vector = CountVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Fit the training data and then return the matrix (Conerting text dataset (messages) as matrix of word frequency)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "(3900, 7155)\n"
     ]
    }
   ],
   "source": [
    "training_data = (count_vector.fit_transform(X_train)).toarray()\n",
    "print(training_data)\n",
    "print(training_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Transform testing data and return the matrix. \n",
    "\n",
    "* Note we are not fitting the testing data into the CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "(1672, 7155)\n"
     ]
    }
   ],
   "source": [
    "testing_data = (count_vector.transform(X_test)).toarray()\n",
    "print(testing_data)\n",
    "print(testing_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instantiate our model\n",
    "naive_bayes = MultinomialNB()\n",
    "\n",
    "# Fit our model to the training data\n",
    "naive_bayes.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict on the test data\n",
    "predictions = naive_bayes.predict(testing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Evaluate the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score:  0.9874401913875598\n",
      "Precision score:  0.9728506787330317\n",
      "Recall score:  0.9347826086956522\n",
      "F1 score:  0.9534368070953437\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy score: ', format(accuracy_score(y_test, predictions)))\n",
    "print('Precision score: ', format(precision_score(y_test, predictions)))\n",
    "print('Recall score: ', format(recall_score(y_test, predictions)))\n",
    "print('F1 score: ', format(f1_score(y_test, predictions)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### In general, there is a five step process that can be used each type you want to use a supervised learning method (which you actually used above):\n",
    "\n",
    "1. Import the model.\n",
    "2. Instantiate the model with the hyperparameters of interest.\n",
    "3. Fit the model to the training data.\n",
    "4. Predict on the test data.\n",
    "5. Evaluate the Score of the model by comparing the predictions to the actual values.\n",
    "\n",
    "Follow the steps through this notebook to perform these steps using each of the ensemble methods: \n",
    "##### BaggingClassifier, RandomForestClassifier, AdaBoostClassifier and SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 1: Import all 3 models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Bagging, RandomForest, and AdaBoost Classifiefrom sklearn\n",
    "\n",
    "from sklearn import ensemble\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn import svm\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 2: Instantiate each model with hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a BaggingClassifier with:\n",
    "# 200 weak learners (n_estimators) and everything else as default values\n",
    "\n",
    "Bag = BaggingClassifier(n_estimators=200)\n",
    "\n",
    "# Instantiate a RandomForestClassifier with:\n",
    "# 200 weak learners (n_estimators) and everything else as default values\n",
    "\n",
    "RF = RandomForestClassifier(n_estimators=200)\n",
    "\n",
    "# Instantiate an a AdaBoostClassifier with:\n",
    "# With 300 weak learners (n_estimators) and a learning_rate of 0.2\n",
    "\n",
    "ADBoost = AdaBoostClassifier(n_estimators=300, learning_rate=0.2)\n",
    "\n",
    "# Instantiate an a SVM with default parameter values:\n",
    "SVM = SVC()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 3: Fit each model with training_data and y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingClassifier(base_estimator=None, bootstrap=True,\n",
       "         bootstrap_features=False, max_features=1.0, max_samples=1.0,\n",
       "         n_estimators=200, n_jobs=1, oob_score=False, random_state=None,\n",
       "         verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit your BaggingClassifier to the training data\n",
    "Bag.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=200, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit your RandomForestClassifier to the training data\n",
    "RF.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "          learning_rate=0.2, n_estimators=300, random_state=None)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit your AdaBoostClassifier to the training data\n",
    "ADBoost.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit your SVM to the training data\n",
    "SVM.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 4: Predict for each model using testing_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict using BaggingClassifier on the test data\n",
    "y_Bag = Bag.predict(testing_data)\n",
    "\n",
    "# Predict using RandomForestClassifier on the test data\n",
    "y_RF = RF.predict(testing_data)\n",
    "\n",
    "# Predict using AdaBoostClassifier on the test data\n",
    "y_ADBoost = ADBoost.predict(testing_data)\n",
    "\n",
    "# Predict using SVM on the test data\n",
    "y_SVM = SVM.predict(testing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 5: Evaluate score (how well each of your models is performing) for each model by comparing prediction to actual values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_metrics(y_true, preds, model_name=None):\n",
    "    '''\n",
    "    INPUT:\n",
    "    y_true = y_test = the y values that are actually true in the dataset (numpy array or pandas series)\n",
    "    preds = predicted value (y_Bag, y_RF, y_ADBoost) = the predictions for those values from some model (numpy array or pandas series)\n",
    "    model_name = BaggingClassifier/RandomForestClassifier/AdaBoostClassifier = a name associated with the model if you would like to add it to the print statements \n",
    "    \n",
    "    OUTPUT:\n",
    "    prints the accuracy, precision, recall, and F1 score\n",
    "    '''\n",
    "    if model_name == None:\n",
    "        print('Accuracy score: ', format(accuracy_score(y_true, preds)))\n",
    "        print('Precision score: ', format(precision_score(y_true, preds)))\n",
    "        print('Recall score: ', format(recall_score(y_true, preds)))\n",
    "        print('F1 score: ', format(f1_score(y_true, preds)))\n",
    "        print('\\n\\n')\n",
    "    \n",
    "    else:\n",
    "        print('Accuracy score for ' + model_name + ' :' , format(accuracy_score(y_true, preds)))\n",
    "        print('Precision score ' + model_name + ' :', format(precision_score(y_true, preds)))\n",
    "        print('Recall score ' + model_name + ' :', format(recall_score(y_true, preds)))\n",
    "        print('F1 score ' + model_name + ' :', format(f1_score(y_true, preds)))\n",
    "        print('\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score for Bagging Classifier : 0.97188995215311\n",
      "Precision score Bagging Classifier : 0.9216589861751152\n",
      "Recall score Bagging Classifier : 0.8695652173913043\n",
      "F1 score Bagging Classifier : 0.8948545861297539\n",
      "\n",
      "\n",
      "\n",
      "Accuracy score for Randomforest Classifier : 0.9784688995215312\n",
      "Precision score Randomforest Classifier : 0.98989898989899\n",
      "Recall score Randomforest Classifier : 0.8521739130434782\n",
      "F1 score Randomforest Classifier : 0.9158878504672896\n",
      "\n",
      "\n",
      "\n",
      "Accuracy score for AdaBoost Classifier : 0.9760765550239234\n",
      "Precision score AdaBoost Classifier : 0.9611650485436893\n",
      "Recall score AdaBoost Classifier : 0.8608695652173913\n",
      "F1 score AdaBoost Classifier : 0.9082568807339451\n",
      "\n",
      "\n",
      "\n",
      "Accuracy score for Naive Bayes Classifier : 0.9874401913875598\n",
      "Precision score Naive Bayes Classifier : 0.9728506787330317\n",
      "Recall score Naive Bayes Classifier : 0.9347826086956522\n",
      "F1 score Naive Bayes Classifier : 0.9534368070953437\n",
      "\n",
      "\n",
      "\n",
      "Accuracy score for SVM : 0.8624401913875598\n",
      "Precision score for SVM : 0.8624401913875597\n",
      "Recall score for SVM : 1.0\n",
      "F1 score for SVM : 0.9261400128452152\n"
     ]
    }
   ],
   "source": [
    "# Print Bagging scores\n",
    "print_metrics(y_true=y_test, preds=y_Bag, model_name=\"Bagging Classifier\")\n",
    "\n",
    "# Print Random Forest scores\n",
    "print_metrics(y_true=y_test, preds=y_RF, model_name=\"Randomforest Classifier\")\n",
    "\n",
    "# Print AdaBoost scores\n",
    "print_metrics(y_true=y_test, preds=y_ADBoost, model_name=\"AdaBoost Classifier\")\n",
    "\n",
    "# Naive Bayes Classifier scores\n",
    "print_metrics(y_true=y_test, preds=predictions, model_name=\"Naive Bayes Classifier\")\n",
    "\n",
    "# SVM Classifier scores\n",
    "\n",
    "print('Accuracy score for SVM :' , format(accuracy_score(y_test, y_SVM)))\n",
    "print('Precision score for SVM :', format(precision_score(y_test, y_SVM, average= 'weighted', labels=np.unique(y_SVM))))\n",
    "print('Recall score for SVM :', format(recall_score(y_test, y_SVM, average= 'weighted', labels=np.unique(y_SVM))))\n",
    "print('F1 score for SVM :', format(f1_score(y_test, y_SVM, average= 'weighted', labels=np.unique(y_SVM))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}