Multi label clasification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "#### Steps\n",
    "1. All the data and categories are copied to their respective arrays(test and training data are already split in corpus).\n",
    "2. A set of features are selected on the training data.\n",
    "    - Replace digits with character ' NN '\n",
    "    - Removed Punctuations\n",
    "    - Filtered Close class words from the documents using POS tag.\n",
    "3. Feature Vector is generated from the extracted features in the previous step using CountVectorizer\n",
    "4. Vectorize the category list using Multi Lable Binarizer\n",
    "5. Extrinsic Evaluation of multiple classifiers from Scikit-learn algorithm cheat-sheet and those taught in class.\n",
    "    - Selected the one with the best score.\n",
    "6. Training one classifier per class on training set.\n",
    "7. Predict tags from the trained clasifiers on Test set.\n",
    "8. Performance analysis on the test set.\n",
    "    - Accuracy, Precision, Recall and F1-score for combined classifier and per class(tag).\n",
    "    - Plot confusion matrix for any class user enters.\n",
    "9. Pipeline - Predict tag for any data user enters\n",
    "    - Copy the data in the string inp_text\n",
    "10. Conclusion and Outlook into future work.\n",
    "\n",
    "#### Note  \n",
    "Cross validaton is skipped because\n",
    "- Only k-fold cross validation is supported in sklearn for multi value classification problems which split the training set randomly\n",
    "    - This can cause the problem of sampling errors that we are trying to avoid\n",
    "- The second approach \"stratified kfold\" which preserves the percentage of samples for each class, does not support multivalue problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing required modules and packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "\n",
    "try:\n",
    "    import sklearn\n",
    "except ModuleNotFoundError:\n",
    "    !pip install scikit-learn   \n",
    "\n",
    "try:\n",
    "    import nltk\n",
    "except ModuleNotFoundError:\n",
    "    !pip install nltk\n",
    "\n",
    "try:\n",
    "    import numpy as np\n",
    "except ModuleNotFoundError:\n",
    "    !pip install numpy\n",
    "\n",
    "try:\n",
    "    import matplotlib.pyplot as plt\n",
    "except ModuleNotFoundError:\n",
    "    !pip install matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This code downloads the required packages.\n",
    "## You can run `nltk.download('all')` to download everything.\n",
    "nltk_packages = [\n",
    "    (\"reuters\", \"corpora/reuters.zip\"),\n",
    "]\n",
    "\n",
    "for pid, fid in nltk_packages:\n",
    "    try:\n",
    "        nltk.data.find(fid)\n",
    "    except LookupError:\n",
    "        nltk.download(pid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import reuters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copying Train and Test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_documents     = [(reuters.raw(i)) for i in reuters.fileids() if i.startswith('training/')]\n",
    "train_categories    = [(reuters.categories(i)) for i in reuters.fileids() if i.startswith('training/')]\n",
    "test_documents      = [(reuters.raw(i)) for i in reuters.fileids() if i.startswith('test/')]\n",
    "test_categories     = [(reuters.categories(i)) for i in reuters.fileids() if i.startswith('test/')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_categories      = sorted(list(set(reuters.categories())))\n",
    "print(\"Data Extracted from corpus !!!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction\n",
    "#### All the numbers and dates are replace by 'NN'\n",
    "- As different values of dates or numbers will not be a relevant feature in our problem\n",
    "- Replacing all with same character might contribute to some tags.\n",
    "\n",
    "#### Remove the puntuations.\n",
    "- Add no semantical meaning to a sentence.\n",
    "\n",
    "#### Filter out close class words.\n",
    "- Close class words are related to syntactics of a sentence, rather than the semantics.\n",
    "- Hence, Open class words(Nouns, Verbs and Adjectives) are selected as features for the classifier.\n",
    "\n",
    "###### Note -  Tried upvoting(copying title twice) but didn't had a positive effect on F1-score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#--------------------------------------------------------------------------------------#\n",
    "# Using regular expression to capture all the digits(decimal, float, dates, years)\n",
    "# Encoding all the ocurances of the found instances with ' NN '\n",
    "for i in range (len(train_documents)):\n",
    "    number = re.findall(r'[0-9]+[,/]?[\\.0-9]*', train_documents[i])\n",
    "    num_cnt = len(number) \n",
    "    if(num_cnt > 0):\n",
    "        for j in range (num_cnt):\n",
    "            big_regex = re.compile('|'.join(map(re.escape, number)))\n",
    "            train_documents[i] = big_regex.sub(\" NN \", train_documents[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#--------------------------------------------------------------------------------------#\n",
    "from nltk.tokenize import word_tokenize\n",
    "#--------------------------------------------------------------------------------------#\n",
    "puntuation_list = set(string.punctuation)\n",
    "tokens_filtered = []\n",
    "for i in range (len(train_documents)):\n",
    "    doc_tokens = word_tokenize(train_documents[i])\n",
    "    #----------------------------------------------------------------------------------#\n",
    "    rem_puntuations = [t for t in doc_tokens if t not in puntuation_list]\n",
    "    #----------------------------------------------------------------------------------#\n",
    "    add_pos = nltk.pos_tag(rem_puntuations, tagset='universal')\n",
    "    #----------------------------------------------------------------------------------#\n",
    "    ## Nouns also capture the character NN we generated to represent digits.\n",
    "    collect_nouns = [(t, pos) for t, pos in add_pos if pos.startswith('NOUN')]\n",
    "    collect_verbs = [(t, pos) for t, pos in add_pos if pos.startswith('VERB')]\n",
    "    collect_adjectives = [(t, pos) for t, pos in add_pos if pos.startswith('ADJ')]\n",
    "    #----------------------------------------------------------------------------------#\n",
    "    tokens_filtered.append(collect_nouns + collect_verbs + collect_adjectives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#--------------------------------------------------------------------------------------#\n",
    "# Filtering out the POS tags from the list of selected features\n",
    "word_features = []\n",
    "for i in range (len(train_documents)):\n",
    "    doc_features = ' '.join([w for w, _ in tokens_filtered[i]])\n",
    "    word_features.append(doc_features)\n",
    "print(\"Features Extracted !!!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying Multi Label Binarizer on categories \n",
    "- It returns a 2D array, each row representing a document\n",
    "- It generates an array of all the unique categories \n",
    "- Assigns a value '1' to a particular class if it belongs to a document, otherwise '0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "\n",
    "mlb = MultiLabelBinarizer()\n",
    "Y_train = mlb.fit_transform(train_categories)\n",
    "Y_test  = mlb.fit_transform(test_categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CountVectorizer on data\n",
    "- Similar to Multi Label Binarizer, CountVectorizer returns a 2D array, each row representing a document\n",
    "- It generates a vocabulary of all the unique words using fit()\n",
    "- Then assign the count of each word in particular document, i.e. assigning importance of that word in the document\n",
    "\n",
    "###### Note - TfidVectorizer is advantageous to filter out the common words in most of the documents, but we have already filtered out those words in feature extraction so it makes sense to use the count value of the words that appear in the document \n",
    "- CountVectorizer gave better accuracy, than TfidVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer(\n",
    "                analyzer='word',\n",
    "                ngram_range=(1, 1),\n",
    "                stop_words='english',                        \n",
    "                max_df=1.0,\n",
    "                min_df=1,\n",
    "                max_features=None)\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Generating the vocabulary\n",
    "count_vect.fit(word_features + test_documents)\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Transforming the features to vectors from the vocabulary\n",
    "X_train = count_vect.transform(word_features)\n",
    "X_test  = count_vect.transform(test_documents)\n",
    "#--------------------------------------------------------------------------------------#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting the classifier\n",
    "- As explained in the lecture, with reasonable amount of data, SVM, Logistic Regression and Decision trees are better. \n",
    "- It is proven here as Naive Bayes had the worst score amoung SGDC, Decision tree and SVM. \n",
    "- Decision tree and Linear SVC(SVM) were similar but Linear SVC are very fast in comparison to Decision trees.\n",
    "\n",
    "## Training the classifier\n",
    "- OrderedDict -  remember the order that items were inserted.\n",
    "- A classifier is trained for each category and then inserted in an OrderedDict arranged in alphabetic order of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "from collections import OrderedDict\n",
    "\n",
    "clfs = OrderedDict()\n",
    "for i, category in enumerate(all_categories):\n",
    "    clf = LinearSVC()\n",
    "    #--------------------------------------------------------------------------------------#\n",
    "    # We train each classifier individually\n",
    "    # So used each column in Y_train vector\n",
    "    y_train_clf = [yt[i] for yt in Y_train]\n",
    "    #--------------------------------------------------------------------------------------#\n",
    "    # fit() will train the model with the training data\n",
    "    clf.fit(X_train, y_train_clf)\n",
    "\n",
    "    clfs[category] = clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = np.zeros((len(Y_test), len(all_categories)))\n",
    "for i, (cat, clf) in enumerate(clfs.items()):\n",
    "    Y_pred[:, i] = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance Measurements\n",
    "- Reuters data set is very skewed. \n",
    "    - There are 90 categoties and 10788 documents and for equal distribution there should have been 120 docs/category, but less than 10% categories satisify the condition.\n",
    "    - The most common category contributes 36.7% of the corpus.\n",
    "    \n",
    "- If we want to measure how good our classifier is for a particular class then \"Macro averaging\" is the way to go\n",
    "    - As it gives equal importance to all the classes.\n",
    "\n",
    "- We are interested in the number of documents our classifier gets correctly, \"Micro averaging\" does that precisely\n",
    "    - combine the confusion matrix of all the classes and then calculate the scores.\n",
    "    \n",
    "###### Hence Micro averaging is selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "print(\"Accuracy : {:.2f}%\".format(metrics.accuracy_score(Y_test, Y_pred)*100))\n",
    "print(\"Precision: {:.2f}%\".format(metrics.precision_score(Y_test, Y_pred, average='micro')*100))\n",
    "print(\"Recall   : {:.2f}%\".format(metrics.recall_score(Y_test, Y_pred, average='micro')*100))\n",
    "print(\"F1-Score : {:.2f}%\".format(metrics.f1_score(Y_test, Y_pred, average='micro')*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.classification_report(y_true=Y_test, y_pred=Y_pred, target_names=mlb.classes_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion matrix for per class\n",
    "- Enter name of any class fom the list below for the confusion matrix of the class.\n",
    "\n",
    "#### List of classes\n",
    "    acq              alum          barley        bop           carcass       castor-oil    cocoa     \n",
    "    coconut          coconut-oil   coffee        copper        copra-cake    corn          cotton        \n",
    "    cotton-oil       cpi           cpu           crude         dfl           dlr           dmk      \n",
    "    earn             fuel          gas           gnp           gold          grain         groundnut    \n",
    "    groundnut-oil    heat          hog           housing       income        instal-debt   interest\n",
    "    ipi              iron-steel    jet           jobs          l-cattle      lead          money-supply\n",
    "    lei              lin-oil       livestock     lumber        meal-feed     money-fx      naphtha           \n",
    "    nat-gas          nickel        nkr           nzdlr         oat           oilseed       orange        \n",
    "    palladium        palm-oil      palmkernel    pet-chem      platinum      potato        propane       \n",
    "    rand             rape-oil      rapeseed      reserves      retail        rice          rubber         \n",
    "    rye              ship          silver        sorghum       soy-meal      soy-oil       soybean      \n",
    "    strategic-metal  sugar         sun-meal      sun-oil       sunseed       tea           tin           \n",
    "    trade            veg-oil       wheat         wpi           yen           zinc \n",
    "\n",
    "##### Format of confusion matrix\n",
    "    Class Name\n",
    "    [[ True Neg   False Pos]\n",
    "     [ False Neg  True Pos ]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_inp = input(\"Enter the name of any class : \")\n",
    "if cat_inp in (all_categories):\n",
    "    for i, cat in enumerate(all_categories):\n",
    "        if(cat_inp == cat):\n",
    "            print(\"\")\n",
    "            print(\"{}\".format(cat))\n",
    "            m = metrics.confusion_matrix(y_true=Y_test[:,i], y_pred=Y_pred[:,i])\n",
    "            print(m)\n",
    "else:\n",
    "     print(\" Enter a valid category name\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "plt.imshow(m, interpolation='nearest', cmap=plt.cm.Wistia)\n",
    "classNames = ['Not ' + cat_inp, cat_inp]\n",
    "plt.ylabel('True label')\n",
    "plt.xlabel('Predicted label')\n",
    "tick_marks = np.arange(len(classNames))\n",
    "plt.xticks(tick_marks, classNames, rotation=45)\n",
    "plt.yticks(tick_marks, classNames)\n",
    "s = [['TN','FP'], ['FN', 'TP']]\n",
    "\n",
    "for i in range(2):\n",
    "    for j in range(2):\n",
    "        plt.text(j,i, str(s[i][j])+\" = \"+str(m[i][j]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline\n",
    "- Enter the text to be tagged, some examples are given to just copy and try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#--------------------------------------------------------------------------------------#\n",
    "# Correct or partly correct tags\n",
    "    #Palm kernel oil is an edible plant oil derived from the kernel of the oil palm Elaeis guineensis.[1] It should not be confused with the other two edible oils derived from palm fruits: palm oil, extracted from the pulp of the oil palm fruit, and coconut oil, extracted from the kernel of the coconut.\n",
    "    #Natural gas is a naturally occurring hydrocarbon gas mixture consisting primarily of methane, but commonly including varying amounts of other higher alkanes, and sometimes a small percentage of carbon dioxide, nitrogen, hydrogen sulfide, or helium.[2] It is formed when layers of decomposing plant and animal matter are exposed to intense heat and pressure under the surface of the Earth over millions of years. The energy that the plants originally obtained from the sun is stored in the form of chemical bonds in the gas.\n",
    "    #Copper rebounded on Friday from a fresh 11-month low, as some investors regard the recent sharp losses as exaggerated and believe the metal has hit bottom. Copper, widely viewed as a bellwether for the global economy, has been battered by escalating trade tensions that resulted in the United States imposing tariffs on $34 billion of Chinese imports and Beijing quickly retaliating. The recent downtrend - which has seen copper shed 14 percent since touching a 4-1/2 year peak of $7,348 in early June - was fuelled by computer-driven speculators and long liquidation by Chinese hedge funds, said Gianclaudio Torlizzi, Partner at consultancy T-Commodity in Milan. Many people are wondering if the long-term bull market is over. I don't think it's over but copper has to hold above $6,200, which is the watershed level for the long-term uptrend, he said. It doesn't make any sense to have such a gloomy sentiment on metals demand. This is a good opportunity to go long again, Torlizzi added, saying he had already taken a long position\n",
    "    #Zinc is a bluish-white, lustrous, diamagnetic metal,[12] though most common commercial grades of the metal have a dull finish.[13] It is somewhat less dense than iron and has a hexagonal crystal structure, with a distorted form of hexagonal close packing, in which each atom has six nearest neighbors (at 265.9 pm) in its own plane and six others at a greater distance of 290.6 pm.[14] The metal is hard and brittle at most temperatures but becomes malleable between 100 and 150 °C.[12][13] Above 210 °C, the metal becomes brittle again and can be pulverized by beating.[15] Zinc is a fair conductor of electricity.[12] For a metal, zinc has relatively low melting (419.5 °C) and boiling points (907 °C).[16] The melting point is the lowest of all the d-block metals aside from mercury and cadmium; for this, among other reasons, zinc, cadmium, and mercury are often not considered to be transition metals like the rest of the d-block metals are.\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Missed Tags\n",
    "    # Aluminium-based alums have a number of common chemical properties. They are soluble in water, have a sweetish taste, react acid to litmus, and crystallize in regular octahedra. In alums each metal ion is surrounded by six water molecules. When heated, they liquefy, and if the heating is continued, the water of crystallization is driven off, the salt froths and swells, and at last an amorphous powder remains.[3] They are astringent and acidic.\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Input sentence\n",
    "inp_text = input(\"Enter the text you want to classify\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline = []\n",
    "pipeline.append(inp_text)\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Extract features\n",
    "example_features = count_vect.transform(pipeline)\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Do prediction\n",
    "example_preds = [clf.predict(example_features)[0] for clf in clfs.values()]\n",
    "#--------------------------------------------------------------------------------------#\n",
    "# Convert predictions back to labels\n",
    "mlb.inverse_transform(np.array([example_preds]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "- The approach is pretty balanced as the precision and recall are pretty high and almost similar.\n",
    "\n",
    "### Outlook into future work\n",
    "- Out of 90 classes there are still 20 which has a precision and recall value of 0.00\n",
    "- To improve further following can be explored\n",
    "    - Lemmatize the selected features.\n",
    "    - Calculate similarity(thisaurous and distributional) to find similar features and collapse then into one word.\n",
    "    - Use dense vectors instad of sparse vectors\n",
    "    - Maybe, training the classfier as OneVsRest,as multiple clasifier support that in sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}