2018/day36_Sentiment_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep learning - Sentiment Analysis \n",
    "\n",
    "## Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adhiman/anaconda3/envs/python3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "/Users/adhiman/anaconda3/envs/python3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "from keras import models\n",
    "from keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the Data\n",
    "\n",
    "Downloading the imdb dataset, which is fortunately already built into Keras. \n",
    "\n",
    "Since we don’t want to have a 50/50 train test split, we will immediately merge the data into data and targets after downloading, so that we can do an 80/20 split later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "(training_data, training_targets), (testing_data, testing_targets) = imdb.load_data(num_words=10000)\n",
    "data = np.concatenate((training_data, testing_data), axis=0)\n",
    "targets = np.concatenate((training_targets, testing_targets), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categories: [0 1]\n",
      "Number of unique words: 9998\n",
      "Average Review length: 234.75892\n",
      "Standard Deviation: 173.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Categories:\", np.unique(targets))\n",
    "print(\"Number of unique words:\", len(np.unique(np.hstack(data))))\n",
    "\n",
    "length = [len(i) for i in data]\n",
    "print(\"Average Review length:\", np.mean(length))\n",
    "print(\"Standard Deviation:\", round(np.std(length)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[0 1]` represent the 2 sentiments of the review\n",
    "\n",
    "**Now we will look at a single training example:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: 1\n",
      "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n"
     ]
    }
   ],
   "source": [
    "print(\"Label:\", targets[0])\n",
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first review of the dataset which is labeled as positive (`1`)\n",
    "\n",
    "The code below retrieves the dictionary mapping word indices back into the original words so that we can read them. It replaces every unknown word with a `#`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert # is an amazing actor and now the same being director # father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for # and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also # to the two little boy's that played the # of norman and paul they were just brilliant children are often left out of the # list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\n"
     ]
    }
   ],
   "source": [
    "index = imdb.get_word_index()\n",
    "\n",
    "reverse_index = dict([(value, key) for (key, value) in index.items()]) \n",
    "decoded = \" \".join( [reverse_index.get(i - 3, \"#\") for i in data[0]] )\n",
    "print(decoded) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "We will vectorize every review and fill it with zeros so that it contains exactly 10,000 numbers. That means we fill every review that is shorter than 10,000 with zeros. We do this because the biggest review is nearly that long and every input for our neural network needs to have the same size. We also transform the targets into floats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vectorize(sequences, dimension = 10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1\n",
    "    return results\n",
    " \n",
    "data = vectorize(data)\n",
    "targets = np.array(targets).astype(\"float32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we split our data into a training and a testing set. The training set will contain 40,000 reviews and the testing set 10,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_x = data[:10000]\n",
    "test_y = targets[:10000]\n",
    "\n",
    "train_x = data[10000:]\n",
    "train_y = targets[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Training the Model\n",
    "We start by defining the type of model we want to build. \n",
    "There are two types of models available in Keras: \n",
    "* the Sequential model and \n",
    "* the Model class used with functional API\n",
    "\n",
    "* Then we simply add the input-, hidden- and output-layers. \n",
    "* Between them, we are using dropout to prevent overfitting. \n",
    "* Note that you should always use a dropout rate between 20% and 50%. \n",
    "* At every layer, we use “Dense” which means that the units are fully connected. \n",
    "* Within the hidden-layers, we use the relu function, because this is always a good start and yields a satisfactory result most of the time. \n",
    "* And at the output-layer, we use the sigmoid function, which maps the values between 0 and 1. \n",
    "* Note that we set the input-shape to 10,000 at the input-layer, because our reviews are 10,000 integers long. \n",
    "* The input-layer takes 10,000 as input and outputs it with a shape of 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 50)                500050    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 505,201\n",
      "Trainable params: 505,201\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "# Input - Layer\n",
    "model.add(layers.Dense(50, activation = \"relu\", input_shape=(10000, )))\n",
    "\n",
    "# Hidden - Layers\n",
    "model.add(layers.Dropout(0.3, noise_shape=None, seed=None))\n",
    "model.add(layers.Dense(50, activation = \"relu\"))\n",
    "model.add(layers.Dropout(0.2, noise_shape=None, seed=None))\n",
    "model.add(layers.Dense(50, activation = \"relu\"))\n",
    "\n",
    "# Output- Layer\n",
    "model.add(layers.Dense(1, activation = \"sigmoid\"))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Now we need to **compile** our model, which is nothing but configuring the model for training. \n",
    "* We use the “**adam**” optimizer. The optimizer is the algorithm that changes the weights and biases during training. \n",
    "* We also choose **binary-crossentropy** as loss (because we deal with binary classification) and **accuracy** as our evaluation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    " optimizer = \"adam\",\n",
    " loss = \"binary_crossentropy\",\n",
    " metrics = [\"accuracy\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now able to train our model. We do this with a batch_size of 500 and only for two epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 11s 266us/step - loss: 0.3634 - acc: 0.8410 - val_loss: 0.2598 - val_acc: 0.8944\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 10s 242us/step - loss: 0.2026 - acc: 0.9222 - val_loss: 0.2694 - val_acc: 0.8883\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 7s 179us/step - loss: 0.1419 - acc: 0.9489 - val_loss: 0.3087 - val_acc: 0.8852\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 7s 186us/step - loss: 0.0960 - acc: 0.9665 - val_loss: 0.3557 - val_acc: 0.8856\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 7s 166us/step - loss: 0.0668 - acc: 0.9773 - val_loss: 0.4236 - val_acc: 0.8844\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 7s 181us/step - loss: 0.0495 - acc: 0.9833 - val_loss: 0.4948 - val_acc: 0.8780\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s 157us/step - loss: 0.0416 - acc: 0.9854 - val_loss: 0.4922 - val_acc: 0.8755\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 7s 176us/step - loss: 0.0402 - acc: 0.9857 - val_loss: 0.5184 - val_acc: 0.8791\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s 153us/step - loss: 0.0329 - acc: 0.9886 - val_loss: 0.5620 - val_acc: 0.8782\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 8s 193us/step - loss: 0.0305 - acc: 0.9890 - val_loss: 0.5753 - val_acc: 0.8797\n"
     ]
    }
   ],
   "source": [
    "results = model.fit(\n",
    " train_x, train_y,\n",
    " epochs= 10,\n",
    " batch_size = 300,\n",
    " validation_data = (test_x, test_y)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8828399963974952\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(results.history[\"val_acc\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}