examples/contrastive/CEM-MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CEMExplainer: MNIST Example\n",
    "- This notebook showcases an example of how to use the CEMExplainer from [AIX360](https://github.com/IBM/AIX360) to obtain contrastive explanations i.e. *pertinent negatives (PNs)* and *pertinent postitives (PPs)* for predictions made by a model trained on MNIST data. \n",
    "- The CEMExplainer is an implementation of the [contrastive explanation method](https://arxiv.org/abs/1802.07623).\n",
    "- The default location of this notebook is aix360/examples/constrastive/ folder. This notebook uses trained models which are accessed from aix360/models/CEM/ folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "from keras.models import model_from_json\n",
    "from PIL import Image\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from aix360.algorithms.contrastive import CEMExplainer, KerasClassifier\n",
    "from aix360.datasets import MNISTDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load MNIST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load MNIST data and normalize it in the range [-0.5, 0.5]\n",
    "data = MNISTDataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNIST train data range  : ( -0.5 , 0.5 )\n",
      "MNIST test data range   : ( -0.5 , 0.5 )\n",
      "MNIST train data shape  : (55000, 28, 28, 1)\n",
      "MNIST test data shape   : (10000, 28, 28, 1)\n",
      "MNIST train labels shape: (10000, 10)\n",
      "MNIST test labels shape : (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "# print the shape of train and test data\n",
    "print(\"MNIST train data range  :\", \"(\", np.min(data.train_data), \",\", np.max(data.train_data), \")\")\n",
    "print(\"MNIST test data range   :\", \"(\", np.min(data.train_data), \",\",  np.max(data.train_data), \")\")\n",
    "print(\"MNIST train data shape  :\", data.train_data.shape)\n",
    "print(\"MNIST test data shape   :\", data.test_data.shape)\n",
    "print(\"MNIST train labels shape:\", data.test_labels.shape)\n",
    "print(\"MNIST test labels shape :\", data.test_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load a trained MNIST model\n",
    "- This notebook uses a trained MNIST model. The code to train this model is available [here](https://github.com/huanzhang12/ZOO-Attack/blob/master/train_models.py). Note that the model outputs logits and does not use a softmax function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0803 10:50:01.064849 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "W0803 10:50:01.068910 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "W0803 10:50:01.106316 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
      "\n",
      "W0803 10:50:01.249321 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
      "\n",
      "W0803 10:50:01.250549 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "W0803 10:50:01.251479 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_5 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 26, 26, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 24, 24, 32)        9248      \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 12, 12, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 10, 10, 64)        18496     \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 10, 10, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 8, 8, 64)          36928     \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 8, 8, 64)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 4, 4, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 200)               205000    \n",
      "_________________________________________________________________\n",
      "activation_11 (Activation)   (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 200)               40200     \n",
      "_________________________________________________________________\n",
      "activation_12 (Activation)   (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 10)                2010      \n",
      "=================================================================\n",
      "Total params: 312,202\n",
      "Trainable params: 312,202\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# path to mnist related models\n",
    "model_path = '../../aix360/models/CEM'\n",
    "\n",
    "def load_model(model_json_file, model_wt_file):\n",
    "    \n",
    "    # read model json file\n",
    "    with open(model_json_file, 'r') as f:\n",
    "        model = model_from_json(f.read())\n",
    "    \n",
    "    # read model weights file\n",
    "    model.load_weights(model_wt_file)\n",
    "    \n",
    "    return model\n",
    "    \n",
    "\n",
    "# load MNIST model using its json and wt files\n",
    "mnist_model = load_model(os.path.join(model_path, 'mnist.json'), os.path.join(model_path, 'mnist'))\n",
    "\n",
    "# print model summary\n",
    "mnist_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Load a trained convolutional autoencoder model (optional)\n",
    "- This notebook uses a trained convolutional autoencoder model. The code to train this model is available [here](https://github.com/chunchentu/autoencoder). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0803 10:50:01.503559 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "sequential_2 (Sequential)    (None, 14, 14, 1)         2625      \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 14, 14, 16)        160       \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 14, 14, 16)        0         \n",
      "_________________________________________________________________\n",
      "up_sampling2d_1 (UpSampling2 (None, 28, 28, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 28, 28, 16)        2320      \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 28, 28, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 28, 28, 1)         145       \n",
      "=================================================================\n",
      "Total params: 5,250\n",
      "Trainable params: 5,250\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# load the trained convolutional autoencoder model\n",
    "ae_model = load_model(os.path.join(model_path, 'mnist_AE_1_decoder.json'), \n",
    "                      os.path.join(model_path, 'mnist_AE_1_decoder.h5'))\n",
    "# print model summary\n",
    "ae_model.summary()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initialize CEM Explainer to explain model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wrap mnist_model into a framework independent class structure\n",
    "mymodel = KerasClassifier(mnist_model)\n",
    "\n",
    "# initialize explainer object\n",
    "explainer = CEMExplainer(mymodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explain an input instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class: [3]\n",
      "Predicted logits: [[-11.279337    0.7362492  -9.008647   19.396713   -8.286124   14.442825\n",
      "   -1.3170446 -11.587322   -0.9921855   1.0182219]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANKklEQVR4nO3db4wc9X3H8c/HJpEQMciAONnkbNzIPLCCYlcWKioqrqJE1EKYPIliicpVDecHBjmiQkD6IEhVpagqKTxAkc/CilsFokjg2g5VE2oZ3D6JMMbGNjSGYlu+0/nMnwexH6Wcv31w4/YwN7Pnndmd7X3fL+m0u/Pd2flqzYeZnd/u/BwRAjD/LWi7AQD9QdiBJAg7kARhB5Ig7EAS1/RzY7Y59Q/0WER4tuW19uy277X9W9sf2H6yzmsB6C13O85ue6Gkk5K+JWlM0puSNkbEuxXrsGcHeqwXe/Y7JX0QER9GxO8l/VzShhqvB6CH6oT9VklnZzweK5Z9ju0R24dsH6qxLQA19fwEXUSMShqVOIwH2lRnzz4uaXjG468WywAMoDphf1PSStsrbH9Z0vck7W2mLQBN6/owPiI+s/2IpF9JWihpZ0ScaKwzAI3qeuitq43xmR3ouZ58qQbA/x+EHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4k0ddLSWP+uf322yvr27dvL629+OKLlevu2LGjq54wO/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEV5dFpU7j6K+++mplfcWKFaW1s2fPltY6rYtyXF0WSI6wA0kQdiAJwg4kQdiBJAg7kARhB5Lg9+zJbdu2rVZ92bJlXW/7zJkzXa+Lq1cr7LZPS7ogaUrSZxGxtommADSviT37n0bExw28DoAe4jM7kETdsIekX9t+y/bIbE+wPWL7kO1DNbcFoIa6h/F3R8S47VskvWb7PyPi4MwnRMSopFGJH8IAbaq1Z4+I8eL2vKTdku5soikAzes67Lavs73o8n1J35Z0vKnGADSrzmH8kKTdti+/zosR8a+NdIXGXHNN9T/xqlWrKuvLly+vrHe6HsLJkydLaw8++GDlumhW12GPiA8lfaPBXgD0EENvQBKEHUiCsANJEHYgCcIOJMFPXOe5LVu2VNY3b97c0+1/8sknpbWxsbGebhufx54dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnH0eWLp0aWntoYceqly3+IlyqQULqvcHly5dqqw//vjjlXX0D3t2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfZ5oGra5DvuuKNy3U6Xgu40jr5v377K+uHDhyvr6B/27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPs88DFixdLa1XXbZekm266qda277rrrsr6ypUrS2snTpyotW1cnY57dts7bZ+3fXzGshttv2b7/eJ2cW/bBFDXXA7jfyrp3iuWPSlpf0SslLS/eAxggHUMe0QclPTpFYs3SNpV3N8l6YGG+wLQsG4/sw9FxERx/5ykobIn2h6RNNLldgA0pPYJuogI26W/poiIUUmjklT1PAC91e3Q26TtJZJU3J5vriUAvdBt2PdK2lTc3yRpTzPtAOgVd/o9s+2XJK2TdLOkSUk/lPTPkn4haZmkM5K+GxFXnsSb7bU4jO+z7du3V9Y7zc/e6brynf77qdr+1q1bK9dFdyJi1n+0jp/ZI2JjSembtToC0Fd8XRZIgrADSRB2IAnCDiRB2IEkOg69Nboxht76bnh4uLJ+6tSpynrdobeJiYnS2n333Ve57tGjRyvrmF3Z0Bt7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH25J555pnK+mOPPVZZ7zSlc5WxsbHK+vLly7t+7cwYZweSI+xAEoQdSIKwA0kQdiAJwg4kQdiBJBhnT+6GG26orK9fv76yPjo6Wlm/9tprS2tTU1OV63a6DPbOnTsr60eOHKmsz1eMswPJEXYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzo5bdu3dX1tetW1daW7RoUa1tT05OVtZXr15dWvvoo49qbXuQdT3Obnun7fO2j89Y9rTtcdtHir/qb14AaN1cDuN/KuneWZb/Q0SsLv7+pdm2ADStY9gj4qCkT/vQC4AeqnOC7hHb7xSH+YvLnmR7xPYh24dqbAtATd2G/SeSviZptaQJSaVXLYyI0YhYGxFru9wWgAZ0FfaImIyIqYi4JGmHpDubbQtA07oKu+0lMx5+R9LxsucCGAwdx9ltvyRpnaSbJU1K+mHxeLWkkHRa0paIKJ+I+/9ei3H2ZLZs2VJae/7552u9dqe545ctW1ZaGx8fr7XtQVY2zn7NHFbcOMviF2p3BKCv+LoskARhB5Ig7EAShB1IgrADSXQ8Gw/UcfTo0bZbQIE9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTh7H9xzzz211n/jjTca6qR5Dz/8cGX9qaeeKq11+olqJwsWsK+6GrxbQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+wNWLp0aWV9z549lfWDBw9W1m+55Zar7mmu7r///sp6p+8IDA0NVdYXLlxYWut0GfMjR45U1jds2FBZP3fuXGU9G/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BExymbG93YPJ2yeXh4uLJ+6tSpynqn33X389/oSnV7u3DhQmntiSeeqFx33759lfWJiY6zhKdUNmVzxz277WHbB2y/a/uE7W3F8httv2b7/eJ2cdNNA2jOXA7jP5P0VxGxStIfSdpqe5WkJyXtj4iVkvYXjwEMqI5hj4iJiDhc3L8g6T1Jt0raIGlX8bRdkh7oVZMA6ruq78bbvk3SGkm/kTQUEZc/NJ2TNOuXpG2PSBrpvkUATZjz2XjbX5H0sqTvR8TvZtZi+izNrGdqImI0ItZGxNpanQKoZU5ht/0lTQf9ZxHxSrF40vaSor5E0vnetAigCR0P4z099vKCpPci4sczSnslbZL0o+K2+nec89jU1FRlvWr4SZKuv/76Jttp1NjYWGX97bffrqw/99xzpbUDBw501RO6M5fP7H8s6c8lHbN9+QfGP9B0yH9he7OkM5K+25sWATShY9gj4j8klX2z4pvNtgOgV/i6LJAEYQeSIOxAEoQdSIKwA0nwE9c+6HQ55jVr1tR6/UcffbS09vrrr1eue+zYscr6s88+201LaFHXP3EFMD8QdiAJwg4kQdiBJAg7kARhB5Ig7EASjLMD8wzj7EByhB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BEx7DbHrZ9wPa7tk/Y3lYsf9r2uO0jxd/63rcLoFsdL15he4mkJRFx2PYiSW9JekDT87FfjIi/n/PGuHgF0HNlF6+Yy/zsE5ImivsXbL8n6dZm2wPQa1f1md32bZLWSPpNsegR2+/Y3ml7cck6I7YP2T5Uq1MAtcz5GnS2vyLpDUl/GxGv2B6S9LGkkPQ3mj7U/8sOr8FhPNBjZYfxcwq77S9J+qWkX0XEj2ep3ybplxHx9Q6vQ9iBHuv6gpO2LekFSe/NDHpx4u6y70g6XrdJAL0zl7Pxd0v6d0nHJF0qFv9A0kZJqzV9GH9a0pbiZF7Va7FnB3qs1mF8Uwg70HtcNx5IjrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BExwtONuxjSWdmPL65WDaIBrW3Qe1LorduNdnb8rJCX3/P/oWN24ciYm1rDVQY1N4GtS+J3rrVr944jAeSIOxAEm2HfbTl7VcZ1N4GtS+J3rrVl95a/cwOoH/a3rMD6BPCDiTRStht32v7t7Y/sP1kGz2UsX3a9rFiGupW56cr5tA7b/v4jGU32n7N9vvF7axz7LXU20BM410xzXir713b05/3/TO77YWSTkr6lqQxSW9K2hgR7/a1kRK2T0taGxGtfwHD9p9IuijpHy9PrWX77yR9GhE/Kv5HuTginhiQ3p7WVU7j3aPeyqYZ/wu1+N41Of15N9rYs98p6YOI+DAifi/p55I2tNDHwIuIg5I+vWLxBkm7ivu7NP0fS9+V9DYQImIiIg4X9y9IujzNeKvvXUVffdFG2G+VdHbG4zEN1nzvIenXtt+yPdJ2M7MYmjHN1jlJQ202M4uO03j30xXTjA/Me9fN9Od1cYLui+6OiD+U9GeSthaHqwMppj+DDdLY6U8kfU3TcwBOSHqmzWaKacZflvT9iPjdzFqb790sffXlfWsj7OOShmc8/mqxbCBExHhxe17Sbk1/7Bgkk5dn0C1uz7fcz/+KiMmImIqIS5J2qMX3rphm/GVJP4uIV4rFrb93s/XVr/etjbC/KWml7RW2vyzpe5L2ttDHF9i+rjhxItvXSfq2Bm8q6r2SNhX3N0na02IvnzMo03iXTTOult+71qc/j4i+/0lar+kz8v8l6a/b6KGkrz+QdLT4O9F2b5Je0vRh3X9r+tzGZkk3Sdov6X1J/ybpxgHq7Z80PbX3O5oO1pKWertb04fo70g6Uvytb/u9q+irL+8bX5cFkuAEHZAEYQeSIOxAEoQdSIKwA0kQdiAJwg4k8T+w/0Mw8Vf76wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# choose an input image\n",
    "image_id = 340\n",
    "input_image = data.test_data[image_id]\n",
    "\n",
    "# rescale values from [-0.5, 0.5] to [0, 255] for plotting\n",
    "plt.imshow((input_image[:,:,0] + 0.5)*255, cmap=\"gray\")\n",
    "\n",
    "# check model prediction\n",
    "print(\"Predicted class:\", mymodel.predict_classes(np.expand_dims(input_image, axis=0)))\n",
    "print(\"Predicted logits:\", mymodel.predict(np.expand_dims(input_image, axis=0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Observation: \n",
    "\n",
    "Although the above image is classified as digit 3 by the model, it could have been classified as digit 5 as well since it has similarities to the digit 5. We now employ the CEMExplainer from AIX360 to compute pertinent positive and pertinent negative explanations, which help us understand why the image was classified as digit 3 by the model and not as digit 5. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Obtain Pertinent Negative (PN) explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0803 10:50:02.457590 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/aix360/algorithms/contrastive/CEM_aen.py:150: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n",
      "\n",
      "W0803 10:50:02.467044 140678942381888 deprecation.py:323] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/learning_rate_schedule.py:409: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "W0803 10:50:02.474853 140678942381888 deprecation_wrapper.py:119] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/aix360/algorithms/contrastive/CEM_aen.py:153: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.\n",
      "\n",
      "W0803 10:50:02.534118 140678942381888 deprecation.py:323] From /home/vijay/anaconda3/envs/aix360/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter:0 const:[10.]\n",
      "Loss_Overall:117.3156, Loss_Attack:65.3037\n",
      "Loss_L2Dist:1.1484, Loss_L1Dist:6.2333, AE_loss:50.240135192871094\n",
      "target_lab_score:15.8465, max_nontarget_lab_score:19.3161\n",
      "\n",
      "iter:500 const:[10.]\n",
      "Loss_Overall:5731.0386, Loss_Attack:0.0000\n",
      "Loss_L2Dist:295.1370, Loss_L1Dist:372.2983, AE_loss:5398.67138671875\n",
      "target_lab_score:-4.0691, max_nontarget_lab_score:9.3513\n",
      "\n",
      "iter:0 const:[5.]\n",
      "Loss_Overall:100.2937, Loss_Attack:59.5759\n",
      "Loss_L2Dist:0.6385, Loss_L1Dist:4.1246, AE_loss:39.666893005371094\n",
      "target_lab_score:18.3216, max_nontarget_lab_score:16.4064\n",
      "\n",
      "iter:500 const:[5.]\n",
      "Loss_Overall:6569.7109, Loss_Attack:16.6774\n",
      "Loss_L2Dist:305.4800, Loss_L1Dist:381.6533, AE_loss:6209.38818359375\n",
      "target_lab_score:1.2297, max_nontarget_lab_score:7.8942\n",
      "\n",
      "iter:0 const:[2.5]\n",
      "Loss_Overall:74.2821, Loss_Attack:33.0332\n",
      "Loss_L2Dist:0.5359, Loss_L1Dist:3.6111, AE_loss:40.35193634033203\n",
      "target_lab_score:18.9114, max_nontarget_lab_score:15.6981\n",
      "\n",
      "iter:500 const:[2.5]\n",
      "Loss_Overall:6238.6172, Loss_Attack:23.4504\n",
      "Loss_L2Dist:281.3679, Loss_L1Dist:360.9008, AE_loss:5897.708984375\n",
      "target_lab_score:2.0889, max_nontarget_lab_score:2.7088\n",
      "\n",
      "iter:0 const:[1.25]\n",
      "Loss_Overall:59.0087, Loss_Attack:17.0873\n",
      "Loss_L2Dist:0.5046, Loss_L1Dist:3.5153, AE_loss:41.065223693847656\n",
      "target_lab_score:19.1260, max_nontarget_lab_score:15.4562\n",
      "\n",
      "iter:500 const:[1.25]\n",
      "Loss_Overall:6604.8657, Loss_Attack:14.3013\n",
      "Loss_L2Dist:281.1414, Loss_L1Dist:361.9403, AE_loss:6273.22900390625\n",
      "target_lab_score:4.8682, max_nontarget_lab_score:3.4272\n",
      "\n",
      "iter:0 const:[0.625]\n",
      "Loss_Overall:51.0226, Loss_Attack:8.6989\n",
      "Loss_L2Dist:0.4934, Loss_L1Dist:3.4979, AE_loss:41.48050308227539\n",
      "target_lab_score:19.2539, max_nontarget_lab_score:15.3357\n",
      "\n",
      "iter:500 const:[0.625]\n",
      "Loss_Overall:5880.4536, Loss_Attack:4.2246\n",
      "Loss_L2Dist:307.5166, Loss_L1Dist:392.9896, AE_loss:5529.41357421875\n",
      "target_lab_score:0.8791, max_nontarget_lab_score:4.1198\n",
      "\n",
      "iter:0 const:[0.3125]\n",
      "Loss_Overall:47.0586, Loss_Attack:4.3939\n",
      "Loss_L2Dist:0.4888, Loss_L1Dist:3.4828, AE_loss:41.827728271484375\n",
      "target_lab_score:19.3268, max_nontarget_lab_score:15.2665\n",
      "\n",
      "iter:500 const:[0.3125]\n",
      "Loss_Overall:6150.1323, Loss_Attack:2.1761\n",
      "Loss_L2Dist:291.2777, Loss_L1Dist:368.6633, AE_loss:5819.8125\n",
      "target_lab_score:0.9448, max_nontarget_lab_score:3.9814\n",
      "\n",
      "iter:0 const:[0.15625]\n",
      "Loss_Overall:45.0654, Loss_Attack:2.2082\n",
      "Loss_L2Dist:0.4868, Loss_L1Dist:3.4763, AE_loss:42.02281951904297\n",
      "target_lab_score:19.3643, max_nontarget_lab_score:15.2319\n",
      "\n",
      "iter:500 const:[0.15625]\n",
      "Loss_Overall:6265.9688, Loss_Attack:0.4966\n",
      "Loss_L2Dist:286.9196, Loss_L1Dist:359.0903, AE_loss:5942.6435546875\n",
      "target_lab_score:-0.7716, max_nontarget_lab_score:6.0504\n",
      "\n",
      "iter:0 const:[0.078125]\n",
      "Loss_Overall:44.0672, Loss_Attack:1.1069\n",
      "Loss_L2Dist:0.4858, Loss_L1Dist:3.4732, AE_loss:42.12709426879883\n",
      "target_lab_score:19.3832, max_nontarget_lab_score:15.2146\n",
      "\n",
      "iter:500 const:[0.078125]\n",
      "Loss_Overall:6456.3413, Loss_Attack:0.7018\n",
      "Loss_L2Dist:317.7498, Loss_L1Dist:398.9077, AE_loss:6097.9990234375\n",
      "target_lab_score:1.5749, max_nontarget_lab_score:2.5923\n",
      "\n",
      "iter:0 const:[0.0390625]\n",
      "Loss_Overall:43.5620, Loss_Attack:0.5541\n",
      "Loss_L2Dist:0.4854, Loss_L1Dist:3.4722, AE_loss:42.17523956298828\n",
      "target_lab_score:19.3921, max_nontarget_lab_score:15.2067\n",
      "\n",
      "iter:500 const:[0.0390625]\n",
      "Loss_Overall:6328.7046, Loss_Attack:0.3000\n",
      "Loss_L2Dist:294.4442, Loss_L1Dist:373.3748, AE_loss:5996.623046875\n",
      "target_lab_score:4.6621, max_nontarget_lab_score:6.9825\n",
      "\n"
     ]
    }
   ],
   "source": [
    "arg_mode = \"PN\" # Find pertinent negative\n",
    "\n",
    "arg_max_iter = 1000 # Maximum number of iterations to search for the optimal PN for given parameter settings\n",
    "arg_init_const = 10.0 # Initial coefficient value for main loss term that encourages class change\n",
    "arg_b = 9 # No. of updates to the coefficient of the main loss term\n",
    "\n",
    "arg_kappa = 10 # Minimum confidence gap between the PNs (changed) class probability and original class' probability\n",
    "arg_beta = 1e-1 # Controls sparsity of the solution (L1 loss)\n",
    "arg_gamma = 100 # Controls how much to adhere to a (optionally trained) autoencoder\n",
    "\n",
    "\n",
    "(adv_pn, delta_pn, info_pn) = explainer.explain_instance(np.expand_dims(input_image, axis=0), arg_mode, ae_model, arg_kappa, arg_b, \n",
    "                                            arg_max_iter, arg_init_const, arg_beta, arg_gamma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]kappa:10, Orig class:3, Perturbed class:5, Delta class: 8, Orig prob:[[-11.279337    0.7362492  -9.008647   19.396713   -8.286124   14.442825   -1.3170446 -11.587322   -0.9921855   1.0182219]], Perturbed prob:[[ -9.823904   -3.0258849 -12.282337   13.914494   -9.164177   24.124342    0.3995837 -12.929418   -0.087002    2.4819136]], Delta prob:[[-0.50663096 -0.84834933  0.7846163   0.59887445 -1.0884676  -0.28783607  -2.5196645   1.3976169   2.129994    0.20926595]]\n"
     ]
    }
   ],
   "source": [
    "print(info_pn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Obtain Pertinent Positive (PP) explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter:0 const:[10.]\n",
      "Loss_Overall:1271.8158, Loss_Attack:105.5809\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:500 const:[10.]\n",
      "Loss_Overall:1271.8158, Loss_Attack:105.5809\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:0 const:[100.]\n",
      "Loss_Overall:2112.5354, Loss_Attack:859.8530\n",
      "Loss_L2Dist:3.8316, Loss_L1Dist:8.9619, AE_loss:1247.9547119140625\n",
      "target_lab_score:2.8555, max_nontarget_lab_score:1.4540\n",
      "\n",
      "iter:500 const:[100.]\n",
      "Loss_Overall:2813.5281, Loss_Attack:817.9916\n",
      "Loss_L2Dist:21.7097, Loss_L1Dist:32.3632, AE_loss:1970.5904541015625\n",
      "target_lab_score:6.2211, max_nontarget_lab_score:4.4010\n",
      "\n",
      "iter:0 const:[55.]\n",
      "Loss_Overall:1771.9391, Loss_Attack:612.6125\n",
      "Loss_L2Dist:0.6849, Loss_L1Dist:2.7694, AE_loss:1158.36474609375\n",
      "target_lab_score:0.2389, max_nontarget_lab_score:1.3773\n",
      "\n",
      "iter:500 const:[55.]\n",
      "Loss_Overall:2663.4966, Loss_Attack:633.3018\n",
      "Loss_L2Dist:20.6486, Loss_L1Dist:31.4669, AE_loss:2006.3995361328125\n",
      "target_lab_score:5.9188, max_nontarget_lab_score:7.4334\n",
      "\n",
      "iter:0 const:[32.5]\n",
      "Loss_Overall:1496.7809, Loss_Attack:345.4348\n",
      "Loss_L2Dist:0.0753, Loss_L1Dist:0.5746, AE_loss:1151.2132568359375\n",
      "target_lab_score:0.0645, max_nontarget_lab_score:0.6933\n",
      "\n",
      "iter:500 const:[32.5]\n",
      "Loss_Overall:1815.2236, Loss_Attack:264.7164\n",
      "Loss_L2Dist:12.4209, Loss_L1Dist:25.7885, AE_loss:1535.5074462890625\n",
      "target_lab_score:8.3142, max_nontarget_lab_score:6.4593\n",
      "\n",
      "iter:0 const:[21.25]\n",
      "Loss_Overall:1387.5613, Loss_Attack:224.0321\n",
      "Loss_L2Dist:0.0015, Loss_L1Dist:0.0518, AE_loss:1163.5225830078125\n",
      "target_lab_score:-0.1502, max_nontarget_lab_score:0.3925\n",
      "\n",
      "iter:500 const:[21.25]\n",
      "Loss_Overall:1555.0825, Loss_Attack:74.3874\n",
      "Loss_L2Dist:9.9007, Loss_L1Dist:19.1401, AE_loss:1468.88037109375\n",
      "target_lab_score:9.0457, max_nontarget_lab_score:2.5462\n",
      "\n",
      "iter:0 const:[15.625]\n",
      "Loss_Overall:1331.2051, Loss_Attack:164.9702\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:500 const:[15.625]\n",
      "Loss_Overall:1331.2051, Loss_Attack:164.9702\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:0 const:[18.4375]\n",
      "Loss_Overall:1360.8997, Loss_Attack:194.6648\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:500 const:[18.4375]\n",
      "Loss_Overall:1360.8997, Loss_Attack:194.6648\n",
      "Loss_L2Dist:0.0000, Loss_L1Dist:0.0000, AE_loss:1166.23486328125\n",
      "target_lab_score:-0.1728, max_nontarget_lab_score:0.3853\n",
      "\n",
      "iter:0 const:[19.84375]\n",
      "Loss_Overall:1374.3131, Loss_Attack:209.2894\n",
      "Loss_L2Dist:0.0003, Loss_L1Dist:0.0183, AE_loss:1165.021484375\n",
      "target_lab_score:-0.1648, max_nontarget_lab_score:0.3821\n",
      "\n",
      "iter:500 const:[19.84375]\n",
      "Loss_Overall:1758.1250, Loss_Attack:171.5620\n",
      "Loss_L2Dist:8.9867, Loss_L1Dist:16.8015, AE_loss:1575.8961181640625\n",
      "target_lab_score:6.1731, max_nontarget_lab_score:4.8187\n",
      "\n",
      "iter:0 const:[19.140625]\n",
      "Loss_Overall:1367.6649, Loss_Attack:202.0110\n",
      "Loss_L2Dist:0.0001, Loss_L1Dist:0.0081, AE_loss:1165.653076171875\n",
      "target_lab_score:-0.1694, max_nontarget_lab_score:0.3846\n",
      "\n",
      "iter:500 const:[19.140625]\n",
      "Loss_Overall:1673.5901, Loss_Attack:135.8866\n",
      "Loss_L2Dist:7.5158, Loss_L1Dist:15.7103, AE_loss:1528.61669921875\n",
      "target_lab_score:6.8402, max_nontarget_lab_score:3.9396\n",
      "\n"
     ]
    }
   ],
   "source": [
    "arg_mode = \"PP\"  # Find pertinent positive\n",
    "(adv_pp, delta_pp, info_pp) = explainer.explain_instance(np.expand_dims(input_image, axis=0), arg_mode, ae_model, arg_kappa, arg_b, \n",
    "                                            arg_max_iter, arg_init_const, arg_beta, arg_gamma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]kappa:10, Orig class:3, Perturbed class:5, Delta class: 3, Orig prob:[[-11.279337    0.7362492  -9.008647   19.396713   -8.286124   14.442825   -1.3170446 -11.587322   -0.9921855   1.0182219]], Perturbed prob:[[-10.512523     2.5331008   -7.8333354   13.738868    -4.697951   13.911682    -0.17094916  -9.634063    -2.869906    -0.1969316 ]], Delta prob:[[-7.0725145  -1.732577    0.35496587 12.47732    -4.7352133   2.2513726  -5.1536884   0.21767989  1.7942489  -1.3421019 ]]\n"
     ]
    }
   ],
   "source": [
    "print(info_pp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot Pertinent Negative (PN) and Pertinent Positive (PP) explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAACPCAYAAAA1FeWWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXi0lEQVR4nO3de7AU5ZnH8d+jiGJAvGDIARFcUaOLCRpDYtTgJmTXazSrUXHNYgU1Wbzg6roq2dzWqiTeosZoCtwocaPGVDCKRuMaCkxSZTTeEBRFd4UAewAxKheBjfLsH92n6R5mzpk+Zy4973w/Vad4et6e6Yd+5p15p6/m7gIAAED1tmt2AgAAAK2GARQAAEBODKAAAAByYgAFAACQEwMoAACAnBhAAQAA5NQWAygzm2Zm/1Hreat4LTez0anpvzOz+6t87olmdm8t8iiaotSjKMzsu2Z2cZXzzjKzY+udE1AJ/RfVCv47z91b7k/S2ZIWSHpX0kpJP5K0a7PzKpOnSxqdmn5a0idT03MlvSFpraT5kk4qef5CSR9p9v8jhFqUq0eNXnOJpI2S1ktaJWmmpIFx2zxJmySNSM0/QdKS1PSeklZIGhBPj4rzXJ/6+3pq/nGSnmn2umxS/cqu62rWM38V1yn9t+f+u17SGkn3Sepo9npo0rrvqe+VXUchfuel/1puC5SZXSrpakmXSRos6ZOSRkp6zMz6l5m/X2MzLM/MPi5psLv/IfXwVEVvtl0knSfpp2bWkWq/J368kPLWIn5OIerRHTM728xm5njKie4+UNKhkg6T9G+ptg2Svt7Nc8+W9LC7byx5fFd3Hxj/XdX1oLs/JWkXMzssR34hqbSue1rPKEH/TXTXfy+I2/aXtKukG2qWaOuptJ7KrqMQv/NKtdQAysx2kfRtSRe6+6/d/S/uvkTSaYp+uZ9lZt8ys1+Y2U/NbK2ks+PHfpp6nX80s6Vm9qaZfd3MlpjZhLgtmdfMRsWbjSeZ2Z/MbI2ZfS31OuPM7Akze9vMOs3sh5U+eCQdK+nx9APu/oK7v9c1KWkHSSNSs8yTdHxv11c9VVOLeL6i1qPm3H2FpEckjUk9/ANJE81s3wpP2+Z9UYV5Kuj7olHKrOue1jNS6L/bqtB/u9r+LGlWubZ2U2k9lVlHQX3nldNSAyhJn5K0k6LNhAl3Xy/pYUmfix86SdIvFI2G70rPa2YHSbpV0j9I6lD0y2t4D8s9UtIBkj4r6RtmdmD8+PuS/lnSEEmHx+1TKrzGwZJeKX3QzB4ys02SnlT05nk61bxI0qj4w65oqq2FVMx61JyZjZB0nKTnUg+vkHSboi+rcsq+LyQtNbPlZnaHmQ0paVsk6aN9zbeVlVnXPa1nZNF/S1Tov11tQySdUq6t3VRaT2XWUWjfedtotQHUEElrUiPYtM64XZKecPf73X1LmV0jp0p60N1/7+7/J+kbikbC3fm2u2909/mK9tt+VJLc/Rl3/4O7vxf/epsuaXyF19hV0rrSB939BEmDFL0h/8vdt6Sa16WeWzTV1kIqZj1q6X4ze1vS7xX94vpOSft3JZ1oZn9d5rml74s1kj6uaFfKxxS9N+4qec46FfM90Qjdrevu1jOy6L9bdfee+kHcNl/RermkAfkUVaX1VGkdhfadt43C788usUbSEDPrV6bjd8TtkrSsm9cYlm5393fN7M0elrsyFb+r6OA5mdn+kr6vaH/wzorW5zMVXuMtRW+abbj7XyQ9YmZTzew1d58dN3XN/3YP+TVDtbWQilmPDDO7VdKZ8WR/Sf3M7OR4+k/u/pFunn6yu/+mUqO7v2FmP5T074oO0k3LvC/iLQBdv8hWmdkFkjrNbJC7d324DFIx3xONsM26NjNJPa5nZNF/t+qu/17k7jU5izAAlfpepXUU2nfeNlptC9QTkjZL+vv0g2Y2UNH+1jnxQ939AuqUtFfquQMk7dHLfH4k6WVJ+8UHxU2TZBXmfUHRQXbd6ScpfQzHgYrOJFrby/zqqdpaSMWsR4a7T3H3Xd19V0W7De7umu7hw7da10r6G0VbldJ6el90rbt0Xz1Q0a89bKvSekYW/Rf1Ftp33jZaagDl7u8oOsbhZjM7xsx2MLNRkn4uabmk/6ziZX6haDP/p+IDFL+lKjtpGYMUnY653sw+LOmfupn3YaU2R5vZh83sWDMbEP8/zpL0aWUPuhuv6GC9wqlRLaTm1aOh3P1tSddL+teSptL3xSfM7AAz287M9lB0cPS8eH13Kez7otm6Wc9Iof+iAYL6ziunpQZQkuTu1yj6ZXKdos72pKJNyJ91981VPP9FSRdK+pmiX0/rJa1W9Gssr39RtNl4naIDWCteBMzdn5X0jpl9In7IFH3YrFZ0XYypkk6P5+syUdFxAIXU11rEr9GUejTJTYoOlE27U9Jx8S93SforSb9W9H9YqGg9TOya2aJTg9fHlzNAeeXWM0rQf1FPIX7nlTL3no73C1u8yfptRZuNX6/zsv5W0hR3P7mKeU+U9CV3P62eORVNI+tRFGb2HUmr3f3GKuadJenH7v5w/TMD8mnH/ovKQv/Oa8sBVFyoOYpGxNdL+oSkQ70dV0YBUA+gddF/0a5abhdejZwk6X/jv/0knUFnbyrqAbQu+i/aUltugQIAAOiLPm2Bis/eeMXMXjOzK2qVFJqDeoaDWoaFeoaDWoaj11ugzGx7SYsVXfJ/uaQ/Spro7i/VLj00CvUMB7UMC/UMB7UMS1+uRD5O0mvu/j+SZGY/U7QvvOIbwczYX9hk7l7pGi256kktm69WtYznoZ5NRt8MB30zLJXq2ZddeMOVvcT/cpW5iaSZnWdmT5vZ06VtKJQe60ktWwZ9Myz0zXDQNwNS93vhufsMSTMkRtKtjlqGhXqGg1qGhXq2hr5sgVohaURqeq/4MbQm6hkOahkW6hkOahmQvgyg/ihpPzPbJ74H0hmSZvfwHBQX9QwHtQwL9QwHtQxIr3fhuft7ZnaBpEclbS/p9vi+SGhB1DMc1DIs1DMc1DIsDb2QJvtym6+bs0NyoZbNV6taStSzCOib4aBvhqUeZ+EBAAC0JQZQAAAAOdX9MgYh23///ZN4+vTpmba77747iW+77baG5YTeoZZAMaX7Zmn/u+uuu5J4xowZDcsJkNgCBQAAkBsDKAAAgJwYQAEAAOTEZQxySO+Ll6Rf/epXSbzPPvtk2pYtW1axrZk4VTpSj1oOHTo0M71x48YkXrt2ba/y7A6nStfXXnvtlZnesGFDEr/11ls1Xx59M1LaNx955JEk7q5vjhw5sr6J5UDfDAuXMQAAAKgRBlAAAAA5cRmDHkydOrVsLEl77713xectXbq0bjmhd3pby9WrVyfxKaeckmkbPnx4Ep977rmZtt/97ndJPGXKlHzJom523nnnJD7zzDMzbeldRNOmTcu0Pfzww0l8/PHH1ym79nTxxReXjSU+Z1FcbIECAADIiQEUAABATgygAAAAcuIYqDL69du6Wg466KAkLj1NNn0JiMWLF2fazjrrrDplhzzStTz99NOTuLtalh5XMWvWrLKxJA0ZMiSJS98D6VOsUXvjxo1L4qeeeqrifDvuuGNmevLkyUl88803Z9o6OjqSeMGCBZm21157rVd5orxafM6WHsMGNBJboAAAAHJiAAUAAJATVyIv4/zzz0/im266KYnNshcjTa+7J554ItN21FFH1Sm7vmm3qx1Ty+q0Sj1D1m5989Of/nQSX3TRRUn8gQ98IDPfkUcemcQvvPBCpu2II46oU3Z9045989RTT03iCRMmJPGbb76Zme+GG25I4jVr1tQ/sRrgSuQAAAA1wgAKAAAgJwZQAAAAOXEZA0nDhg3LTJ9zzjlJnD5WZrvtsuPNLVu2JPFll11Wp+yQB7UEWsOgQYOSOH1sU/pWO1L2sgaXXnpp/RNDVcaOHZuZvvPOO5N406ZNSbx27drMfOljTB966KE6ZdcYbIECAADIiQEUAABATuzC07Z3+z744IOTOH16e3o3jyQ9+OCDSfzss8/WKTvkQS2B1nDGGWckcf/+/ZN4xIgRmfnSV4CnbxbH4YcfnpmePXt2Eq9fvz6Jn3nmmcx8rb7bLo0tUAAAADn1OIAys9vNbLWZLUw9truZPWZmr8b/7lbfNFEr1DMc1DIs1DMc1LI9VLMFaqakY0oeu0LSHHffT9KceBqtYaaoZyhmilqGZKaoZyhmiloGr6pbuZjZKEkPufuYePoVSUe7e6eZdUia5+4HVPE6hbwk/ZgxYzLTc+bMSeI99tgjiUtv/5G+DP1nPvOZTNuLL75YyxRrxt2tFvWkls1Xq1rGzytkPdtJu/XNuXPnJnG1ffPoo4/OtNE30Qi1vpXLUHfvjOOVkob28nVQDNQzHNQyLNQzHNQyMH0+C8+joXbFEbKZnSfpvL4uB43RXT2pZWuhb4aFvhkO+mYYejuAWmVmHalNkasrzejuMyTNkIq7KXLhwoWZ6fvvvz+JJ0+eXPF56c3OU6ZMybSdf/75NcquIaqqJ7VsCUH1TYTbN3/5y18mcfqOAaXSffPYY4/NtBV1F14F9M0S48ePz0w//vjjTcqkd3q7C2+2pElxPEnSA7VJB01CPcNBLcNCPcNBLQNTzWUM7pH0hKQDzGy5mU2W9D1JnzOzVyVNiKfRAqhnOKhlWKhnOKhle+hxF567T6zQ9Nka54IGoJ7hoJZhoZ7hoJbtoarLGNRsYS2yLzd9K4HXX389iUtPr02vu87OzkzbCSeckMTz58+vdYq9Vul0zLxapZajRo1K4vQtIahlVqvUs1+/rb/53nvvvSZmUnvt1jfTn7NLly5N4gULFmTmS/e50mOerr766jpl1zft2Dcr+eIXv1hxOv35LEnXX399Et977711zSuPWl/GAAAAoG0xgAIAAMiJXXg9SG9SvOSSSzJtW7Zsqfi85cuXJ/HIkSNrn1gvtdtugjRqWVkr1jM07dw3L7jggiRO79qTpI0bNybxwIEDM20bNmxI4m9+85t1yi6/du+b6csT3H777Zm2QYMGJfGqVasybUuWLEniE088sT7J9QK78AAAAGqEARQAAEBO7MLrweDBg5P4uOOOy7TNmDEjiQcMGJBpe//995N4+vTpmbb0Js3nn3++JnlWq513E6RrWXq18WnTpiVxu9VSas16po0ePToznT7jslW0c9/szsyZM5N48+bNmbZFixYl8csvv5xpW7lyZRLTNxtr+PDhSXzddddl2t59990k3nvvvTNtEyZMSOJrrrkm03b55ZfXMsVc2IUHAABQIwygAAAAcmIABQAAkBPHQPVB+m7iRx99dKYtfapmqfSpm2PHjs20vfHGG7VJrgKOsyivnWsphVfPVkTfLO+BB7bec7e0b6aPdSu9Svn222+fxJMmTcq01fsq9vTN3pk4cesdcL7whS9k2p577rkkvvvuuzNt6SvZ1wPHQAEAANQIAygAAICc+vU8CypJb2L8yle+kmm75ZZbKj7vQx/6UBL379+/9okhN2oJFNNJJ52UxF/96lczbbfeemsSl95NIH1Zg9BuPB2qe+65J4nTNw6Xsjd1f/TRRzNt9d6FVwlboAAAAHJiAAUAAJATAygAAICcOAaqRubPn9/sFFAj1BIoptJbsqRv17Js2bJM25577tmQnFA7Y8aMSeJx48Zl2hYvXtzodHrEFigAAICcGEABAADkFPQuvPHjx1dse/zxx/v8+ueee24SX3nllZk2s8oXot1uO8ateVFLoJjq3TfTrz9hwoRMW/qK1MOGDcu0HXXUUUk8fPjwTNuKFSv6nBd655hjjkniL3/5y5m2Aw88MIlLd9emLyeTvkRFM/HpDwAAkBMDKAAAgJwYQAEAAOQU1DFQpfvA03fx/u1vf5tp++AHP1jVa37+859P4tJ9/UOHDk3i9J2/Jcl96w20S/flpm9NkD4NF1t1V8vS05WvvvrqJN68eXPF16SWxXTAAQdkptO3Zdi0aVOj00EPSvvmgw8+mMSlxzyl+1V39t133yTebbfdMm077bRTEr/11luZth133DGJBw8enGkbO3ZsEtM3Gyv9uXjaaadl2kaMGJHEo0ePzrQtWbIkia+99tpMW1GOe0pjCxQAAEBOPQ6gzGyEmc01s5fM7EUzmxo/vruZPWZmr8b/7tbTa6H5qGU46JthoZbhoG+2B0vvnig7g1mHpA53f9bMBkl6RtLJks6W9Gd3/56ZXSFpN3e/vIfX6n5hfZTeNChJr7/+enrZmbae/t/ldPca69aty7RdfvnWVZHexC1JnZ2duZddQx+jlsHUcphapG+iKi3ZN9O7XHvbN9O72BYuXJhp27hxYxK/8847mbb77rsviZ988slMG32zedJXFD/11FMzbeldraWXgZk+fXoSv/TSS3XKLj93L3stmx63QLl7p7s/G8frJC2SNFzSSZJ+Es/2E0VvDhQctQwHfTMs1DIc9M32kOsgcjMbJekQSU9KGuruXUP8lZLKHi1oZudJOq/3KaIeqGVYqGc4qGVYqGe4qj6I3MwGSpol6WJ3X5tu82g7bdnNjO4+w90Pc/fD+pQpaoZahoV6hoNahoV6hq2qLVBmtoOiN8Fd7t6103mVmXW4e2d8nNTqeiVZrffffz8znT6WZZdddunz6y9fvjwz/dxzzyXxTTfdlGmbO3dun5dXD9QyEkItpdapJ3rWKrWsR99MX1am9LIW8+fPT+I77rgj00bfLKb0JQeuuuqqTFvp+6eVVXMWnkn6saRF7v79VNNsSZPieJKkB0qfi0KiloGgbwaHWgaCvtkeqtkCdYSkL0laYGZdVxGcJul7kn5uZpMlLZV0WoXno1ioZTjom2GhluGgb7aBHi9jUNOFNfh0zPTVpg855JCK81144YWZ6Xnz5iXxggULkvjGG2+sXXJNUul0zLyKVMsBAwYk8TnnnJNpo5bVKdKp0gMHDkzi9evXNzGTxgqhbx566KEV57vooosy0+ndb+m+ecMNN9Qwu+YItW+2q15fxgAAAABZDKAAAAByYgAFAACQU9DHQGFbrXqcBbbFcRZhoW+Gg74ZFo6BAgAAqBEGUAAAADkxgAIAAMiJARQAAEBODKAAAAByYgAFAACQEwMoAACAnBhAAQAA5MQACgAAICcGUAAAADkxgAIAAMiJARQAAEBODKAAAAByYgAFAACQEwMoAACAnBhAAQAA5NSvwctbI2mppCFx3GztlsfIGr4WtaysEbnUspZSlO8Gtdc6rAZ9s++KkodE36yFotSz6X3T3L0Byy9ZqNnT7n5YwxdMHjVXlNyLkodUrFzyKFLeRcmlKHn0RlFyL0oeUrFyyaNIeRcllyLkwS48AACAnBhAAQAA5NSsAdSMJi23FHn0XVFyL0oeUrFyyaNIeRcll6Lk0RtFyb0oeUjFyiWPIuVdlFyankdTjoECAABoZezCAwAAyKmhAygzO8bMXjGz18zsigYv+3YzW21mC1OP7W5mj5nZq/G/uzUgjxFmNtfMXjKzF81sarNy6QtqGU4tJeoZLzOIelLLcGopUc8i17JhAygz217SLZKOlXSQpIlmdlCjli9ppqRjSh67QtIcd99P0px4ut7ek3Spux8k6ZOSzo/XQzNy6RVqmWj5WkrUM6Xl60ktEy1fS4l6xopbS3dvyJ+kwyU9mpq+UtKVjVp+vMxRkhampl+R1BHHHZJeaWQ+8XIfkPS5IuRCLduvltQzrHpSy3BqST2LX8tG7sIbLmlZanp5/FgzDXX3zjheKWloIxduZqMkHSLpyWbnkhO1LNHCtZSo5zZauJ7UskQL11KinhlFqyUHkcc8GsY27JREMxsoaZaki919bTNzCQ21DAv1DAe1DEsj12ERa9nIAdQKSSNS03vFjzXTKjPrkKT439WNWKiZ7aDojXCXu9/XzFx6iVrGAqilRD0TAdSTWsYCqKVEPRUvp5C1bOQA6o+S9jOzfcysv6QzJM1u4PLLmS1pUhxPUrRvta7MzCT9WNIid/9+M3PpA2qpYGopUU9JwdSTWiqYWkrUs9i1bPDBX8dJWizpvyV9rcHLvkdSp6S/KNqPPFnSHoqO3n9V0m8k7d6API5UtKnxBUnPx3/HNSMXakktqWd49aSW4dSSeha7llyJHAAAICcOIgcAAMiJARQAAEBODKAAAAByYgAFAACQEwMoAACAnBhAAQAA5MQACgAAICcGUAAAADn9P3RoMWKMTXGaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# rescale values from [-0.5, 0.5] to [0, 255] for plotting\n",
    "fig0 = (input_image[:,:,0] + 0.5)*255\n",
    "\n",
    "fig1 = (adv_pn[0,:,:,0] + 0.5) * 255\n",
    "fig2 = (fig1 - fig0) #rescaled delta_pn\n",
    "fig3 = (adv_pp[0,:,:,0] + 0.5) * 255\n",
    "fig4 = (fig0 - fig3) #rescaled delta_pp\n",
    "\n",
    "f, axarr = plt.subplots(1, 5, figsize=(10,10))\n",
    "axarr[0].set_title(\"Original\" + \"(\" + str(mymodel.predict_classes(np.expand_dims(input_image, axis=0))[0]) + \")\")\n",
    "axarr[1].set_title(\"Original + PN\" + \"(\" + str(mymodel.predict_classes(adv_pn)[0]) + \")\")\n",
    "axarr[2].set_title(\"PN\")\n",
    "axarr[3].set_title(\"Original + PP\")\n",
    "axarr[4].set_title(\"PP\" + \"(\" + str(mymodel.predict_classes(delta_pp)[0]) + \")\")\n",
    "\n",
    "axarr[0].imshow(fig0, cmap=\"gray\")\n",
    "axarr[1].imshow(fig1, cmap=\"gray\")\n",
    "axarr[2].imshow(fig2, cmap=\"gray\")\n",
    "axarr[3].imshow(fig3, cmap=\"gray\")\n",
    "axarr[4].imshow(fig4, cmap=\"gray\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Explanation: \n",
    "- The PP highlights the minimum set of pixels which were present in the image for it to be classified as digit 3. Note that both the original image and PP are classified as digit 3 by the classifier. \n",
    "- The PN highlights a small horizontal line at the top whose presence would change the classification of the original image to digit 5 and thus should be absent for the classification to remain digit 3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}