19_training_and_deploying_at_scale.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 19 – Training and Deploying TensorFlow Models at Scale**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code in chapter 19._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/19_training_and_deploying_at_scale.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU was detected. CNNs can be very slow without a GPU.\n"
     ]
    }
   ],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    !echo \"deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" > /etc/apt/sources.list.d/tensorflow-serving.list\n",
    "    !curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -\n",
    "    !apt update && apt-get install -y tensorflow-model-server\n",
    "    !pip install -q -U tensorflow-serving-api\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deploy\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying TensorFlow models to TensorFlow Serving (TFS)\n",
    "We will use the REST API or the gRPC API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save/Load a `SavedModel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.mnist.load_data()\n",
    "X_train_full = X_train_full[..., np.newaxis].astype(np.float32) / 255.\n",
    "X_test = X_test[..., np.newaxis].astype(np.float32) / 255.\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "X_new = X_test[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.7018 - accuracy: 0.8223 - val_loss: 0.3722 - val_accuracy: 0.9022\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 2s 36us/sample - loss: 0.3528 - accuracy: 0.9021 - val_loss: 0.3000 - val_accuracy: 0.9170\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 2s 36us/sample - loss: 0.3032 - accuracy: 0.9150 - val_loss: 0.2659 - val_accuracy: 0.9280\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.2730 - accuracy: 0.9233 - val_loss: 0.2442 - val_accuracy: 0.9342\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.2504 - accuracy: 0.9305 - val_loss: 0.2272 - val_accuracy: 0.9346\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.2319 - accuracy: 0.9353 - val_loss: 0.2104 - val_accuracy: 0.9418\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.2156 - accuracy: 0.9395 - val_loss: 0.1987 - val_accuracy: 0.9484\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 2s 36us/sample - loss: 0.2019 - accuracy: 0.9434 - val_loss: 0.1893 - val_accuracy: 0.9496\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.1898 - accuracy: 0.9471 - val_loss: 0.1765 - val_accuracy: 0.9526\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.1791 - accuracy: 0.9495 - val_loss: 0.1691 - val_accuracy: 0.9550\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x13d74aba8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28, 1]),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.99, 0.01, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.96, 0.01, 0.  , 0.  , 0.  , 0.  , 0.01, 0.01, 0.  ]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(model.predict(X_new), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my_mnist_model/0001'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_version = \"0001\"\n",
    "model_name = \"my_mnist_model\"\n",
    "model_path = os.path.join(model_name, model_version)\n",
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {model_name}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.saved_model.save(model, model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_mnist_model/\n",
      "    0001/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(model_name):\n",
    "    indent = '    ' * root.count(os.sep)\n",
    "    print('{}{}/'.format(indent, os.path.basename(root)))\n",
    "    for filename in files:\n",
    "        print('{}{}'.format(indent + '    ', filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel contains the following tag-sets:\n",
      "serve\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:\n",
      "SignatureDef key: \"__saved_model_init_op\"\n",
      "SignatureDef key: \"serving_default\"\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --tag_set serve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['flatten_2_input'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 28, 28, 1)\n",
      "      name: serving_default_flatten_2_input:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['dense_5'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 10)\n",
      "      name: StatefulPartitionedCall:0\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --tag_set serve \\\n",
    "                      --signature_def serving_default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
      "\n",
      "signature_def['__saved_model_init_op']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['__saved_model_init_op'] tensor_info:\n",
      "        dtype: DT_INVALID\n",
      "        shape: unknown_rank\n",
      "        name: NoOp\n",
      "  Method name is: \n",
      "\n",
      "signature_def['serving_default']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['flatten_2_input'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 28, 28, 1)\n",
      "        name: serving_default_flatten_2_input:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['dense_5'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 10)\n",
      "        name: StatefulPartitionedCall:0\n",
      "  Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the new instances to a `npy` file so we can pass them easily to our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(\"my_mnist_tests.npy\", X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'flatten_2_input'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_name = model.input_names[0]\n",
    "input_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's use `saved_model_cli` to make predictions for the instances we just saved:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-06-10 10:56:43.396851: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0610 10:56:43.397369 140735810999168 deprecation.py:323] From /Users/ageron/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/tools/saved_model_cli.py:339: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.\n",
      "W0610 10:56:43.421489 140735810999168 deprecation.py:323] From /Users/ageron/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "Result for output key dense_5:\n",
      "[[1.17575204e-04 1.13160660e-07 5.96997386e-04 2.08104262e-03\n",
      "  2.57820852e-06 6.44166794e-05 2.77263990e-08 9.96703804e-01\n",
      "  3.96052455e-05 3.93810158e-04]\n",
      " [1.22226949e-03 2.92685600e-05 9.86054957e-01 9.63000767e-03\n",
      "  8.81790996e-08 2.88744748e-04 1.58111588e-03 1.12290488e-09\n",
      "  1.19344448e-03 1.09315742e-07]\n",
      " [6.40679718e-05 9.63618696e-01 9.04400647e-03 2.98595289e-03\n",
      "  5.95759891e-04 3.74212675e-03 2.50709383e-03 1.14931818e-02\n",
      "  5.52693009e-03 4.22279176e-04]]\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli run --dir {model_path} --tag_set serve \\\n",
    "                     --signature_def serving_default    \\\n",
    "                     --inputs {input_name}=my_mnist_tests.npy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.99, 0.01, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.96, 0.01, 0.  , 0.  , 0.  , 0.  , 0.01, 0.01, 0.  ]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round([[1.1739199e-04, 1.1239604e-07, 6.0210604e-04, 2.0804715e-03, 2.5779348e-06,\n",
    "           6.4079795e-05, 2.7411186e-08, 9.9669880e-01, 3.9654213e-05, 3.9471846e-04],\n",
    "          [1.2294615e-03, 2.9207937e-05, 9.8599273e-01, 9.6755642e-03, 8.8930705e-08,\n",
    "           2.9156188e-04, 1.5831805e-03, 1.1311053e-09, 1.1980456e-03, 1.1113169e-07],\n",
    "          [6.4066830e-05, 9.6359509e-01, 9.0598064e-03, 2.9872139e-03, 5.9552520e-04,\n",
    "           3.7478798e-03, 2.5074568e-03, 1.1462728e-02, 5.5553433e-03, 4.2495009e-04]], 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install [Docker](https://docs.docker.com/install/) if you don't have it already. Then run:\n",
    "\n",
    "```bash\n",
    "docker pull tensorflow/serving\n",
    "\n",
    "export ML_PATH=$HOME/ml # or wherever this project is\n",
    "docker run -it --rm -p 8500:8500 -p 8501:8501 \\\n",
    "   -v \"$ML_PATH/my_mnist_model:/models/my_mnist_model\" \\\n",
    "   -e MODEL_NAME=my_mnist_model \\\n",
    "   tensorflow/serving\n",
    "```\n",
    "Once you are finished using it, press Ctrl-C to shut down the server."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, if `tensorflow_model_server` is installed (e.g., if you are running this notebook in Colab), then the following 3 cells will start the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"MODEL_DIR\"] = os.path.split(os.path.abspath(model_path))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash --bg\n",
    "nohup tensorflow_model_server \\\n",
    "     --rest_api_port=8501 \\\n",
    "     --model_name=my_mnist_model \\\n",
    "     --model_base_path=\"${MODEL_DIR}\" >server.log 2>&1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-06 13:04:12.267136: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-11-06 13:04:12.283035: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.\n",
      "2019-11-06 13:04:12.300096: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: /content/my_mnist_model/0002\n",
      "2019-11-06 13:04:12.304438: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 39993 microseconds.\n",
      "2019-11-06 13:04:12.304900: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at /content/my_mnist_model/0002/assets.extra/tf_serving_warmup_requests\n",
      "2019-11-06 13:04:12.305057: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: my_mnist_model version: 2}\n",
      "2019-11-06 13:04:12.306462: I tensorflow_serving/model_servers/server.cc:353] Running gRPC ModelServer at 0.0.0.0:8500 ...\n",
      "[warn] getaddrinfo: address family for nodename not supported\n",
      "2019-11-06 13:04:12.307179: I tensorflow_serving/model_servers/server.cc:373] Exporting HTTP/REST API at:localhost:8501 ...\n",
      "[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...\n"
     ]
    }
   ],
   "source": [
    "!tail server.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "input_data_json = json.dumps({\n",
    "    \"signature_name\": \"serving_default\",\n",
    "    \"instances\": X_new.tolist(),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\'{\"signature_name\": \"serving_default\", \"instances\": [[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.3294117748737335, 0.7254902124404907, 0.6235294342041016, 0.5921568870544434, 0.23529411852359772, 0.1411764770746231, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.8705882430076599, 0.9960784316062927, 0.9960784316062927, 0.9960784316062927, 0.9960784316062927, 0.9450980424880981, 0.7764706015586853, 0.7764706015586853, 0.7764706015586853, 0.776470...'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repr(input_data_json)[:1500] + \"...\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use TensorFlow Serving's REST API to make predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict'\n",
    "response = requests.post(SERVER_URL, data=input_data_json)\n",
    "response.raise_for_status() # raise an exception in case of error\n",
    "response = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['predictions'])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.99, 0.01, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.96, 0.01, 0.  , 0.  , 0.  , 0.  , 0.01, 0.01, 0.  ]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba = np.array(response[\"predictions\"])\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the gRPC API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow_serving.apis.predict_pb2 import PredictRequest\n",
    "\n",
    "request = PredictRequest()\n",
    "request.model_spec.name = model_name\n",
    "request.model_spec.signature_name = \"serving_default\"\n",
    "input_name = model.input_names[0]\n",
    "request.inputs[input_name].CopyFrom(tf.make_tensor_proto(X_new))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import grpc\n",
    "from tensorflow_serving.apis import prediction_service_pb2_grpc\n",
    "\n",
    "channel = grpc.insecure_channel('localhost:8500')\n",
    "predict_service = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n",
    "response = predict_service.Predict(request, timeout=10.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outputs {\n",
       "  key: \"dense_4\"\n",
       "  value {\n",
       "    dtype: DT_FLOAT\n",
       "    tensor_shape {\n",
       "      dim {\n",
       "        size: 3\n",
       "      }\n",
       "      dim {\n",
       "        size: 10\n",
       "      }\n",
       "    }\n",
       "    float_val: 2.0824443708988838e-05\n",
       "    float_val: 1.4913139168015732e-08\n",
       "    float_val: 0.0004813199338968843\n",
       "    float_val: 0.001888890634290874\n",
       "    float_val: 2.682592992186983e-07\n",
       "    float_val: 8.666840585647151e-06\n",
       "    float_val: 1.6853943241024183e-10\n",
       "    float_val: 0.9975269436836243\n",
       "    float_val: 3.833709342870861e-05\n",
       "    float_val: 3.4738284739432856e-05\n",
       "    float_val: 0.00017358684272039682\n",
       "    float_val: 0.0002858016814570874\n",
       "    float_val: 0.9816810488700867\n",
       "    float_val: 0.0157401692122221\n",
       "    float_val: 1.1949770339914068e-10\n",
       "    float_val: 0.00023017563216853887\n",
       "    float_val: 3.078056761296466e-05\n",
       "    float_val: 5.393230750883049e-09\n",
       "    float_val: 0.0018584482604637742\n",
       "    float_val: 1.8884094288296183e-09\n",
       "    float_val: 3.397366526769474e-05\n",
       "    float_val: 0.9835277795791626\n",
       "    float_val: 0.001533020636998117\n",
       "    float_val: 0.0014515116345137358\n",
       "    float_val: 0.00018795969663187861\n",
       "    float_val: 0.0011680654715746641\n",
       "    float_val: 0.0014667459763586521\n",
       "    float_val: 0.006120447069406509\n",
       "    float_val: 0.004315734840929508\n",
       "    float_val: 0.00019466254161670804\n",
       "  }\n",
       "}\n",
       "model_spec {\n",
       "  name: \"my_mnist_model\"\n",
       "  version {\n",
       "    value: 2\n",
       "  }\n",
       "  signature_name: \"serving_default\"\n",
       "}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the response to a tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.98, 0.02, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.98, 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  ]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_name = model.output_names[0]\n",
    "outputs_proto = response.outputs[output_name]\n",
    "y_proba = tf.make_ndarray(outputs_proto)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or to a NumPy array if your client does not include the TensorFlow library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.98, 0.02, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.98, 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  ]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_name = model.output_names[0]\n",
    "outputs_proto = response.outputs[output_name]\n",
    "shape = [dim.size for dim in outputs_proto.tensor_shape.dim]\n",
    "y_proba = np.array(outputs_proto.float_val).reshape(shape)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying a new model version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.7035 - accuracy: 0.8060 - val_loss: 0.3445 - val_accuracy: 0.9032\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.3213 - accuracy: 0.9084 - val_loss: 0.2660 - val_accuracy: 0.9252\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.2663 - accuracy: 0.9236 - val_loss: 0.2304 - val_accuracy: 0.9392\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.2331 - accuracy: 0.9331 - val_loss: 0.2069 - val_accuracy: 0.9430\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.2105 - accuracy: 0.9390 - val_loss: 0.1910 - val_accuracy: 0.9446\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.1924 - accuracy: 0.9442 - val_loss: 0.1732 - val_accuracy: 0.9518\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.1771 - accuracy: 0.9489 - val_loss: 0.1679 - val_accuracy: 0.9526\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.1650 - accuracy: 0.9527 - val_loss: 0.1574 - val_accuracy: 0.9546\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.1540 - accuracy: 0.9555 - val_loss: 0.1446 - val_accuracy: 0.9590\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 2s 35us/sample - loss: 0.1448 - accuracy: 0.9583 - val_loss: 0.1414 - val_accuracy: 0.9608\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x12f58f908>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28, 1]),\n",
    "    keras.layers.Dense(50, activation=\"relu\"),\n",
    "    keras.layers.Dense(50, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my_mnist_model/0002'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_version = \"0002\"\n",
    "model_name = \"my_mnist_model\"\n",
    "model_path = os.path.join(model_name, model_version)\n",
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.saved_model.save(model, model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_mnist_model/\n",
      "    0002/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n",
      "    0001/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(model_name):\n",
    "    indent = '    ' * root.count(os.sep)\n",
    "    print('{}{}/'.format(indent, os.path.basename(root)))\n",
    "    for filename in files:\n",
    "        print('{}{}'.format(indent + '    ', filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: You may need to wait a minute before the new model is loaded by TensorFlow Serving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict'\n",
    "            \n",
    "response = requests.post(SERVER_URL, data=input_data_json)\n",
    "response.raise_for_status()\n",
    "response = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['predictions'])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.99, 0.01, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.96, 0.01, 0.  , 0.  , 0.  , 0.  , 0.01, 0.01, 0.  ]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba = np.array(response[\"predictions\"])\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the model to Google Cloud AI Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow the instructions in the book to deploy the model to Google Cloud AI Platform, download the service account's private key and save it to the `my_service_account_private_key.json` in the project directory. Also, update the `project_id`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_id = \"onyx-smoke-242003\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import googleapiclient.discovery\n",
    "\n",
    "os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"my_service_account_private_key.json\"\n",
    "model_id = \"my_mnist_model\"\n",
    "model_path = \"projects/{}/models/{}\".format(project_id, model_id)\n",
    "model_path += \"/versions/v0001/\" # if you want to run a specific version\n",
    "ml_resource = googleapiclient.discovery.build(\"ml\", \"v1\").projects()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X):\n",
    "    input_data_json = {\"signature_name\": \"serving_default\",\n",
    "                       \"instances\": X.tolist()}\n",
    "    request = ml_resource.predict(name=model_path, body=input_data_json)\n",
    "    response = request.execute()\n",
    "    if \"error\" in response:\n",
    "        raise RuntimeError(response[\"error\"])\n",
    "    return np.array([pred[output_name] for pred in response[\"predictions\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.99, 0.01, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.96, 0.01, 0.  , 0.  , 0.  , 0.  , 0.01, 0.01, 0.  ]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_probas = predict(X_new)\n",
    "np.round(Y_probas, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.test.is_gpu_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.test.gpu_device_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.test.is_built_with_cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[name: \"/device:CPU:0\"\n",
       " device_type: \"CPU\"\n",
       " memory_limit: 268435456\n",
       " locality {\n",
       " }\n",
       " incarnation: 11178133101787456811]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.python.client.device_lib import list_local_devices\n",
    "\n",
    "devices = list_local_devices()\n",
    "devices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_model():\n",
    "    return keras.models.Sequential([\n",
    "        keras.layers.Conv2D(filters=64, kernel_size=7, activation=\"relu\",\n",
    "                            padding=\"same\", input_shape=[28, 28, 1]),\n",
    "        keras.layers.MaxPooling2D(pool_size=2),\n",
    "        keras.layers.Conv2D(filters=128, kernel_size=3, activation=\"relu\",\n",
    "                            padding=\"same\"), \n",
    "        keras.layers.Conv2D(filters=128, kernel_size=3, activation=\"relu\",\n",
    "                            padding=\"same\"),\n",
    "        keras.layers.MaxPooling2D(pool_size=2),\n",
    "        keras.layers.Flatten(),\n",
    "        keras.layers.Dense(units=64, activation='relu'),\n",
    "        keras.layers.Dropout(0.5),\n",
    "        keras.layers.Dense(units=10, activation='softmax'),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "model = create_model()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(X_train, y_train, epochs=10,\n",
    "          validation_data=(X_valid, y_valid), batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0603 15:31:26.178871 140735810999168 cross_device_ops.py:1178] There is non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "distribution = tf.distribute.MirroredStrategy()\n",
    "\n",
    "# Change the default all-reduce algorithm:\n",
    "#distribution = tf.distribute.MirroredStrategy(\n",
    "#    cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())\n",
    "\n",
    "# Specify the list of GPUs to use:\n",
    "#distribution = tf.distribute.MirroredStrategy(devices=[\"/gpu:0\", \"/gpu:1\"])\n",
    "\n",
    "# Use the central storage strategy instead:\n",
    "#distribution = tf.distribute.experimental.CentralStorageStrategy()\n",
    "\n",
    "#resolver = tf.distribute.cluster_resolver.TPUClusterResolver()\n",
    "#tf.tpu.experimental.initialize_tpu_system(resolver)\n",
    "#distribution = tf.distribute.experimental.TPUStrategy(resolver)\n",
    "\n",
    "with distribution.scope():\n",
    "    model = create_model()\n",
    "    model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                  optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "                  metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100 # must be divisible by the number of workers\n",
    "model.fit(X_train, y_train, epochs=10,\n",
    "          validation_data=(X_valid, y_valid), batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.09101252, 0.07083996, 0.06410537, 0.11957529, 0.06693752,\n",
       "        0.05124901, 0.04676544, 0.23180223, 0.13522181, 0.12249089],\n",
       "       [0.08099081, 0.12387844, 0.14915964, 0.13171668, 0.05875394,\n",
       "        0.08834281, 0.16267018, 0.06899565, 0.07834874, 0.05714307],\n",
       "       [0.04303756, 0.2682051 , 0.0909673 , 0.11496522, 0.06084979,\n",
       "        0.07125981, 0.08520001, 0.08517107, 0.09236596, 0.0879782 ]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom training loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "K = keras.backend\n",
    "\n",
    "distribution = tf.distribute.MirroredStrategy()\n",
    "\n",
    "with distribution.scope():\n",
    "    model = create_model()\n",
    "    optimizer = keras.optimizers.SGD()\n",
    "\n",
    "with distribution.scope():\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)).repeat().batch(batch_size)\n",
    "    input_iterator = distribution.make_dataset_iterator(dataset)\n",
    "    \n",
    "@tf.function\n",
    "def train_step():\n",
    "    def step_fn(inputs):\n",
    "        X, y = inputs\n",
    "        with tf.GradientTape() as tape:\n",
    "            Y_proba = model(X)\n",
    "            loss = K.sum(keras.losses.sparse_categorical_crossentropy(y, Y_proba)) / batch_size\n",
    "\n",
    "        grads = tape.gradient(loss, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "        return loss\n",
    "\n",
    "    per_replica_losses = distribution.experimental_run(step_fn, input_iterator)\n",
    "    mean_loss = distribution.reduce(tf.distribute.ReduceOp.SUM,\n",
    "                                    per_replica_losses, axis=None)\n",
    "    return mean_loss\n",
    "\n",
    "n_epochs = 10\n",
    "with distribution.scope():\n",
    "    input_iterator.initialize()\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {}/{}\".format(epoch + 1, n_epochs))\n",
    "        for iteration in range(len(X_train) // batch_size):\n",
    "            print(\"\\rLoss: {:.3f}\".format(train_step().numpy()), end=\"\")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100 # must be divisible by the number of workers\n",
    "model.fit(X_train, y_train, epochs=10,\n",
    "          validation_data=(X_valid, y_valid), batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training across multiple servers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A TensorFlow cluster is a group of TensorFlow processes running in parallel, usually on different machines, and talking to each other to complete some work, for example training or executing a neural network. Each TF process in the cluster is called a \"task\" (or a \"TF server\"). It has an IP address, a port, and a type (also called its role or its job). The type can be `\"worker\"`, `\"chief\"`, `\"ps\"` (parameter server) or `\"evaluator\"`:\n",
    "* Each **worker** performs computations, usually on a machine with one or more GPUs.\n",
    "* The **chief** performs computations as well, but it also handles extra work such as writing TensorBoard logs or saving checkpoints. There is a single chief in a cluster. If no chief is specified, then the first worker is the chief.\n",
    "* A **parameter server** (ps) only keeps track of variable values, it is usually on a CPU-only machine.\n",
    "* The **evaluator** obviously takes care of evaluation. There is usually a single evaluator in a cluster.\n",
    "\n",
    "The set of tasks that share the same type is often called a \"job\". For example, the \"worker\" job is the set of all workers.\n",
    "\n",
    "To start a TensorFlow cluster, you must first specify it. This means defining all the tasks (IP address, TCP port, and type). For example, the following cluster specification defines a cluster with 3 tasks (2 workers and 1 parameter server). It's a dictionary with one key per job, and the values are lists of task addresses:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"worker\": [\"my-worker0.example.com:9876\", \"my-worker1.example.com:9876\"],\n",
    "    \"ps\": [\"my-ps0.example.com:9876\"]\n",
    "}\n",
    "```\n",
    "\n",
    "Every task in the cluster may communicate with every other task in the server, so make sure to configure your firewall to authorize all communications between these machines on these ports (it's usually simpler if you use the same port on every machine).\n",
    "\n",
    "When a task is started, it needs to be told which one it is: its type and index (the task index is also called the task id). A common way to specify everything at once (both the cluster spec and the current task's type and id) is to set the `TF_CONFIG` environment variable before starting the program. It must be a JSON-encoded dictionary containing a cluster specification (under the `\"cluster\"` key), and the type and index of the task to start (under the `\"task\"` key). For example, the following `TF_CONFIG` environment variable defines a simple cluster with 2 workers and 1 parameter server, and specifies that the task to start is the first worker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF_CONFIG='{\"cluster\": {\"worker\": [\"my-work0.example.com:9876\", \"my-work1.example.com:9876\"], \"ps\": [\"my-ps0.example.com:9876\"]}, \"task\": {\"type\": \"worker\", \"index\": 0}}'\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "os.environ[\"TF_CONFIG\"] = json.dumps({\n",
    "    \"cluster\": {\n",
    "        \"worker\": [\"my-work0.example.com:9876\", \"my-work1.example.com:9876\"],\n",
    "        \"ps\": [\"my-ps0.example.com:9876\"]\n",
    "    },\n",
    "    \"task\": {\"type\": \"worker\", \"index\": 0}\n",
    "})\n",
    "print(\"TF_CONFIG='{}'\".format(os.environ[\"TF_CONFIG\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some platforms (e.g., Google Cloud ML Engine) automatically set this environment variable for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then you would write a short Python script to start a task. The same script can be used on every machine, since it will load the `TF_CONFIG` variable, which will tell it which task to start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "resolver = tf.distribute.cluster_resolver.TFConfigClusterResolver()\n",
    "worker0 = tf.distribute.Server(resolver.cluster_spec(),\n",
    "                               job_name=resolver.task_type,\n",
    "                               task_index=resolver.task_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to specify the cluster specification is directly in Python, rather than through an environment variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_spec = tf.train.ClusterSpec({\n",
    "    \"worker\": [\"127.0.0.1:9901\", \"127.0.0.1:9902\"],\n",
    "    \"ps\": [\"127.0.0.1:9903\"]\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then start a server simply by passing it the cluster spec and indicating its type and index. Let's start the two remaining tasks (remember that in general you would only start a single task per machine; we are starting 3 tasks on the localhost just for the purpose of this code example):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "#worker1 = tf.distribute.Server(cluster_spec, job_name=\"worker\", task_index=1)\n",
    "ps0 = tf.distribute.Server(cluster_spec, job_name=\"ps\", task_index=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'{\"cluster\": {\"worker\": [\"127.0.0.1:9901\", \"127.0.0.1:9902\"], \"ps\": [\"127.0.0.1:9903\"]}, \"task\": {\"type\": \"worker\", \"index\": 1}}'\n"
     ]
    }
   ],
   "source": [
    "os.environ[\"TF_CONFIG\"] = json.dumps({\n",
    "    \"cluster\": {\n",
    "        \"worker\": [\"127.0.0.1:9901\", \"127.0.0.1:9902\"],\n",
    "        \"ps\": [\"127.0.0.1:9903\"]\n",
    "    },\n",
    "    \"task\": {\"type\": \"worker\", \"index\": 1}\n",
    "})\n",
    "print(repr(os.environ[\"TF_CONFIG\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "distribution = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n",
    "\n",
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "os.environ[\"TF_CONFIG\"] = json.dumps({\n",
    "    \"cluster\": {\n",
    "        \"worker\": [\"127.0.0.1:9901\", \"127.0.0.1:9902\"],\n",
    "        \"ps\": [\"127.0.0.1:9903\"]\n",
    "    },\n",
    "    \"task\": {\"type\": \"worker\", \"index\": 1}\n",
    "})\n",
    "#CUDA_VISIBLE_DEVICES=0 \n",
    "\n",
    "with distribution.scope():\n",
    "    model = create_model()\n",
    "    model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                  optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "                  metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "\n",
    "# At the beginning of the program (restart the kernel before running this cell)\n",
    "distribution = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n",
    "\n",
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.mnist.load_data()\n",
    "X_train_full = X_train_full[..., np.newaxis] / 255.\n",
    "X_test = X_test[..., np.newaxis] / 255.\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "X_new = X_test[:3]\n",
    "\n",
    "n_workers = 2\n",
    "batch_size = 32 * n_workers\n",
    "dataset = tf.data.Dataset.from_tensor_slices((X_train[..., np.newaxis], y_train)).repeat().batch(batch_size)\n",
    "    \n",
    "def create_model():\n",
    "    return keras.models.Sequential([\n",
    "        keras.layers.Conv2D(filters=64, kernel_size=7, activation=\"relu\",\n",
    "                            padding=\"same\", input_shape=[28, 28, 1]),\n",
    "        keras.layers.MaxPooling2D(pool_size=2),\n",
    "        keras.layers.Conv2D(filters=128, kernel_size=3, activation=\"relu\",\n",
    "                            padding=\"same\"), \n",
    "        keras.layers.Conv2D(filters=128, kernel_size=3, activation=\"relu\",\n",
    "                            padding=\"same\"),\n",
    "        keras.layers.MaxPooling2D(pool_size=2),\n",
    "        keras.layers.Flatten(),\n",
    "        keras.layers.Dense(units=64, activation='relu'),\n",
    "        keras.layers.Dropout(0.5),\n",
    "        keras.layers.Dense(units=10, activation='softmax'),\n",
    "    ])\n",
    "\n",
    "with distribution.scope():\n",
    "    model = create_model()\n",
    "    model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                  optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "                  metrics=[\"accuracy\"])\n",
    "\n",
    "model.fit(dataset, steps_per_epoch=len(X_train)//batch_size, epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameter tuning\n",
    "\n",
    "# Only talk to ps server\n",
    "config_proto = tf.ConfigProto(device_filters=['/job:ps', '/job:worker/task:%d' % tf_config['task']['index']])\n",
    "config = tf.estimator.RunConfig(session_config=config_proto)\n",
    "# default since 1.10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "strategy.num_replicas_in_sync"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# My exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These were theoretical questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Train a model (any model you like) and deploy it to TF Serving or Google Cloud\n",
    "AI Platform. Write the client code to query it using the REST API or the gRPC API. Update the model and deploy the new version. Your client code will now\n",
    "query the new version. Roll back to the first version._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a model on the MNIST dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's load and split the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "test_set, valid_set, train_set = tfds.load(\n",
    "    \"mnist\",\n",
    "    split=[\"train[:15%]\", \"train[15%:25%]\", \"train[25%:]\"],\n",
    "    as_supervised=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: I could have scaled the inputs (divided by 255), but validation accuracy seems good nevertheless."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu', padding=\"SAME\", input_shape=(28, 28, 1)),\n",
    "    tf.keras.layers.Conv2D(filters=64, kernel_size=3, activation='relu', padding=\"SAME\"),\n",
    "    tf.keras.layers.MaxPooling2D(),\n",
    "    tf.keras.layers.Conv2D(filters=128, kernel_size=3, activation='relu', padding=\"SAME\"),\n",
    "    tf.keras.layers.Conv2D(filters=256, kernel_size=3, activation='relu', padding=\"SAME\"),\n",
    "    tf.keras.layers.MaxPooling2D(),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(units=1024, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(units=10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"nadam\", loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "1407/1407 [==============================] - 334s 238ms/step - loss: 0.3481 - accuracy: 0.9622 - val_loss: 0.0562 - val_accuracy: 0.9818\n",
      "Epoch 2/3\n",
      "1407/1407 [==============================] - 333s 237ms/step - loss: 0.0498 - accuracy: 0.9841 - val_loss: 0.0571 - val_accuracy: 0.9815\n",
      "Epoch 3/3\n",
      "1407/1407 [==============================] - 332s 236ms/step - loss: 0.0428 - accuracy: 0.9877 - val_loss: 0.0719 - val_accuracy: 0.9835\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7c45fce290>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_set.batch(32), validation_data=valid_set.batch(32), epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me save the model now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: I copied this and the code below from the notebook above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my_mnist_model/0001'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_version = \"0001\"\n",
    "model_name = \"my_mnist_model\"\n",
    "model_path = os.path.join(model_name, model_version)\n",
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_mnist_model/0001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_mnist_model/0001/assets\n"
     ]
    }
   ],
   "source": [
    "tf.saved_model.save(model, model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_mnist_model/\n",
      "    0001/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(model_name):\n",
    "    indent = '    ' * root.count(os.sep)\n",
    "    print('{}{}/'.format(indent, os.path.basename(root)))\n",
    "    for filename in files:\n",
    "        print('{}{}'.format(indent + '    ', filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-03-22 18:43:15.458691: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:15.458767: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:15.458782: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "The given SavedModel contains the following tag-sets:\n",
      "serve\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-03-22 18:43:19.876134: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:19.876195: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:19.876205: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:\n",
      "SignatureDef key: \"__saved_model_init_op\"\n",
      "SignatureDef key: \"serving_default\"\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --tag_set serve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-03-22 18:43:27.247408: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:27.247472: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:27.247481: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['conv2d_4_input'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 28, 28, 1)\n",
      "      name: serving_default_conv2d_4_input:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['dense_3'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 10)\n",
      "      name: StatefulPartitionedCall:0\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --tag_set serve \\\n",
    "                      --signature_def serving_default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-03-22 18:43:58.221626: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:58.221710: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory\n",
      "2021-03-22 18:43:58.221724: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\n",
      "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
      "\n",
      "signature_def['__saved_model_init_op']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['__saved_model_init_op'] tensor_info:\n",
      "        dtype: DT_INVALID\n",
      "        shape: unknown_rank\n",
      "        name: NoOp\n",
      "  Method name is: \n",
      "\n",
      "signature_def['serving_default']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['conv2d_4_input'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 28, 28, 1)\n",
      "        name: serving_default_conv2d_4_input:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['dense_3'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 10)\n",
      "        name: StatefulPartitionedCall:0\n",
      "  Method name is: tensorflow/serving/predict\n",
      "WARNING:tensorflow:From /home/mislav/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "\n",
      "Defined Functions:\n",
      "  Function Name: '__call__'\n",
      "    Option #1\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          conv2d_4_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='conv2d_4_input')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: True\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #2\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: True\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #3\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          conv2d_4_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='conv2d_4_input')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: False\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #4\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: False\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "\n",
      "  Function Name: '_default_save_signature'\n",
      "    Option #1\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          conv2d_4_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='conv2d_4_input')\n",
      "\n",
      "  Function Name: 'call_and_return_all_conditional_losses'\n",
      "    Option #1\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: False\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #2\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: True\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #3\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          conv2d_4_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='conv2d_4_input')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: True\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n",
      "    Option #4\n",
      "      Callable with:\n",
      "        Argument #1\n",
      "          conv2d_4_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='conv2d_4_input')\n",
      "        Argument #2\n",
      "          DType: bool\n",
      "          Value: False\n",
      "        Argument #3\n",
      "          DType: NoneType\n",
      "          Value: None\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {model_path} --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install [Docker](https://docs.docker.com/install/) if you don't have it already. Then run:\n",
    "\n",
    "```bash\n",
    "docker pull tensorflow/serving\n",
    "\n",
    "export ML_PATH=$HOME/ml # or wherever this project is\n",
    "docker run -it --rm -p 8500:8500 -p 8501:8501 \\\n",
    "   -v \"$ML_PATH/my_mnist_model:/models/my_mnist_model\" \\\n",
    "   -e MODEL_NAME=my_mnist_model \\\n",
    "   tensorflow/serving\n",
    "```\n",
    "Once you are finished using it, press Ctrl-C to shut down the server."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, if `tensorflow_model_server` is installed (e.g., if you are running this notebook in Colab), then the following 3 cells will start the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"MODEL_DIR\"] = os.path.split(os.path.abspath(model_path))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash --bg\n",
    "nohup tensorflow_model_server \\\n",
    "     --rest_api_port=8501 \\\n",
    "     --model_name=my_mnist_model \\\n",
    "     --model_base_path=\"${MODEL_DIR}\" >server.log 2>&1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-06 13:04:12.267136: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-11-06 13:04:12.283035: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.\n",
      "2019-11-06 13:04:12.300096: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: /content/my_mnist_model/0002\n",
      "2019-11-06 13:04:12.304438: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 39993 microseconds.\n",
      "2019-11-06 13:04:12.304900: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at /content/my_mnist_model/0002/assets.extra/tf_serving_warmup_requests\n",
      "2019-11-06 13:04:12.305057: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: my_mnist_model version: 2}\n",
      "2019-11-06 13:04:12.306462: I tensorflow_serving/model_servers/server.cc:353] Running gRPC ModelServer at 0.0.0.0:8500 ...\n",
      "[warn] getaddrinfo: address family for nodename not supported\n",
      "2019-11-06 13:04:12.307179: I tensorflow_serving/model_servers/server.cc:373] Exporting HTTP/REST API at:localhost:8501 ...\n",
      "[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...\n"
     ]
    }
   ],
   "source": [
    "!tail server.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First I need to preprocess the test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set_numpy_generator = tfds.as_numpy(test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object _eager_dataset_iterator at 0x7fede99f8f50>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set_numpy_generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set_numpy_array = []\n",
    "\n",
    "for entry in test_set_numpy_generator:\n",
    "    #print(entry)\n",
    "    test_set_numpy_array.append(entry[0].tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [84],\n",
       "  [254],\n",
       "  [101],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [174],\n",
       "  [253],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [31],\n",
       "  [247],\n",
       "  [202],\n",
       "  [29],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [1],\n",
       "  [1],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [141],\n",
       "  [253],\n",
       "  [168],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [66],\n",
       "  [208],\n",
       "  [56],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [186],\n",
       "  [253],\n",
       "  [120],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [57],\n",
       "  [253],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [28],\n",
       "  [249],\n",
       "  [240],\n",
       "  [25],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [34],\n",
       "  [253],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [109],\n",
       "  [254],\n",
       "  [197],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [53],\n",
       "  [253],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [135],\n",
       "  [254],\n",
       "  [133],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [133],\n",
       "  [254],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [27],\n",
       "  [240],\n",
       "  [255],\n",
       "  [35],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [7],\n",
       "  [235],\n",
       "  [253],\n",
       "  [208],\n",
       "  [151],\n",
       "  [169],\n",
       "  [215],\n",
       "  [253],\n",
       "  [206],\n",
       "  [2],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [97],\n",
       "  [253],\n",
       "  [253],\n",
       "  [253],\n",
       "  [254],\n",
       "  [253],\n",
       "  [253],\n",
       "  [253],\n",
       "  [86],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [150],\n",
       "  [244],\n",
       "  [145],\n",
       "  [119],\n",
       "  [101],\n",
       "  [82],\n",
       "  [253],\n",
       "  [253],\n",
       "  [14],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [84],\n",
       "  [254],\n",
       "  [172],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [174],\n",
       "  [253],\n",
       "  [119],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [237],\n",
       "  [252],\n",
       "  [56],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [50],\n",
       "  [241],\n",
       "  [182],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [187],\n",
       "  [254],\n",
       "  [249],\n",
       "  [105],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [186],\n",
       "  [253],\n",
       "  [206],\n",
       "  [21],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [227],\n",
       "  [242],\n",
       "  [32],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [236],\n",
       "  [219],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]],\n",
       " [[0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0],\n",
       "  [0]]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set_numpy_array[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set_numpy_array = np.array(test_set_numpy_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "input_data_json = json.dumps({\n",
    "    \"signature_name\": \"serving_default\",\n",
    "    \"instances\": test_set_numpy_array.tolist()\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\'{\"signature_name\": \"serving_default\", \"instances\": [[[[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [84], [254], [101], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [174], [253], [119], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [31], [247], [202], [29], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [1], [1], [0], [0], [0], [0], [0], [141], [253], [168], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [66], [208], [56], [0], [0], [0], [0], [186], [253], [120], [0], [0], [0], [0], [0], [0], [0], [0],...'"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repr(input_data_json)[:1500] + \"...\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use TensorFlow Serving's REST API to make predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict'\n",
    "response = requests.post(SERVER_URL, data=input_data_json)\n",
    "response.raise_for_status() # raise an exception in case of error\n",
    "response = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['predictions'])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 1., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba = np.array(response[\"predictions\"])\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the second image. Based on the predictions above, it should be a 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAOhklEQVR4nO3db4xc9XXG8e/j3cUmthfbMZgqSdcB4ZiSFFKWOFJUSAUNCpIVWkttCKn6oshRq32RVw2tgFqJ0zSqRCoqgurWAYMIAiqTBlBTiQq3Kig0q6SmbLuQEHeDy59iBcyuMeu1ffpiZqXJsHN3vHNn7tjn+Ugjee+5c+/RD5793ZnfzF5FBGaWy7KqGzCz3nPwzRJy8M0ScvDNEnLwzRJy8M0SGqzipGdpeaxgZRWnNktjmjcORcS5C9VKCb6kdcBu4FPAIeBPIuLbrfZfwUq26OoyTm1mLTwRfz/VqlbWjH8ncAzYAFwGPC5pf0RMlHR8MytRx6/xJa0EtgG3RsRMRPwb8F3g9zo9tpl1Rxlv7m0CTkTECw3b9gOXNO4kabukcUnjc8yWcFozW6oygr8KONy07TCwunFDROyKiNGIGB1ieQmnNbOlKiP4M8Bw07ZhYLqEY5tZF5QR/BeAQUkXNWy7FPAbe2Z9quPgR8QRYC/wZUkrJX0C+AxwX6fHNrPuKOuTe38EnA38H/AA8IdeyjPrX6Ws40fEz4HryziWmXWfP6tvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvlpCDb5aQg2+WkINvllApN800W8yy1atb1qbuHil87qqzZwvra7f+tPjkJ08U1xMqZcaXtE/SO5Jm6o/nyziumXVHmZf6YxGxqv74UInHNbOS+TW+WUJlBv9rkg5JekrSJ5uLkrZLGpc0PkfxazYz666ygv8l4ALgfcAu4FFJFzbuEBG7ImI0IkaHWF7Sac1sKUoJfkQ8ExHTETEbEXuAp4Dryji2mZWvW6/xA1CXjm1mHep4HV/SGmAL8C/AceB3gSuBL3Z6bDuNqPj3/MF7P9Cy9p8fu7ejU1975U2F9YF9P+zo+GeiMj7AMwTsBDYDJ4BJ4PqI8Fq+WZ/qOPgR8TpwRQm9mFmPeB3fLCEH3ywhB98sIQffLCF/LddKER//1cL6f3zs7iUf+8XjRwvrZ73yVmHdX8p9N8/4Zgk5+GYJOfhmCTn4Zgk5+GYJOfhmCTn4Zgl5Hd9K8eLvnN21Y//DdPFnBE48/5OunftM5RnfLCEH3ywhB98sIQffLCEH3ywhB98sIQffLCGv41spHvut2xfZY0XLytE4VvjMb991bWH9PJ5e5NzWzDO+WUIOvllCDr5ZQg6+WUIOvllCDr5ZQg6+WUJex7e2DGw4r7C+aaj1Ov1inpldWVg/706v05etrRlf0pikcUmzku5pql0taVLS25KelDTSlU7NrDTtXuq/DOwEvtW4UdJ6YC9wK7AOGAceLLNBMytfW5f6EbEXQNIo8P6G0m8DExHxcL2+AzgkaXNETJbcq5mVpNM39y4B9s//EBFHgBfr23+BpO31lwvjc8x2eFoz60SnwV8FHG7adhhY3bxjROyKiNGIGB1ieYenNbNOdBr8GWC4adswMN3hcc2sizoN/gRw6fwPklYCF9a3m1mfauvNPUmD9X0HgAFJK4DjwCPAX0raBjwO3AY86zf2zjyTX/9A14791wevWWSP17p27qzanfFvAY4CNwOfr//7loh4HdgGfBV4A9gCfLYLfZpZidpdztsB7GhRewLYXF5LZtZt/qy+WUIOvllCDr5ZQg6+WUL+Wq615ftX37HIHku/TfZrf/PBwvqwl/NK5xnfLCEH3ywhB98sIQffLCEH3ywhB98sIQffLCGv41tPTM61/nNr5zz8w8LnRtnNmGd8s4wcfLOEHHyzhBx8s4QcfLOEHHyzhBx8s4S8jm8AnLzqo4X1Vfr3jo6/9btfbFm7aO6Zjo5tp84zvllCDr5ZQg6+WUIOvllCDr5ZQg6+WUIOvllCXsc3AM7Z+VJh/T3Lziqsz5x8p7B+8TdebVk7XvhM64a2ZnxJY5LGJc1Kuqdh+0ZJIWmm4XFr17o1s1K0O+O/DOwErmXhW6asiQj/4jY7TbQV/IjYCyBpFHh/Vzsys64r6829KUkHJd0taf1CO0jaXn+5MD5H67+/Zmbd12nwDwFXACPA5cBq4P6FdoyIXRExGhGjQyzv8LRm1omO3tWPiBlgvP7ja5LGgFckDUfEWx13Z2ZdUfY6/vxfQlbJxzWzErU140sarO87AAxIWkFt+fVy4E3gx8Ba4A5gX0Qc7k67tlQDv7KpsP5XI98qrJ+I9xTW97x1UWH9+IGpwrr1Vrsz/i3AUeBm4PP1f98CXAB8D5gGngNmgRvKb9PMytTuct4OYEeL8gNlNWNmveHP6psl5OCbJeTgmyXk4Jsl5K/lJjF586rC+i8NFC/XLWbX324trJ/P0x0d38rlGd8sIQffLCEH3ywhB98sIQffLCEH3ywhB98sIa/jJzH2a/u6evzzv+F1+tOJZ3yzhBx8s4QcfLOEHHyzhBx8s4QcfLOEHHyzhLyOfwY5+esfbVn7gzXfXOTZxXc3+o3nthXWz+bAIse3fuIZ3ywhB98sIQffLCEH3ywhB98sIQffLCEH3ywhr+OfQS66/b9b1lapeJ1+Mcv/fE1Hz7f+suiML2m5pN2SpiRNS/qRpE831K+WNCnpbUlPShrpbstm1ql2LvUHgZeAq4BzgFuBhyRtlLQe2Fvftg4YBx7sUq9mVpJFL/Uj4giwo2HTY5IOAJcD7wUmIuJhAEk7gEOSNkfEZPntmlkZTvnNPUkbgE3ABHAJsH++Vv8l8WJ9e/PztksalzQ+x+zSOzazjp1S8CUNAfcDe+oz+irgcNNuh4HVzc+NiF0RMRoRo0OLfCHEzLqr7eBLWgbcBxwDxuqbZ4Dhpl2HgelSujOzrmhrOU+SgN3ABuC6iJirlyaA32/YbyVwYX27lWxg7drC+rZ1/7rkY0/MHSusDz7TeqkQ4OSSz2xVaHfGvwu4GNgaEUcbtj8CfFjSNkkrgNuAZ/3Gnll/a2cdfwT4AnAZ8Kqkmfrjxoh4HdgGfBV4A9gCfLabDZtZ59pZzpsCVFB/AthcZlNm1l3+rL5ZQg6+WUIOvllCDr5ZQv5a7mnk2GUfLKx/csU/L/nYf/zT4j+fHe/875KPbf3HM75ZQg6+WUIOvllCDr5ZQg6+WUIOvllCDr5ZQl7HP40c2HpW147983t/ubC+Fq/jn0k845sl5OCbJeTgmyXk4Jsl5OCbJeTgmyXk4Jsl5HX808jm218qrP/g+mhZO9H676UCsP47/1VYP1FYtdONZ3yzhBx8s4QcfLOEHHyzhBx8s4QcfLOEHHyzhBZdx5e0HPgmcA2wDvgJ8KcR8Y+SNgIHgCMNT/l6RHyl/Fbt+MHi78T/2QWXd3D0wx0810437XyAZxB4CbgK+BlwHfCQpI807LMmIo53oT8z64JFL/Uj4khE7IiI/4mIkxHxGLVZvpPpxcwqdMqv8SVtADYBEw2bpyQdlHS3pPUtnrdd0rik8Tlml9iumZXhlIIvaQi4H9gTEZPAIeAKYITaFcDqev1dImJXRIxGxOgQyzvr2sw60vaXdCQtA+4DjgFjABExA4zXd3lN0hjwiqThiHir7GbNrBxtBV+SgN3ABuC6iJhrsev818OKvwpmZpVqd8a/C7gYuCYijs5vlLQFeBP4MbAWuAPYFxFeGzLrY4u+xpc0AnwBuAx4VdJM/XEjcAHwPWAaeA6YBW7oYr9mVoJFZ/yImKL40v2B8toxs17wR3bNEnLwzRJy8M0ScvDNEnLwzRJy8M0ScvDNEnLwzRJy8M0ScvDNEnLwzRJy8M0ScvDNElJE61srd+2k0uvAVMOm9dT+jFc/cm9L495OXdl9jUTEuQsVKgn+u5qQxiNitOo+FuLelsa9nbpe9uVLfbOEHHyzhPol+LuqbqCAe1sa93bqetZXX7zGN7Pe6pcZ38x6yME3S8jBN0uo0uBLWifpEUlHJE1J+lyV/TSStE/SOw33EXi+wl7G6jccnZV0T1PtakmTkt6W9GT9PgiV9yZpo6RoGL8ZSbf2sK/lknbX/7+alvQjSZ9uqFc2bkW99Wrc2r53XpfcSe1efBuo3bDjcUn7I2Ki+Gk9MxYRf1d1E8DLwE7gWuDs+Y31OxPvBW4CHgW+AjwIfLzq3hqsiYjjPexn3iDwEnAV8DPgOuAhSR8BZqh23Ip6m9fdcYuISh7ASmqh39Sw7T7gL6rqqam/fcBNVffR1NNO4J6Gn7cDTzeN6VFgcx/0tpHavRQHqx63hp6eBbb107gt0FtPxq3KS/1NwImIeKFh237gkor6WcjXJB2S9JSkT1bdzAIuoTZmAETEEeBF+msMpyQdlHR3/QqlEpI2UPt/boI+G7em3uZ1ddyqDP4qoPnmmoeB1RX0spAvUbs34PuofbDiUUkXVtvSu/TzGB4CrgBGgMup9XR/FY1IGqqfe09ETNJH47ZAbz0ZtyqDPwMMN20bpnYDzspFxDMRMR0RsxGxB3iK2muxftK3YxgRMxExHhHHI+I1YAz4lKTmfrtK0jJqLyGP1XuAPhm3hXrr1bhVGfwXgEFJFzVsu5RfvNzpJ0HxzUOrMEFtzACQtBK4kP4cw/mPiPZsDCUJ2E3tzeNtETFXL1U+bgW9NevKuFUW/Prrqr3AlyWtlPQJ4DPUfgNWStIaSddKWiFpsH5L8CuBf6qon0FJK4ABYGC+L+AR4MOSttXrtwHP1i8ZK+1N0hZJH5K0TNJ7gTuAfRHRfIndTXcBFwNbI+Jow/bKx61Vbz0bt4rfZV0HfAc4Qm1Z43NV9tPQ17nAD6hd+r0JfB/4zQr72UHtN3/jY0e9dg0wSe1d6X3Axn7oDbgBOFD/b/sKcC9wfg/7Gqn38g61S/v5x41Vj1tRb70aN39Jxywhf2TXLCEH3ywhB98sIQffLCEH3ywhB98sIQffLCEH3yyh/wdigKGlYW/1EgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.imshow(np.array(test_set_numpy_array[1].reshape(28, 28)), interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the second image. It should be a 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQa0lEQVR4nO3dfZBV9X3H8fdn2Q0gsCIidGpSQEbEQaMd1mJ0onbwoTLjmIa2PqWJmTo4Sbet04lNmgLDWGw6Y6edOFETKgV1DKMm0NTY2moq7RSjda2BcaeIGoMa0LCt4i7Cuizf/rF3O9d179nLnnMf4Pd5zewMnO95+PqTz55z7+/ecxQRmFlaWhrdgJnVn4NvliAH3yxBDr5Zghx8swQ5+GYJam3EQT+miTGJKY04tFkyenmnJyJOGa1WSPAlzQDWA5cDPcCfRsR3K60/iSks0dIiDm1mFTwZ39tdqVbUGf8u4ANgNnAu8Jik7RHRXdD+zaxAuV/jS5oCLAdWRURfRPwH8A/A7+bdt5nVRhFv7i0ABiNiV9my7cCi8pUkrZDUJalrgP4CDmtm41VE8KcC+0cs2w9MK18QEesioiMiOtqYWMBhzWy8igh+H9A+Ylk70FvAvs2sBooI/i6gVdLpZcvOAfzGnlmTyh38iDgAbAZukzRF0oXA1cADefdtZrVR1Cf3vgxMBn4BbAK+5Kk8s+ZVyDx+RPwv8Jki9mVmtefP6pslyME3S5CDb5YgB98sQQ6+WYIcfLMEOfhmCXLwzRLk4JslyME3S5CDb5YgB98sQQ25vbYdf7R4UWa9/ZtvVaxtmvdE5rZnPf2FzPrcG3+aWT9y4EBmPUU+45slyME3S5CDb5YgB98sQQ6+WYIcfLMEOfhmCfI8vhVi70UnZtYfnXd/xdqRMfa944KNmfVF67+YWZ9/466KtSOHDo1x9OOTz/hmCXLwzRLk4JslyME3S5CDb5YgB98sQQ6+WYI8j29Vee+68zPrj9xyxxh7mFRcMyN0f3pDZv3sr3dWrM1Z/eOi2zkmFHLGl7RV0iFJfaWfl4rYr5nVRpGX+p0RMbX0c0aB+zWzgvk1vlmCigz+NyT1SNom6ZKRRUkrJHVJ6hqgv8DDmtnRKir4XwVOA04F1gGPSppfvkJErIuIjojoaGNiQYc1s/EoJPgR8WxE9EZEf0TcB2wDlhWxbzMrXq1e4wegGu3bzHLKPY8vaTqwBPg34DBwDXARcEvefVv9TGhvz6xf9CfPZNbntdZunj6v73/+ryvW/nj1p+rYSfMo4gM8bcBaYCEwCOwEPhMRnss3a1K5gx8R+4DzCujFzOrE8/hmCXLwzRLk4JslyME3S5C/lmsAtD56Qmb9L2ZtzayPdYvsLGt7PplZXzlzR469wwkazLX98chnfLMEOfhmCXLwzRLk4JslyME3S5CDb5YgB98sQZ7HT8SeWy/IrD912li3x56c6/ibemdXrD13yazMba/5wW9k1h+a/3hmfUpL5VtDtJxzZua2R7b/d2b9WOUzvlmCHHyzBDn4Zgly8M0S5OCbJcjBN0uQg2+WIM/jH0fe+qPKc/X//od/lbnt1JZ8t8e+/71TM+vf/62LK9YG38m+IfPP3s33DNaTMv7b3rzspMxtf3l7rkM3LZ/xzRLk4JslyME3S5CDb5YgB98sQQ6+WYIcfLMEeR7/GDLh9NMy61+6+QcVa1NbJuY69tZDbZn1jV+/OrN+QvezuY5fK20X/U/2CtkffzhmVXXGl9QpqUtSv6SNI2pLJe2U9L6kpyTNqUmnZlaYai/19wBrgb8rXyhpJrAZWAXMALqAh4ps0MyKV9WlfkRsBpDUAXy8rPRZoDsiHinV1wA9khZGxM6CezWzguR9c28R8P+fZo6IA8CrpeUfImlF6eVC1wD9OQ9rZnnkDf5UYP+IZfuBaSNXjIh1EdERER1t5HujyczyyRv8PqB9xLJ2oDfnfs2shvIGvxs4Z/gvkqYA80vLzaxJVfXmnqTW0roTgAmSJgGHgS3AHZKWA48Bq4EdfmNvfCYsmJ9Z//XNP8ms/96JrxfZzod03n9zZv1Xtjxds2PX0jXz/iuz/q9MqVMn9VXtGX8lcBD4GvC50p9XRsQ+YDlwO/AOsAS4tgZ9mlmBqp3OWwOsqVB7ElhYXEtmVmv+rL5Zghx8swQ5+GYJcvDNEuSv5TaRSfe+l1m/5aRdNTv22p5PZtbn3f1yZn2wyGbqaMPmyzLrczg2pynH4jO+WYIcfLMEOfhmCXLwzRLk4JslyME3S5CDb5Ygz+PXUe+152fW/3H+3WPsQeM+9t7B9zPrz/3OmZn1wX2vjPvYeUmRWW/JMS5tfePe9JjmM75Zghx8swQ5+GYJcvDNEuTgmyXIwTdLkINvliDP4xdowuxZmfXVazdk1o+QPV89ljcPH6xY++3bbs3c9uSXfpzr2Hm0TJqUWT9jxr7M+ljj1h8DFWtTf34kc9vjlc/4Zgly8M0S5OCbJcjBN0uQg2+WIAffLEEOvlmCPI9fIE3Ono9eOjn7O/F5ffYnN1WszVrfuHn6sfz0z341s/7i3G/l2v++wcMVa+3ffSbXvo9VVZ3xJXVK6pLUL2lj2fK5kkJSX9nPqpp1a2aFqPaMvwdYC1wBTB6lPj0iKv9aNbOmUlXwI2IzgKQO4OM17cjMaq6oN/d2S3pT0gZJM0dbQdKK0suFrgH6CzqsmY1H3uD3AOcBc4DFwDTgwdFWjIh1EdERER1tTMx5WDPLI9e7+hHRB3SV/vq2pE5gr6T2iMh+9KuZNUzR8/jD348c//2OzazmqjrjS2otrTsBmCBpEnCYocv7d4GXgZOAO4GtEbG/Nu023sGrf61i7d47/2aMrbPn+cdyhDG+O/7EjFz7b5Qll3bXdP/TWyqf3w4vXZy5beuPni+6naZQ7Rl/JXAQ+BrwudKfVwKnAY8DvcCLQD9wXfFtmlmRqp3OWwOsqVDeVFQzZlYf/qy+WYIcfLMEOfhmCXLwzRLkr+Uepatu/1HF2rzWfNN1Y9nUe2pmfda3nq7p8fM4sHxJxdrdn/jmGFu35Tr2QMbttz/Wk/1V6eP15ts+45slyME3S5CDb5YgB98sQQ6+WYIcfLMEOfhmCfI8/lH6zj9dXrF2y/W7anrs2/7lNzPrp/NsTY+f5e0/uCCzfmvnQxVrE5Vvnn4sn77/KxVrc7c3723Ha8lnfLMEOfhmCXLwzRLk4JslyME3S5CDb5YgB98sQZ7HP0qTf9G4RwbExyp/rxxgz1cqz6W/f+7BzG1bfp59L4GzP/VKZv3JeXdk1k9sGf+9CgZiMLN+4fOfz6zPXfWf4z728cpnfLMEOfhmCXLwzRLk4JslyME3S5CDb5YgB98sQZ7HP4bsuuqehh27TRMy6wMx/nn6bYeyv49/86bOzPrclWl+pz6PMc/4kiZKWi9pt6ReSS9IurKsvlTSTknvS3pK0pzatmxmeVVzqd8KvAFcDJwIrAIeljRX0kxgc2nZDKALqHyrFTNrCmNe6kfEAWBN2aIfSnoNWAycDHRHxCMAktYAPZIWRsTO4ts1syIc9Zt7kmYDC4BuYBGwfbhW+iXxamn5yO1WSOqS1DVA//g7NrPcjir4ktqAB4H7Smf0qcD+EavtB6aN3DYi1kVER0R0tDFxvP2aWQGqDr6kFuAB4ANg+G3WPqB9xKrtQG8h3ZlZTVQ1nSdJwHpgNrAsIgZKpW7gC2XrTQHml5Yflz6xZW/F2l03zs/c9venv1p0O3UzGNkPjH5+jFdv1z/25Yq1hd9+N3Pbud2eritatWf8e4AzgasiovyL3VuAsyQtlzQJWA3s8Bt7Zs2tmnn8OcDNwLnAW5L6Sj83RMQ+YDlwO/AOsAS4tpYNm1l+1Uzn7QYq3nYmIp4EFhbZlJnVlj+rb5YgB98sQQ6+WYIcfLME+Wu5R2nwldcq1p5Yekbmtn/7xWWZ9btu+nZm/cJJA5n1PM7edmNmfdrjUzPrs57ak1k//bXKj/DOvnm21YLP+GYJcvDNEuTgmyXIwTdLkINvliAH3yxBDr5ZghSR/ejlWmjXjFiipXU/rllKnozvPR8RHaPVfMY3S5CDb5YgB98sQQ6+WYIcfLMEOfhmCXLwzRLk4JslyME3S5CDb5YgB98sQQ6+WYIcfLMEOfhmCXLwzRJUzdNyJ0paL2m3pF5JL0i6slSbKynKnqDbJ2lV7ds2szyqeaBGK/AGcDHwOrAMeFjS2WXrTI+IwzXoz8xqYMwzfkQciIg1EfGziDgSET8EXgMW1749M6uFo36NL2k2sADoLlu8W9KbkjZImllhuxWSuiR1DdA/znbNrAhHFXxJbcCDwH0RsRPoAc4D5jB0BTCtVP+IiFgXER0R0dHGxHxdm1kuVT80U1IL8ADwAdAJEBF9QFdplbcldQJ7JbVHxHtFN2tmxagq+JIErAdmA8siotJjW4dv2asCejOzGqn2jH8PcCZwaUQcHF4oaQnwLvAycBJwJ7A1IvYX3aiZFaeaefw5wM3AucBbZfP1NwCnAY8DvcCLQD9wXQ37NbMCjHnGj4jdZF+6byquHTOrB39k1yxBDr5Zghx8swQ5+GYJcvDNEuTgmyXIwTdLkINvliAH3yxBDr5Zghx8swQ5+GYJcvDNEqSIGHutog8q7QN2ly2aydBtvJqRexsf93b0iu5rTkScMlqhIcH/SBNSV0R0NLqP0bi38XFvR6+efflS3yxBDr5Zgpol+Osa3UAG9zY+7u3o1a2vpniNb2b11SxnfDOrIwffLEEOvlmCGhp8STMkbZF0QNJuSdc3sp9ykrZKOlT2HIGXGthLZ+mBo/2SNo6oLZW0U9L7kp4qPQeh4b1JmispysavT9KqOvY1UdL60r+rXkkvSLqyrN6wccvqrV7jVvWz82rkLoaexTeboQd2PCZpe0R0Z29WN50RcW+jmwD2AGuBK4DJwwtLTybeDNwEPAr8OfAQcH6jeyszPSIO17GfYa3AG8DFwOvAMuBhSWcDfTR23LJ6G1bbcYuIhvwAUxgK/YKyZQ8Af9monkb0txW4qdF9jOhpLbCx7O8rgKdHjOlBYGET9DaXoWcptjZ63Mp62gEsb6ZxG6W3uoxbIy/1FwCDEbGrbNl2YFGD+hnNNyT1SNom6ZJGNzOKRQyNGQARcQB4leYaw92S3pS0oXSF0hCSZjP0b66bJhu3Eb0Nq+m4NTL4U4GRD9fcD0xrQC+j+SpDzwY8laEPVjwqaX5jW/qIZh7DHuA8YA6wmKGeHmxEI5LaSse+LyJ20kTjNkpvdRm3Rga/D2gfsaydoQdwNlxEPBsRvRHRHxH3AdsYei3WTJp2DCOiLyK6IuJwRLwNdAKXSxrZb01JamHoJeQHpR6gScZttN7qNW6NDP4uoFXS6WXLzuHDlzvNJMh+eGgjdDM0ZgBImgLMpznHcPgjonUbQ0kC1jP05vHyiBgolRo+bhm9jVSTcWtY8EuvqzYDt0maIulC4GqGfgM2lKTpkq6QNElSa+mR4BcB/9ygflolTQImABOG+wK2AGdJWl6qrwZ2lC4ZG9qbpCWSzpDUIulk4E5ga0SMvMSupXuAM4GrIuJg2fKGj1ul3uo2bg1+l3UG8PfAAYamNa5vZD9lfZ0CPMfQpd+7wDPAZQ3sZw1Dv/nLf9aUapcCOxl6V3orMLcZegOuA14r/b/dC9wP/FId+5pT6uUQQ5f2wz83NHrcsnqr17j5SzpmCfJHds0S5OCbJcjBN0uQg2+WIAffLEEOvlmCHHyzBDn4Zgn6P/TAEhrtuK7fAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.imshow(np.array(test_set_numpy_array[2].reshape(28, 28)), interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the gRPC API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow_serving.apis.predict_pb2 import PredictRequest\n",
    "\n",
    "request = PredictRequest()\n",
    "request.model_spec.name = model_name\n",
    "request.model_spec.signature_name = \"serving_default\"\n",
    "input_name = model.input_names[0]\n",
    "request.inputs[input_name].CopyFrom(tf.make_tensor_proto(test_set_numpy_array))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "ename": "_InactiveRpcError",
     "evalue": "<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INVALID_ARGUMENT\n\tdetails = \"Expects arg[0] to be float but int64 is provided\"\n\tdebug_error_string = \"{\"created\":\"@1616512646.227664978\",\"description\":\"Error received from peer ipv6:[::1]:8500\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1061,\"grpc_message\":\"Expects arg[0] to be float but int64 is provided\",\"grpc_status\":3}\"\n>",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31m_InactiveRpcError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-105-96931aba7d1c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mchannel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsecure_channel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'localhost:8500'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mpredict_service\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprediction_service_pb2_grpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPredictionServiceStub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchannel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict_service\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10.0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/lib/python3.7/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, request, timeout, metadata, credentials, wait_for_ready, compression)\u001b[0m\n\u001b[1;32m    824\u001b[0m         state, call, = self._blocking(request, timeout, metadata, credentials,\n\u001b[1;32m    825\u001b[0m                                       wait_for_ready, compression)\n\u001b[0;32m--> 826\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_end_unary_response_blocking\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    827\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    828\u001b[0m     def with_call(self,\n",
      "\u001b[0;32m~/.local/lib/python3.7/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m_end_unary_response_blocking\u001b[0;34m(state, call, with_call, deadline)\u001b[0m\n\u001b[1;32m    727\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    728\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 729\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0m_InactiveRpcError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    730\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    731\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31m_InactiveRpcError\u001b[0m: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INVALID_ARGUMENT\n\tdetails = \"Expects arg[0] to be float but int64 is provided\"\n\tdebug_error_string = \"{\"created\":\"@1616512646.227664978\",\"description\":\"Error received from peer ipv6:[::1]:8500\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1061,\"grpc_message\":\"Expects arg[0] to be float but int64 is provided\",\"grpc_status\":3}\"\n>"
     ]
    }
   ],
   "source": [
    "import grpc\n",
    "from tensorflow_serving.apis import prediction_service_pb2_grpc\n",
    "\n",
    "channel = grpc.insecure_channel('localhost:8500')\n",
    "predict_service = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n",
    "response = predict_service.Predict(request, timeout=10.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: OK, I'll skip this part with the gRPC, since REST works and gRPC is used when I want maximum performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outputs {\n",
       "  key: \"dense_4\"\n",
       "  value {\n",
       "    dtype: DT_FLOAT\n",
       "    tensor_shape {\n",
       "      dim {\n",
       "        size: 3\n",
       "      }\n",
       "      dim {\n",
       "        size: 10\n",
       "      }\n",
       "    }\n",
       "    float_val: 2.0824443708988838e-05\n",
       "    float_val: 1.4913139168015732e-08\n",
       "    float_val: 0.0004813199338968843\n",
       "    float_val: 0.001888890634290874\n",
       "    float_val: 2.682592992186983e-07\n",
       "    float_val: 8.666840585647151e-06\n",
       "    float_val: 1.6853943241024183e-10\n",
       "    float_val: 0.9975269436836243\n",
       "    float_val: 3.833709342870861e-05\n",
       "    float_val: 3.4738284739432856e-05\n",
       "    float_val: 0.00017358684272039682\n",
       "    float_val: 0.0002858016814570874\n",
       "    float_val: 0.9816810488700867\n",
       "    float_val: 0.0157401692122221\n",
       "    float_val: 1.1949770339914068e-10\n",
       "    float_val: 0.00023017563216853887\n",
       "    float_val: 3.078056761296466e-05\n",
       "    float_val: 5.393230750883049e-09\n",
       "    float_val: 0.0018584482604637742\n",
       "    float_val: 1.8884094288296183e-09\n",
       "    float_val: 3.397366526769474e-05\n",
       "    float_val: 0.9835277795791626\n",
       "    float_val: 0.001533020636998117\n",
       "    float_val: 0.0014515116345137358\n",
       "    float_val: 0.00018795969663187861\n",
       "    float_val: 0.0011680654715746641\n",
       "    float_val: 0.0014667459763586521\n",
       "    float_val: 0.006120447069406509\n",
       "    float_val: 0.004315734840929508\n",
       "    float_val: 0.00019466254161670804\n",
       "  }\n",
       "}\n",
       "model_spec {\n",
       "  name: \"my_mnist_model\"\n",
       "  version {\n",
       "    value: 2\n",
       "  }\n",
       "  signature_name: \"serving_default\"\n",
       "}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the response to a tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.98, 0.02, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.98, 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  ]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_name = model.output_names[0]\n",
    "outputs_proto = response.outputs[output_name]\n",
    "y_proba = tf.make_ndarray(outputs_proto)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or to a NumPy array if your client does not include the TensorFlow library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  , 0.  , 0.  ],\n",
       "       [0.  , 0.  , 0.98, 0.02, 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 0.98, 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  ]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_name = model.output_names[0]\n",
    "outputs_proto = response.outputs[output_name]\n",
    "shape = [dim.size for dim in outputs_proto.tensor_shape.dim]\n",
    "y_proba = np.array(outputs_proto.float_val).reshape(shape)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying a new model version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "1407/1407 [==============================] - 4s 3ms/step - loss: 15.7676 - accuracy: 0.1134 - val_loss: 2.3079 - val_accuracy: 0.1090\n",
      "Epoch 2/3\n",
      "1407/1407 [==============================] - 2s 2ms/step - loss: 2.3012 - accuracy: 0.1135 - val_loss: 2.3080 - val_accuracy: 0.1090\n",
      "Epoch 3/3\n",
      "1407/1407 [==============================] - 2s 2ms/step - loss: 2.3012 - accuracy: 0.1135 - val_loss: 2.3080 - val_accuracy: 0.1090\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28, 1]),\n",
    "    keras.layers.Dense(50, activation=\"relu\"),\n",
    "    keras.layers.Dense(50, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-2),\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set.batch(32), epochs=3, validation_data=valid_set.batch(32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my_mnist_model/0002'"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_version = \"0002\"\n",
    "model_name = \"my_mnist_model\"\n",
    "model_path = os.path.join(model_name, model_version)\n",
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_mnist_model/0002/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_mnist_model/0002/assets\n"
     ]
    }
   ],
   "source": [
    "tf.saved_model.save(model, model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_mnist_model/\n",
      "    0001/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n",
      "    0002/\n",
      "        saved_model.pb\n",
      "        variables/\n",
      "            variables.data-00000-of-00001\n",
      "            variables.index\n",
      "        assets/\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(model_name):\n",
    "    indent = '    ' * root.count(os.sep)\n",
    "    print('{}{}/'.format(indent, os.path.basename(root)))\n",
    "    for filename in files:\n",
    "        print('{}{}'.format(indent + '    ', filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: You may need to wait a minute before the new model is loaded by TensorFlow Serving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict'\n",
    "            \n",
    "response = requests.post(SERVER_URL, data=input_data_json)\n",
    "response.raise_for_status()\n",
    "response = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['predictions'])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ],\n",
       "       [0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ],\n",
       "       [0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ],\n",
       "       ...,\n",
       "       [0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ],\n",
       "       [0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ],\n",
       "       [0.1 , 0.11, 0.1 , ..., 0.1 , 0.1 , 0.1 ]])"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba = np.array(response[\"predictions\"])\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK. Now let me roll back to the first version of the model. To do this, I'll delete the folder pertaining to the second version of the model and then I'll query the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 01_the_machine_learning_landscape.ipynb\r\n",
      " 02_end_to_end_machine_learning_project.ipynb\r\n",
      " 03_classification.ipynb\r\n",
      " 04_training_linear_models.ipynb\r\n",
      " 05_support_vector_machines.ipynb\r\n",
      " 06_decision_trees.ipynb\r\n",
      " 07_ensemble_learning_and_random_forests.ipynb\r\n",
      " 08_dimensionality_reduction.ipynb\r\n",
      " 09_unsupervised_learning.ipynb\r\n",
      " 10_neural_nets_with_keras.ipynb\r\n",
      " 11_training_deep_neural_networks.ipynb\r\n",
      " 12_custom_models_and_training_with_tensorflow.ipynb\r\n",
      " 13_loading_and_preprocessing_data.ipynb\r\n",
      " 14_deep_computer_vision_with_cnns.ipynb\r\n",
      " 15_processing_sequences_using_rnns_and_cnns.ipynb\r\n",
      " 16_nlp_with_rnns_and_attention.ipynb\r\n",
      " 17_autoencoders_and_gans.ipynb\r\n",
      " 18_reinforcement_learning.ipynb\r\n",
      " 19_training_and_deploying_at_scale.ipynb\r\n",
      " apt.txt\r\n",
      " assets\r\n",
      " book_equations.pdf\r\n",
      " changes_in_2nd_edition.md\r\n",
      " checkpoint\r\n",
      " datasets\r\n",
      " docker\r\n",
      " environment-windows.yml\r\n",
      " environment.yml\r\n",
      " extra_autodiff.ipynb\r\n",
      " extra_gradient_descent_comparison.ipynb\r\n",
      "'How to Grid Search Data Preparation Techniques.ipynb'\r\n",
      " images\r\n",
      " index.ipynb\r\n",
      " INSTALL.md\r\n",
      " LICENSE\r\n",
      " logs\r\n",
      " math_differential_calculus.ipynb\r\n",
      " math_linear_algebra.ipynb\r\n",
      " my_cifar10_logs\r\n",
      " my_cifar10_model.h5\r\n",
      " my_compressed.tfrecord\r\n",
      " my_contacts.tfrecord\r\n",
      " my_data.tfrecord\r\n",
      " my_fashion_mnist_model.png\r\n",
      " my_keras_model.h5\r\n",
      " my_keras_weights.ckpt.data-00000-of-00001\r\n",
      " my_keras_weights.ckpt.index\r\n",
      " my_logs\r\n",
      " my_mnist.data\r\n",
      " my_mnist_model\r\n",
      " my_model.pkl\r\n",
      " my_sketchrnn\r\n",
      " my_test_0.tfrecord\r\n",
      " my_test_1.tfrecord\r\n",
      " my_test_2.tfrecord\r\n",
      " my_test_3.tfrecord\r\n",
      " my_test_4.tfrecord\r\n",
      " my_tfhub_cache\r\n",
      " my_train_0.tfrecord\r\n",
      " person.desc\r\n",
      " person_pb2.py\r\n",
      " person.proto\r\n",
      " __pycache__\r\n",
      " README.md\r\n",
      " requirements.txt\r\n",
      " saved_model.pb\r\n",
      " titanic\r\n",
      " tools_matplotlib.ipynb\r\n",
      " tools_numpy.ipynb\r\n",
      " tools_pandas.ipynb\r\n",
      " variables\r\n",
      " work_in_progress\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"my_mnist_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0001  0002\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -r 0002"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0001\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"..\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 01_the_machine_learning_landscape.ipynb\r\n",
      " 02_end_to_end_machine_learning_project.ipynb\r\n",
      " 03_classification.ipynb\r\n",
      " 04_training_linear_models.ipynb\r\n",
      " 05_support_vector_machines.ipynb\r\n",
      " 06_decision_trees.ipynb\r\n",
      " 07_ensemble_learning_and_random_forests.ipynb\r\n",
      " 08_dimensionality_reduction.ipynb\r\n",
      " 09_unsupervised_learning.ipynb\r\n",
      " 10_neural_nets_with_keras.ipynb\r\n",
      " 11_training_deep_neural_networks.ipynb\r\n",
      " 12_custom_models_and_training_with_tensorflow.ipynb\r\n",
      " 13_loading_and_preprocessing_data.ipynb\r\n",
      " 14_deep_computer_vision_with_cnns.ipynb\r\n",
      " 15_processing_sequences_using_rnns_and_cnns.ipynb\r\n",
      " 16_nlp_with_rnns_and_attention.ipynb\r\n",
      " 17_autoencoders_and_gans.ipynb\r\n",
      " 18_reinforcement_learning.ipynb\r\n",
      " 19_training_and_deploying_at_scale.ipynb\r\n",
      " apt.txt\r\n",
      " assets\r\n",
      " book_equations.pdf\r\n",
      " changes_in_2nd_edition.md\r\n",
      " checkpoint\r\n",
      " datasets\r\n",
      " docker\r\n",
      " environment-windows.yml\r\n",
      " environment.yml\r\n",
      " extra_autodiff.ipynb\r\n",
      " extra_gradient_descent_comparison.ipynb\r\n",
      "'How to Grid Search Data Preparation Techniques.ipynb'\r\n",
      " images\r\n",
      " index.ipynb\r\n",
      " INSTALL.md\r\n",
      " LICENSE\r\n",
      " logs\r\n",
      " math_differential_calculus.ipynb\r\n",
      " math_linear_algebra.ipynb\r\n",
      " my_cifar10_logs\r\n",
      " my_cifar10_model.h5\r\n",
      " my_compressed.tfrecord\r\n",
      " my_contacts.tfrecord\r\n",
      " my_data.tfrecord\r\n",
      " my_fashion_mnist_model.png\r\n",
      " my_keras_model.h5\r\n",
      " my_keras_weights.ckpt.data-00000-of-00001\r\n",
      " my_keras_weights.ckpt.index\r\n",
      " my_logs\r\n",
      " my_mnist.data\r\n",
      " my_mnist_model\r\n",
      " my_model.pkl\r\n",
      " my_sketchrnn\r\n",
      " my_test_0.tfrecord\r\n",
      " my_test_1.tfrecord\r\n",
      " my_test_2.tfrecord\r\n",
      " my_test_3.tfrecord\r\n",
      " my_test_4.tfrecord\r\n",
      " my_tfhub_cache\r\n",
      " my_train_0.tfrecord\r\n",
      " person.desc\r\n",
      " person_pb2.py\r\n",
      " person.proto\r\n",
      " __pycache__\r\n",
      " README.md\r\n",
      " requirements.txt\r\n",
      " saved_model.pb\r\n",
      " titanic\r\n",
      " tools_matplotlib.ipynb\r\n",
      " tools_numpy.ipynb\r\n",
      " tools_pandas.ipynb\r\n",
      " variables\r\n",
      " work_in_progress\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict'\n",
    "            \n",
    "response = requests.post(SERVER_URL, data=input_data_json)\n",
    "response.raise_for_status()\n",
    "response = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['predictions'])"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 1., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba = np.array(response[\"predictions\"])\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model rolled back to the previous version. Great!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me now try doing the same thing on Google Cloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Cloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir(\"../../../Downloads\")\n",
    "os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"cnn-mnist-308715-5e6accf96137.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from googleapiclient.discovery import build\n",
    "\n",
    "project_id = \"cnn-mnist-308715\" # change this to your project ID\n",
    "model_id = \"cnn_mnist_model\"\n",
    "model_path = \"projects/{}/models/{}\".format(project_id, model_id)\n",
    "ml_resource = build(\"ml\", \"v1\").projects()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X):\n",
    "    input_data_json = {\"signature_name\": \"serving_default\",\n",
    "    \"instances\": X.tolist()}\n",
    "    request = ml_resource.predict(name=model_path, body=input_data_json)\n",
    "    response = request.execute()\n",
    "    if \"error\" in response:\n",
    "        raise RuntimeError(response[\"error\"])\n",
    "    return np.array([pred[output_name] for pred in response[\"predictions\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "HttpError",
     "evalue": "<HttpError 400 when requesting https://ml.googleapis.com/v1/projects/cnn-mnist-308715/models/cnn_mnist_model:predict?alt=json returned \"Request payload size exceeds the limit: 1572864 bytes.\">",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHttpError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-8e0a4d48e6e1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mY_probas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtest_set_numpy_array\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-17-eefc2e0fe0b7>\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(X)\u001b[0m\n\u001b[1;32m      3\u001b[0m     \"instances\": X.tolist()}\n\u001b[1;32m      4\u001b[0m     \u001b[0mrequest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mml_resource\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmodel_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_data_json\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m\"error\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"error\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/googleapiclient/_helpers.py\u001b[0m in \u001b[0;36mpositional_wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    132\u001b[0m                 \u001b[0;32melif\u001b[0m \u001b[0mpositional_parameters_enforcement\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mPOSITIONAL_WARNING\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m                     \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mpositional_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/googleapiclient/http.py\u001b[0m in \u001b[0;36mexecute\u001b[0;34m(self, http, num_retries)\u001b[0m\n\u001b[1;32m    905\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    906\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 907\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mHttpError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muri\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    908\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpostproc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    909\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mHttpError\u001b[0m: <HttpError 400 when requesting https://ml.googleapis.com/v1/projects/cnn-mnist-308715/models/cnn_mnist_model:predict?alt=json returned \"Request payload size exceeds the limit: 1572864 bytes.\">"
     ]
    }
   ],
   "source": [
    "Y_probas = predict(test_set_numpy_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 84]\n",
      "  [254]\n",
      "  [101]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [174]\n",
      "  [253]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 31]\n",
      "  [247]\n",
      "  [202]\n",
      "  [ 29]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  1]\n",
      "  [  1]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [141]\n",
      "  [253]\n",
      "  [168]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 66]\n",
      "  [208]\n",
      "  [ 56]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [186]\n",
      "  [253]\n",
      "  [120]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 57]\n",
      "  [253]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 28]\n",
      "  [249]\n",
      "  [240]\n",
      "  [ 25]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 34]\n",
      "  [253]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [109]\n",
      "  [254]\n",
      "  [197]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 53]\n",
      "  [253]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [135]\n",
      "  [254]\n",
      "  [133]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [133]\n",
      "  [254]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 27]\n",
      "  [240]\n",
      "  [255]\n",
      "  [ 35]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  7]\n",
      "  [235]\n",
      "  [253]\n",
      "  [208]\n",
      "  [151]\n",
      "  [169]\n",
      "  [215]\n",
      "  [253]\n",
      "  [206]\n",
      "  [  2]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 97]\n",
      "  [253]\n",
      "  [253]\n",
      "  [253]\n",
      "  [254]\n",
      "  [253]\n",
      "  [253]\n",
      "  [253]\n",
      "  [ 86]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [150]\n",
      "  [244]\n",
      "  [145]\n",
      "  [119]\n",
      "  [101]\n",
      "  [ 82]\n",
      "  [253]\n",
      "  [253]\n",
      "  [ 14]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 84]\n",
      "  [254]\n",
      "  [172]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [174]\n",
      "  [253]\n",
      "  [119]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [237]\n",
      "  [252]\n",
      "  [ 56]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [ 50]\n",
      "  [241]\n",
      "  [182]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [187]\n",
      "  [254]\n",
      "  [249]\n",
      "  [105]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [186]\n",
      "  [253]\n",
      "  [206]\n",
      "  [ 21]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [227]\n",
      "  [242]\n",
      "  [ 32]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [236]\n",
      "  [219]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]\n",
      "\n",
      " [[  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]\n",
      "  [  0]]]\n"
     ]
    }
   ],
   "source": [
    "print(test_set_numpy_array[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "ename": "HttpError",
     "evalue": "<HttpError 404 when requesting https://ml.googleapis.com/v1/projects/cnn-mnist-308715/models/cnn_mnist_model:predict?alt=json returned \"Field: name Error: The model resource: \"cnn_mnist_model\" was not found. Please create the model resource first by using 'gcloud ai-platform models create cnn_mnist_model'.\". Details: \"[{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'name', 'description': 'The model resource: \"cnn_mnist_model\" was not found. Please create the model resource first by using \\'gcloud ai-platform models create cnn_mnist_model\\'.'}]}]\">",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHttpError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-34-f23c04069c3c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mY_probas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtest_set_numpy_array\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-33-eefc2e0fe0b7>\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(X)\u001b[0m\n\u001b[1;32m      3\u001b[0m     \"instances\": X.tolist()}\n\u001b[1;32m      4\u001b[0m     \u001b[0mrequest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mml_resource\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmodel_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_data_json\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m\"error\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"error\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/googleapiclient/_helpers.py\u001b[0m in \u001b[0;36mpositional_wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    132\u001b[0m                 \u001b[0;32melif\u001b[0m \u001b[0mpositional_parameters_enforcement\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mPOSITIONAL_WARNING\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m                     \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mpositional_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/googleapiclient/http.py\u001b[0m in \u001b[0;36mexecute\u001b[0;34m(self, http, num_retries)\u001b[0m\n\u001b[1;32m    905\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    906\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 907\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mHttpError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muri\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    908\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpostproc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    909\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mHttpError\u001b[0m: <HttpError 404 when requesting https://ml.googleapis.com/v1/projects/cnn-mnist-308715/models/cnn_mnist_model:predict?alt=json returned \"Field: name Error: The model resource: \"cnn_mnist_model\" was not found. Please create the model resource first by using 'gcloud ai-platform models create cnn_mnist_model'.\". Details: \"[{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'name', 'description': 'The model resource: \"cnn_mnist_model\" was not found. Please create the model resource first by using \\'gcloud ai-platform models create cnn_mnist_model\\'.'}]}]\">"
     ]
    }
   ],
   "source": [
    "Y_probas = predict(test_set_numpy_array[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I followed this tutorial (https://cloud.google.com/ai-platform/docs/getting-started-keras) and was able to successfully deploy my model and make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Train any model across multiple GPUs on the same machine using the Mirrored\n",
    "Strategy (if you do not have access to GPUs, you can use Colaboratory with a\n",
    "GPU Runtime and create two virtual GPUs). Train the model again using the\n",
    "CentralStorageStrategy and compare the training time._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I ran this on Google Colab. The Mirrored Strategy ran for 175 seconds, while the Central Storage Strategy ran for 258 seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Train a small model on Google Cloud AI Platform, using black box hyperparameter tuning._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have already done this for the AI Safety Camp, but I can do this again for a small project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I followed this tutorial: https://cloud.google.com/ai-platform/docs/getting-started-keras"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}