06-Lab-Tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ray Tune Lab/Demo\n",
    "\n",
    "In this walkthrough, we'll see a minimal realistic example of tuning, from a Ray Tune example.\n",
    "\n",
    "We'll use\n",
    "* Tensorflow\n",
    "* MNIST data\n",
    "* Perceptron classifier architecture with SGD optimizer\n",
    "\n",
    "And we'll tune\n",
    "* Number of neurons in the hidden layer\n",
    "* SGD learning rate\n",
    "* SGD momentum\n",
    "\n",
    "First, we'll do some imports and set up the training call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from ray.tune.integration.keras import TuneReportCallback\n",
    "\n",
    "def train_mnist(config):\n",
    "    # https://github.com/tensorflow/tensorflow/issues/32159\n",
    "    import tensorflow as tf\n",
    "    batch_size = 128\n",
    "    num_classes = 10\n",
    "    epochs = 12\n",
    "\n",
    "    (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "    x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "    model = tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "        tf.keras.layers.Dense(config[\"hidden\"], activation=\"relu\"),\n",
    "        tf.keras.layers.Dropout(0.2),\n",
    "        tf.keras.layers.Dense(num_classes, activation=\"softmax\")\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        optimizer=tf.keras.optimizers.SGD(\n",
    "            lr=config[\"lr\"], momentum=config[\"momentum\"]),\n",
    "        metrics=[\"accuracy\"])\n",
    "\n",
    "    model.fit(\n",
    "        x_train,\n",
    "        y_train,\n",
    "        batch_size=batch_size,\n",
    "        epochs=epochs,\n",
    "        verbose=0,\n",
    "        validation_data=(x_test, y_test),\n",
    "        callbacks=[TuneReportCallback({\n",
    "            \"mean_accuracy\": \"accuracy\"\n",
    "        })])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the `AsyncHyperBandScheduler` for managing our trials. Ray recommends this variant (described in https://arxiv.org/abs/1810.05934) over the \"base\" Hyperband implementation.\n",
    "\n",
    "For an overview of various strategies -- including Hyperband -- this is a great introduction: https://medium.com/criteo-labs/hyper-parameter-optimization-algorithms-2fe447525903"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ray\n",
    "from ray import tune\n",
    "from ray.tune.schedulers import AsyncHyperBandScheduler\n",
    "mnist.load_data()\n",
    "\n",
    "sched = AsyncHyperBandScheduler(\n",
    "    time_attr=\"training_iteration\",\n",
    "    metric=\"mean_accuracy\",\n",
    "    mode=\"max\",\n",
    "    max_t=400,\n",
    "    grace_period=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we start Ray and configure our work in a call to `tune.run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ray.shutdown()\n",
    "ray.init()\n",
    "\n",
    "tune.run(\n",
    "    train_mnist,\n",
    "    name=\"exp\",\n",
    "    scheduler=sched,\n",
    "    stop={\n",
    "        \"mean_accuracy\": 0.99,\n",
    "        \"training_iteration\": 5\n",
    "    },\n",
    "    num_samples=10,\n",
    "    resources_per_trial={\n",
    "        \"cpu\": 1,\n",
    "        \"gpu\": 0\n",
    "    },\n",
    "    config={\n",
    "        \"threads\": 2,\n",
    "        \"lr\": tune.sample_from(lambda spec: np.random.uniform(0.001, 0.1)),\n",
    "        \"momentum\": tune.sample_from(\n",
    "            lambda spec: np.random.uniform(0.1, 0.9)),\n",
    "        \"hidden\": tune.sample_from(\n",
    "            lambda spec: np.random.randint(32, 512)),\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see -- and as you've probably experienced if you've tried to hand-tune a network -- even in a simple problem like this, the resulting accuracy comes from subtle interplay between the influences of the hyperparams.\n",
    "\n",
    "Simply put, the network size, learning rate, and momentum all have to live in a sweet spot to get optimal results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ray.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}