doc/source/notebooks/models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handling models in GPflow\n",
    "--\n",
    "\n",
    "*James Hensman November 2015, January 2016*,\n",
    "*Artem Artemev December 2017*\n",
    "\n",
    "One of the key ingredients in GPflow is the model class, which allows the user to carefully control parameters. This notebook shows how some of these parameter control features work, and how to build your own model with GPflow. First we'll look at\n",
    "\n",
    " - How to view models and parameters\n",
    " - How to set parameter values\n",
    " - How to constrain parameters (e.g. variance > 0)\n",
    " - How to fix model parameters\n",
    " - How to apply priors to parameters\n",
    " - How to optimize models\n",
    "\n",
    "Then we'll show how to build a simple logistic regression model, demonstrating the ease of the parameter framework. \n",
    "\n",
    "GPy users should feel right at home, but there are some small differences.\n",
    "\n",
    "First, let's deal with the usual notebook boilerplate and make a simple GP regression model. See the Regression notebook for specifics of the model: we just want some parameters to play with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gpflow\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a very simple GPR model without building it in TensorFlow graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "X = np.random.rand(20, 1)\n",
    "Y = np.sin(12 * X) + 0.66 * np.cos(25 * X) + np.random.randn(20,1) * 0.01\n",
    "\n",
    "with gpflow.defer_build():\n",
    "    m = gpflow.models.GPR(X, Y, kern=gpflow.kernels.Matern32(1) + gpflow.kernels.Linear(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing, getting and setting parameters\n",
    "You can display the state of the model in a terminal with `print(m)`, and by simply returning it in a notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/lengthscales</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/1/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.models.gpr.GPR at 0x7f29344dc9e8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model has four parameters. The kernel is made of the sum of two parts: the first (counting from zero) is an RBF kernel that has a variance parameter and a lengthscale parameter; the second is a linear kernel that only has a variance parameter. There is also a parameter controlling the variance of the noise, as part of the likelihood. \n",
    "\n",
    "All of the model variables have been initialized at one. Individual parameters can be accessed in the same way as they are displayed in the table: to see all the parameters that are part of the likelihood, do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.likelihoods.Gaussian at 0x7f29344dca58>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.likelihood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gets more useful with more complex models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To set the value of a parameter, just assign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/lengthscales</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/1/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.models.gpr.GPR at 0x7f29344dc9e8>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.kern.kernels[0].lengthscales = 0.5\n",
    "m.likelihood.variance = 0.01\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constraints and trainable variables\n",
    "\n",
    "GPflow helpfully creates an unconstrained representation of all the variables. Above, all the variables are constrained positive (see right hand table column), the unconstrained representation is given by $\\alpha = \\log(\\exp(\\theta)-1)$. `read_trainables()` returns the constrained values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'GPR/kern/kernels/0/lengthscales': array(0.5),\n",
       " 'GPR/kern/kernels/0/variance': array(1.),\n",
       " 'GPR/kern/kernels/1/variance': array(1.),\n",
       " 'GPR/likelihood/variance': array(0.01)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.read_trainables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each parameter has an `unconstrained_tensor` attribute that allows accessing the unconstrained value as a tensorflow Tensor (though only after the model has been compiled). We can also check the unconstrained value as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.4327546710632299"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = m.kern.kernels[0].lengthscales\n",
    "p.transform.backward(p.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constraints are handled by the `Transform` classes. You might prefer the constraint $\\alpha = \\log(\\theta)$: this is easily done by changing the transform attribute on a parameter, with one simple condition - the model has not been compiled yet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.kern.kernels[0].lengthscales.transform = gpflow.transforms.Exp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though the lengthscale itself remains the same, the unconstrained lengthscale has changed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.6931491805619453"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.transform.backward(p.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another helpful feature is the ability to fix parameters. This is done by simply setting the `trainable` attribute to False: this is shown in the 'trainable' column of the representation, and the corresponding variable is removed from the free state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/lengthscales</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>Exp</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/1/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>False</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.models.gpr.GPR at 0x7f29344dc9e8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.kern.kernels[1].variance.trainable = False\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'GPR/kern/kernels/0/lengthscales': array(0.5),\n",
       " 'GPR/kern/kernels/0/variance': array(1.),\n",
       " 'GPR/likelihood/variance': array(0.01)}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.read_trainables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To unfix a parameter, just flip the boolean back and set the parameter to be trainable again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/lengthscales</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>Exp</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/1/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.models.gpr.GPR at 0x7f29344dc9e8>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.kern.kernels[1].variance.trainable = True\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Priors\n",
    "\n",
    "Priors are set just like transforms and trainability, using members of the `gpflow.priors` module. Let's set a Gamma prior on the RBF-variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/lengthscales</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>Exp</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/0/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>Ga(2.0,3.0)</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/kern/kernels/1/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPR/likelihood/variance</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>+ve</td>\n",
       "      <td>True</td>\n",
       "      <td>()</td>\n",
       "      <td>True</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<gpflow.models.gpr.GPR at 0x7f29344dc9e8>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.kern.kernels[0].variance.prior = gpflow.priors.Gamma(2, 3)\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization\n",
    "\n",
    "Optimization is done by creating an instance of optimizer, in our case it is `gpflow.train.ScipyOptimizer`, which has optional arguments that are passed through to `scipy.optimize.minimize` (we minimize the negative log-likelihood) and calling `minimize` method of that optimizer with model as optimization target. Variables that have priors are MAP-estimated, i.e. we add the log prior to the log likelihood, otherwise using Maximum Likelihood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:gpflow.logdensities:Shape of x must be 2D at computation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
      "  Objective function value: 1.884455\n",
      "  Number of iterations: 41\n",
      "  Number of functions evaluations: 46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
      "  Objective function value: 1.884455\n",
      "  Number of iterations: 41\n",
      "  Number of functions evaluations: 46\n"
     ]
    }
   ],
   "source": [
    "m.compile()\n",
    "opt = gpflow.train.ScipyOptimizer()\n",
    "opt.minimize(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building new models\n",
    "\n",
    "To build new models, you'll need to inherit from `gpflow.models.Model`. Parameters are instantiated with `gpflow.Param`. You may also be interested in `gpflow.params.Parameterized` which acts as a 'container' of `Param`s (e.g. kernels are Parameterized). \n",
    "\n",
    "In this very simple demo, we'll implement linear multiclass classification. There will be two parameters: a weight matrix and a 'bias' (offset). The key thing to implement is the private `_build_likelihood` method, which should return a tensorflow scalar representing the (log) likelihood. By decorating the function with `@gpflow.params_as_tensors`, Param objects can be used inside `_build_likelihood`: they will appear as appropriate (constrained) tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "class LinearMulticlass(gpflow.models.Model):\n",
    "    def __init__(self, X, Y, name=None):\n",
    "        super().__init__(name=name) # always call the parent constructor\n",
    "        \n",
    "        self.X = X.copy() # X is a numpy array of inputs\n",
    "        self.Y = Y.copy() # Y is a 1-of-k (one-hot) representation of the labels\n",
    "        \n",
    "        self.num_data, self.input_dim = X.shape\n",
    "        _, self.num_classes = Y.shape\n",
    "        \n",
    "        #make some parameters\n",
    "        self.W = gpflow.Param(np.random.randn(self.input_dim, self.num_classes))\n",
    "        self.b = gpflow.Param(np.random.randn(self.num_classes))\n",
    "       \n",
    "        # ^^ You must make the parameters attributes of the class for\n",
    "        # them to be picked up by the model. i.e. this won't work:\n",
    "        #\n",
    "        # W = gpflow.Param(...    <-- must be self.W\n",
    "    \n",
    "    @gpflow.params_as_tensors\n",
    "    def _build_likelihood(self): # takes no arguments\n",
    "        p = tf.nn.softmax(tf.matmul(self.X, self.W) + self.b) # Param variables are used as tensorflow arrays. \n",
    "        return tf.reduce_sum(tf.log(p) * self.Y) # be sure to return a scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and that's it. Let's build a really simple demo to show that it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAFpCAYAAAB0/VUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//H3mZnsLCEMiGHfXAFBFFFQccFdEVuPXaytXWhrvbXXpb9ae9W217Ze2169tXvrra1tb7+1UuuKYBVXEFEsKCg7QlgMkITsmcz390cAWbKcSWbmzMx5PR8PHo/OnM9M3v0Qkw9nvud7HGutAAAAgCAL+R0AAAAA8BtDMQAAAAKPoRgAAACBx1AMAACAwGMoBgAAQOAxFAMAACDwGIoBAAAQeAzFAAAACDyGYgAAAAQeQzEAAAACL+LT1+Xe0gAAAEgXp6sCv4ZiVVRUJPyaaDSqysrKFKTJLfTJO3rlDX3yjl55Q5+8o1fe0Cdvgtin8vJyT3UsnwAAAEDgMRQDAAAg8BiKAQAAEHgMxQAAAAg8hmIAAAAEHkMxAAAAAo+hGAAAAIHn2z7FAAAAyGxVO2q08E+v6pVHXlfV9moVFOfr+NOP1jmfnq4xk0f6HS+pGIoBAABwmOULV+kncx5QU13T/uca65r0yiOv65VHXtc5nz5dV3/3CoVCubHwIDf+XwAAACBpNix/X/d99tcHDcSHevbBF/XIPU+mMVVqMRQDAADgIHN//LRammJd1j31i3+qpnJPGhKlHkMxAAAA9ttVUaW3FrztqTbW0qoXzWspTpQeDMUAAADY7/2VW2St9Vy/6e3NKUyTPgzFAAAA2M/7ONy9+kzFUAwAAID9Bo8dlNL6TMVQDAAAgP0GDOuv48842lNtKBzS6VedkuJE6ZG0fYpd1w1Lel3SFmPMJcl6XwAAAKTX5V87XytfWa14LN5p3VlXT1PZkaVpSpVayTxTfIOklUl8PwAAAPjgqCmj9eX7r1EkL9xhzdRZJ+oTd85OY6rUSsqZYtd1h0i6WNJdkm5MxnsCAADAP1MumaRhxw3Ws79/Sa8+8rr27KpTOC+s46cfpXM+c7pOOPs4OY7jd8ykSdbyiXslfV1S7yS9HwAAAHw2aNRAffLOK/TJO69QrDmmcF44pwbhA/V4KHZd9xJJO4wxS13XndFJ3RxJcyTJGKNoNJrw14pEIt16XdDQJ+/olTf0yTt65Q198o5eeUOfvKFPHXMS2Zy5Pa7rfl/SpyTFJBVK6iPpEWPM1Z28zFZUVCT8taLRqCorK7uVM0jok3f0yhv65B298oY+eUevvKFP3gSxT+Xl5ZLU5entHp8pNsbcKulWSdp7pvjmLgZiAAAAIKOwTzEAAAACL2n7FEuSMeZ5Sc8n8z0BAACAVONMMQAAAAKPoRgAAACBl9TlEwAAZKKNKzbr2Qdf0pvzl6thT6N69SvRlEsn6exPTdOgUQP9jgcgAzAUAwBylrVWj947T3N/9NRBz+/eVq15v35e8x94QdfefZXO+NhUnxICyBQsnwAA5KwFv3vxsIH4QPHWuB645f/05vwVaUwFIBMxFAMAclJzY4v+/uOnu6yz1urhux9XT29mBSC7MRQDAHLS0qfeUu3uOk+1m1dt1ZqlG1IbCEBGYygGAOSk91dWJFi/JUVJAGQDLrQDcsT2+jo9um6t5m/aoMrGBhWGI5pyxCDNHj1WEwdwdT2CJ9HVEKyeAIKNoRjIAS9VbNEdi19WU2vr/uda4s16dvMmPbt5k2aNGqMbJ01W2OHDIQTHkKOPTGk9gNzCb0ggy729s1L/seilgwbiQz26bo1+teJfaUwF+O+ki09QSd9iT7XlRw3SUVNGpTgRgEzGUAxkuf9duUIt8XiXdWb1u9rd2JiGREBmKCjK12U3nOep9iM3XyTHcVKcCEAmYygGstjWulot3rbVU21LPK4nN65LcSIgs5z/hRm6+CvndnjcCTm65q6P6qSLTkhjKgCZiDXFQBZbV12tRK4NWltdlbIsQCZyHEfurZfqxPPG69kHX9QbzyxXY22TSkqLdcplk3TONdM15Jhyv2MCyAAMxUAWS/Riea6uR1CNmTxCYyaPkCTF43GFQnxQCuBg/FQAstiovn0TrC9NURIgezAQA2gPPxmALFZe0ktTjhjkqTbiOLpoxMgUJwIAIDsxFANZ7jPHjlPYw1XzHxlzlPoXFqUhEQAA2YehGMhyE6ID9J2p05TXyUfCFw0fqesmTExjKgAAsgsX2uW4xlhMG/fUqNVaHVlcon6FhX5HQgqcOXio/nj+xfr72tWat2mjdjY2qDAc1slHHKkrRo/VSQOPYA9WAAA6wVCcoyob6vXQuyv11Ib1qou1SGr7WGBa+RB98uhjNa5/1N+ASLrykl66bsIkXTdhkqy1DMEAACSAoTgHrdu9S597dp52HnL3srikFys26+WtW/Stk6fqvGEjfMmH1GMgBgAgMawpzjHNra364lOPHTYQHyhure5askjv7d6VxmQAAACZi6E4xyzc8r7er6npsq7VWpk176YhEQAAQOZjKM4xT21c77n22fc3qTEWS2EaAACA7MBQnGO219d7rm2Jx1XV1JTCNAAAANmBC+1yTGd71bZbH+bfRQCSa9fWKi169A3t3lalgqJ8HT/9aB1z2hguAAWQ0RiKc8zEAQO1prrKU+3QXr1VVsC+xQCSo6G2UQ/earT4H28q3hrf//xjP5mv8rFH6Nq7r9JRU0b7mBAAOtbjodh13UJJL0gq2Pt+Dxtj7ujp+6J7Zo8aq4fXvOetdjRnbgAkR1NDs+75xM+19o0N7R6vWL1dd3/sZ7r5oS/p2NPGpjccAHiQjM/OmySdbYw5QdJESRe4rjs1Ce+Lbhjep4+uPWFSl3XH9CvTZSPHpCERgCB44v4FHQ7E+8SaY/rlV/+gWEtrekIBQAJ6fKbYGGMl1e59mLf3j+3p+6L7bpk6TS2NjfrTuysVb+f45IFH6LtTp6kwwuoZAD0Xa47p+T+94ql297ZqvfnMcl346SNSnAoAEpOUqch13bCkpZLGSPqpMWZxMt4X3RNyHH1p/ETNHj1Wj61fq7d37lTMxjWkV29dMmKUjivrz7IJAEmz7q1Nqv5gj+f6N+ev0IWfPjeFiQAgcY61yTup67puqaS5kv7NGLPikGNzJM2RJGPM5Obm5oTfPxKJKMa+ul2iT97RK2/ok3dB7NXr85bpjln3eK6feulkfXvu1wPXp+4K4vdUd9Anb4LYp/z8fEnq8mxgUj8/N8ZUua77nKQLJK045NivJP1q70NbWVmZ8PtHo1F153VBQ5+8o1fe0CfvgtireDixNcIFvfIVi8UC16fuCuL3VHfQJ2+C2Kfy8nJPdT2+0M513QF7zxDLdd0iSTMlrerp+wIAssOICUM1YHh/z/WnXNb1xcAAkG7J2H3iSEnPua77L0lLJM03xjyehPcFAGSBUCikmdee4am2/KhBOnYaW7IByDzJ2H3iX5L4Zz8ABNjMa8/Q6iXrteSJZR3W9Cor0fW/uFahBO+8CQDpwE8mBFarjas13t6mdQASFQqH9OWfXqPZN12o3mUlBx1zQo4mnTdOtz/67xp81CCfEgJA59ioFoFS39KiJzeu16Pr1mh9TbUkaUSfPpo1cowuHjFKxXl5PicEslc4Etbl/36BLr7uXC1fuFK7t1WroChfx5w6RtEhZX7HA4BOMRQjMCrqanXji89pc23tQc9vqKnRfW+9oYfXvKcfn36WBvfq5VNCIDfkFUR04nnj/Y4BAAlh+QQCoSEW040vPn/YQHygLXW1uuml51Qfa0ljMgAAkAkYihEI8zZt0Obaru+4tbm2VvM2bkh9IAAAkFEYihEI/1i3xnPtownUAgCA3MBQjEDYsPeiOm+1NSlMAgAAMhFDMQKiy1uef1jpvRQAAOQIhmIEwtjSUs+1Y/r2S2ESAACQiRiKEQiXjxrjvXa091oAAJAbGIoRCGcPHa4xfbs+Wzy6b6nOHTo8DYkAAEAmYShGIBSEw/rR6TN0VGnHSyPGlpbqR9NnqCAcTmMyAACQCbijHQKjf2GRfnn2TD2/+X09um6N1lRXKeSENLJPH80aNUYzBg9VPgMxAGSN6g9q9MJfFuvdV9dI1lHfI3pr+pVTdMypY+Rw1TQSxFCMQMkLhTVz2AjNHDZCkhSNRlVZWelvKABAQqy1euoX/9TD//WEWltaDzr20l9f06iJw/Vvv/6syo70fpE1wPIJAACQVZ76xT/1l7v+cdhAvM+6ZRt191U/Ve3uujQnQzZjKAYAAFmj+oMaPfxfT3RZt23dDj31y3+mIRFyBcsnAuS93bv0yNrVWrJjmxpjMfUvLNLMYSN0ychR6ldQ6Hc8AAC6tPDPizo8Q9xe7ewbL1Qkn3EHXeO7JACstfrZ8mX683urDnq+urlZv1zxlh5a9Y7+89TpOvmIQT4lBADAm1WL1niu3bOzVhVrtmvYcYNTmAi5guUTAfC7lW8fNhAfqC7Wom+88oLe270rjakAAEhcrCmWUH1LgvUILobiHFfd1KQ/rHqny7qm1lY98M6KNCQCAKD7+g/ueL/5duvL2YEC3jAU57inN65Xc9zb2qtXtm7R9nqu1AUAZK7p7hTPteNnHKvSI/qmMA1yCUNxjnuvarfn2rikNVVVqQsDAEAPHTftKI2YMNRT7YVfOivFaZBLGIpzXNzahOqtEqsHACCdHMfRDb/5vI4YEe207urvfETHTz86TamQCxiKc9zwPn0Sq+/Nx0wAgMxWVl6q2x+7URd9+Rz1Lis56Nj4M4/R1/98nWZ+9gyf0iFbsSVbjrt4xCg98M4KT2eMTxwwUEN7905DKgAAeqZXvxJdddtluuLmi1SxZptKCksUKna4tTO6jTPFOW5AUbFmjRrTZV1I0qePPT71gQAASKK8goiGHz9Ex5wyloEYPcJQHAA3nHCizhrS8UUJYcfRN0+eqskDuXkHAAAIJpZPBEAkFNK3T5mmc4du1iNrV+uNHdtlJRVHIjpv2AhdMfoojerLWmIAABBcDMUBEXIcnTl4qM4cPFSxeFzNra0qikTkOI7f0QAAAHzX46HYdd2hkn4v6QhJVtKvjDH39fR9kTqRUEiRECtnAAAA9knGmeKYpJuMMW+4rttb0lLXdecbY7q+tzAAAElgrdXaNzbouT++oi3vbpXkaNjxg3XW1dM00uONHgAEW4+HYmPMVklb9/7vPa7rrpQ0WBJDMQAg5RpqG/Xz6x7UW/88+NfO+rc2aeGfXtXJF0/UF+79pAqK8n1KCCAbJHVNseu6IyRNkrQ4me8LAEB7WmOtuu+zv9HKV1Z3WLPkiWVqaY7pht9+TiGWjgHogGMTvA1wR1zX7SVpoaS7jDGPtHN8jqQ5kmSMmdzc3Jzw14hEIorFYj2NmvPok3f0SmpubdWC9Wv1t1UrtbmmWnnhsCYNOlIfP368josOkESfEkGvvElWnxaaV/Vf19zvqfaOuTdryoWTevw1043vKW/okzdB7FN+fr4kdbmzQFKGYtd18yQ9LmmeMebHHl5iKyoqEv460WhUlZWVCb8uaOiTd0Hv1ZbaPbrl5YXatGdPu8cvHTlaN006SYMGDgx0nxIR9O8pr5LVp+995H/07uK1nmpPOOc43fjgF3v8NdON7ylv6JM3QexTeXm55GEo7vHnSK7rOpJ+K2mlx4EYQAbY3dSoG154rsOBWJIeW79W/71saRpTAd5Za7Vm6XrP9Wte35C6MACyXjLWFE+T9ClJy13XXbb3uW8aY55MwnsDSBGz+l1tq6/rsu7RdWs0Z/cucXsXZBobt2qNxT3Xx1qC9ZExgMQkY/eJl+ThlDSAzBGLx/X4em8fOUvSn99eri8dc3wKEwGJC4VDig4pU+XmXZ7qBwztn+JEALIZl+ECAbStvk67m5o81y/fsT2FaYDuO/2qUzzXnvHxqSlMAiDbMRQDAdQaT+wC25a494+ogXSa8cnTVNy3qMu6PgN6a/qVU9KQCEC2YigGAmhAcZHyEtivdUTf0hSmAbqvdGAf3fjgF1XUu7DDml5lJbrp919USd/iNCYDkG0YioEAKo7k6dyhwz3XX3ks64mRucaeNFLfefoWnfPp6SooKdj/fFGfQp33+TP1nadv0Yjx3OoZQOeSekc7ANnj40cdowXvb+xyacS4sqimDh6inTt3pilZ97y/skLb1u1QOBLW8PFD1L+8n9+RkEYDh0d1zV1X6qpvzdKuit1yHEdl5f2UX5jXrfer3V2nhj2NKu5bxBlmICAYioGAGtW3VN+dOl23L3pJzR0MxmP6lup7p02X42TuBjPLFqzQo/c+o3XLNu5/zgk5mnju8bri5os07LjBPqZDuhUU5evI0Ud067XWWr32+DIt+N2Leu+AG4KMO/MYzfzsGZp4Dp+YALmMoRgIsOnlg/W/516ov655V09v3KDG1rZ9XIf06q3LR43RrFFjVBTJ3B8TC373ov7wrYcPe97Grd58ZoXeeek93fj7L+qYqWN8SIdsEm+N69c3/kmv/G3JYcdWLFylFQtX6fwvzNDHb788o/+RCKD7Mve3HYC0GN6nj24+8WTdMPFEVTU1KeKEVFpQkPG/+Ne+uVEP/cffOq1pqm/W/3z+t/rhK7eruE/XOxQguOb++Kl2B+IDzfv18xowrL9mXntGmlIBSCcutAMgScoLhTWgqFj9CgszfiCWpPm/XShru95arq6qXi93Mewg2BpqGzXvNws91T7x0wVqjbWmOBEAPzAUA8g6LU0xLXlyWdeFe706d2kK0yDbLXl8mZrqvN3MZve2ai1fuCrFiQD4gaEYQNapq6pTrNn72brqHTUpTINst33DBwnV71ifWD2A7MBQDCDrFBQXdF10YH1JfoqSIBeEw+HE6vMSqweQHbjQzmfrqqv1UsVm7WlpVp/8As0YPFRDe/f2OxaQ0Yp6F2rM5BFas3SDp/pxZxyT2kDIaqNP9H4jG0kaNWlEaoIgJ1Ru3qU35i1XXXW9SvoW68Tzxys6pMzvWPCAodgnW2prdffSxXrjgx0HPf/LFW9p6qAj9Y3JUxQtYsN4oCNnXzPd01DsOI7O/tS01AdC1ho/41hFh5ap8v1dXdaOmjhcIydwdzwcrmpHjR681ejN+Stk4x9eBPynb8/VpJnj9Onvuyod2MfHhOhKYJZP1DQ3aeWunVq1e5fqW1p8zVJRV6svPz//sIF4n0XbtupLzy1QZUNDmpMB2ePUyydr0nnjuqybffOFGjRqYBoSIVuFwiF94o7ZXe66Eo6EddW3LktTKmSTqh01+s/L79Ub85YfNBBLbfumvzFvuf7z8ntVxfUNGS3nh+I1Vbt15+JXdNljc/WFfz6jzz87T5c9Pld3L31NW2prfcl0z9Il2tXY2GnNtvo63buMK+aBjoTCIX3l59dqxidPUyh8+I+ygpICfeLO2brsq+f5kA7ZZvIFEzTnvquVV9D+B6iFvQr01d9+jhvBoF1/uO2v+mDTzk5rPti0U7//5l/TlAjdkdPLJ17dWqHbXn3xsFvYNra26rH1a7Vwy/v68fQZOqasf9oybayp0ZId2zzVvlCxWR801GsAyyiAduUVRHTt3Vdp1g3n66WHX9O2dTsUCoc0auIwnTr7JBX1KvQ7IrLIaVecpHFnHK0X/m+R3pi3XPU1DSrpV6JTLpmoaVdOUUlffhbjcDu37NLSecs91b7xzHLt3LJL/QezxjgT5exQXFFXq/9Y9NJhA/GBapqbdcvLC/Wn8y9R7/z0XJ3+8tYtnmvj1uqVrRWaNYozE0BnyspLOSOMpOgT7a1Lrp+pS66f6XcUZIk357992JKJjti41RvPrOCuiBkqZ5dPPLJmtRpbu97HdHdTk57euD4NidrUJrieuc7n9c8AAKBjddX1CdXXV3O9UKbKyaHYWqunNq7zXP9UGofivgmeke5bkNh+rAAAIH0SXVZTUsoynEyVk0NxU2urqpubPddvq69LYZqDnTl4qOem54VCmnbk4JTmySXWWm2prdWaqt3a2ci/xAEA3WOtt+UQknTieePbvdi3PaFwSCeeN767sZBiObmmOBJKbNbPD6Xv7kSDSkp0+uAhWrhlc5e1M4cNVylnirvUEm/V39eu0dx1q7Vpz579z08ecITcsUdrWjn/sAAAdG5XRZWe++PLevnhJdq1tUr5hXk6dtpROuea6Ro/45gOt+wrKy/V5AsmaMkTy7r8GieeP15l5aXJjo4kydmheHz/qJbvrPRUP3FAevcwvXnSyVpXXa33a/d0WDOmb6m+esKJaUyVnRpiMf2/lxe2u+fz0g+2a+kH23XNMcdrzrgJPqQDAGSD5c+v1E/mPKCm+g8/ZW6qb9ay+Su0bP4KnTLrRM2592pFOrjF9zV3fVSb3t6s7Rs6njuOGBHVp793ZdKzI3lycvmEJM0ePdZ7bZp3d+hXWKifnXWuzh82QnmHnNXOD4V16cjRun/GOeqVl54dMbLZD99Y0uFNUPb5/aq307puHACQPTau2Kz7PvebgwbiQy1+9A09dPvfOjzeJ9pb3/r713TyJRMPW0oRCod08iUT9a2/f019or2TlhvJl5NniiXp7CHD9OSGdXp9x/ZO6y4YPkITogPSlOpD/QoK9R9TTtX1EyZp0bYK7WlpUd/8fJ16ZLn65LNkwott9XWav2mDp9o/rnpHFwwb0eUdqwAAwfKP++appSnWZd3zD72ii687RwOGtn9vgz7R3rr+F9dq19YqvTl/heqr61Xct1iTZo5T2ZEsmcgGOTsUR0Ihff+0M/SfS17tcP3urFFj9O8TJ/s6KPUrLNSFI0b59vWz2dMb1qvjXagPtmFPjd7etVPj+kdTmgkAkD2qdtTojWdWeKq11mrhnxfpo1+/uNO6siNLdc4105MRD2mWs0OxJBVFIrrr1NO1umq3Hlu/VhtraiRHOrq0ny4bNUZDevExRjbbmuCuIdvq6xiKAQD7bV29XfFWr6dXpM2rKlKYBn7L6aF4n7Gl/XTjpJP8joEkO3Q9drLrAQA5jhV1OEBShmLXdR+QdImkHcaYccl4T6Ar4/tH9fd1azzVhhxHx5W1vw4MABBM5WOPUCgc8ny2eOix5SlOBD8l69TZ7yRdkKT3AjyZMWSY+nq8KHH6kYM1oIi7CAEAPtR3QB9NvsDblp1OyNGZHz81xYngp6QMxcaYFyTtSsZ7AV4VhMP66gmTuqzrlZenL447IQ2JAADZ5rIbzlNeYV6XdWd/apqiQ8rSkAh+YZElstr5w0fqG5OndHhXwoFFRbr3jLM1vE+fNCcDAGSDYccN1tce+LwKSzr+5PHU2ZP1iTuvSGMq+MFJ5P7enXFdd4SkxztaU+y67hxJcyTJGDO5ubnjTbI7EolEFIt1vZdg0AWxT7sbGvS3Ve/oxfc3qa6lWf2LinXJ2KN0/qgxyg93fBvvIPaqO+iTd/TKG/rkHb3ypqd92lmxW/Me+KcWPPSidm7ZpfyifJ0w43hdPOdcTTxnXM7scx/E76f8/HzJw2WVaRuKD2ErKhLf1iQajaqy0tutm4OMPnlHr7yhT97RK2/ok3f0yhv65E0Q+1ReXi55GIpZPgEAAIDAS8pQ7LrunyW9Kulo13U3u677uWS8LwAAAJAOSdmn2Bjz8WS8DwAAAOAHlk8AAAAg8BiKAQAAEHgMxQAAAAg8hmIAAAAEHkMxAFlrFYvH/Y4BAIBvkrL7BIDsY63VK1srNHfdar2+fbtiNq7SggKdP2yErhg9VoN79fY7IgAAacNQDARQLB7Xd197Vc9u3nTQ81VNTfrL6nf1yNrVuvOU03Tm4KE+JQQAIL1YPgEE0L3Llh42EB+oJR7XHYte1oqdwboVKJDrGvY06oP3d6p2d53fUYCMw5liIGC21dfpH+vWdFkXs1a/W7lCP5w+I/WhAKTUv55bqfkPLNTy51fJWitJOmrKKJ3z6dM15dKJCoU4RwYwFAMB88T6dfJ6Sd3ibVu1ra5O0Wg0pZkApIa1Vn/9/mN64mfPHnbsvdfW6b3X1mnZghWac+/VCoUZjBFs/BcABMz6mmrPtVbSxj01qQsDIKWee+jldgfiA706d6ke/q8n0pQIyFwMxUDAOE6C9amJASDF4q1xPX7/Ak+1C/73BdXXNKQ4EZDZGIqBgBnbt5/n2pCkUX37pi4MgJR5+8V3tXPLbk+1TfXNWvyPN1OcCMhsDMVAwFw8cpTCHk8XTy8fomhRcYoTAUiF7RsS2z1mx4YPUpQEyA4MxUDA9C8s0lVjj+6yriAc1meOPT4NiQCkQjiS2K/4UIL1QK7hvwAggL44/gTNGjWmw+PFkYh+cNrpOqpfWRpTAUim0SeOSKx+UmL1QK5hSzYggMJOSLeceLLOHzZCc9eu1us7tqmxtVXRwiJdMHykLh05SmWFRX7HBNADw44brDGTR2jN0g1d1paVl+qEc45LfSgggzEUAwE2ITpAE6ID/I4BIEWu+tYs3e3er1hLa6d1H7/9coUj4TSlSgarfC1VgbNEjhrVavtLrR+VxDUQ6D6WTwAAkKOOOnmUbnjgCyrqVdju8UheWJ//8Sc05ZJJaU7WfXlarqhzjcpCN6vE+YuKnUfVO/SAItUXqdS5TY6878UOHIgzxQAA5LAJZx2rHy2+Qy8/vESLH3tTtbvqVNS7UCeeP15nfGyqSgf28TuiZ3n6l8qcm+U4zYcds5IKnZcV1g3aZX8iq97pD4isxlAMAECOK+lbrPM+d6bO+9yZfkfpgVb1db7f7kAsfXijoTxng3rpd9pj/y190ZATWD4BAAAyXoEWK+Js9VRbpKflqD7FiZBrGIoBAEDGK3AWea4NOXXK0/IUpkEuYigGAAAZL9Ezv44aUpQEuYqhGAAAZLy4+iVYX5qiJMhVDMUAACDjNdqzJUnWdl3bageoReNSnAi5hqEYAABkvBYdo2Z7nByn45p9A3O9nS022EKiGIoBAEAWcFRlb1erbbsLZ3tnjB1HarTTVSc3zdmQC5LyzyjXdS+QdJ+ksKTfGGN+kIz3BQAA2CeuQdppf6be+qUK9byk2P5j1ilTbess1emT4iwxuqPH3zWu64bw+W3NAAAX8UlEQVQl/VTSTEmbJS1xXfcfxph3evreAAAAB4prgKrtt7RH1ynfLpOjBrWqv/r0P191O7nFM7ovGf+UmiJpjTFmnSS5rvt/kmZJYigGAAApEVeZGnX2h084ef6FQU5IxpriwZLeP+Dx5r3PAQAAAFkhbYtuXNedI2mOJBljFI1GE36PSCTSrdcFDX3yjl55Q5+8o1fe0Cfv6JU39Mkb+tSxZAzFWyQNPeDxkL3PHcQY8ytJv9r70FZWVib8haLRqLrzuqChT97RK2/ok3f0yhv65B298oY+eRPEPpWXl3uqS8ZQvETSWNd1R6ptGP6YpE8k4X0BAACAtOjxmmJjTEzS9ZLmSVrZ9pR5u6fvCwAAgs1RlYr1iHo7P1Mv5zfK12JJcb9jIUclZU2xMeZJSU8m470AAEDQtai38zMV63E5TsuHTztSzJarxn5VzZrqXzzkJO5oBwAAMkhMpc5/qMSZK6nlsKMRp0L9nG+qQC+kPxpyGkMxAADIGEV6QoXOIlnbdtvm9jhOXH2dH8hRXXrDIacxFAMAgAxhVezMldTxQLxPyKlXkZ5JQyYEBUMxAADICGFVKM/Z4Lm+wHkldWEQOAzFAAAgIziqT7C+NkVJEEQMxQAAICPE1VeSZG1i9UAyMBQDAICMENdANdvjulxPvE+jPTu1gRAoDMUAACBj1NuPSOr6bHGr7adGnZmGRAgKhmIAAJAxGnW26u2l7Z4t3jcox22hqux3JBWkNRtyW1LuaAcAAJAcjmrsjYrZoSpxjMJO5YdHHKnJTtIee51iGutjRuQihmIAAJBhHNXLVb29QgV2icLaJqs8NWu8WjXc73DIUQzFAAAgQ0XUpFP9DoGAYE0xAAAAAo+hGAAAAIHHUAwAAIDAYygGAABA4DEUAwAAIPAYigEAABB4DMUAAAAIPPYpRsarj7Vo/qaNemrDem1vqFNeKKyJ0QGaPXqsji3r73c8AACQAxiKkdFWV+3WLS8tVGVjw0HPV9TV6smN63XZyNG6cdJJioT40AMAAHQfkwQy1ra6On3thecOG4gP9I/1a3X/v95MYyoAAJCLGIqRsR569x1VNzd1Wfe3Ne+poq42DYkAAECuYihGRqqPtWjepg2eaq2kf6xbk9I8AAAgt7GmGBlp0549aojFPNev2r0rhWkAIP1C2qkiPa5w1bM6wtkmq0I1a5Lq7eVq1kRJjt8RcQhHNSrSUypynlJEFbLKU4vGqd7OUpOminORmY2hGBkpbm1K6wEgk+XrdZU6tyvk1EtxSY7kqFmFWqhCZ6Hq7QWqsTeLX+OZI6J31c/5fwo7Vfufc9SsAi1WgbNYjXaqquydkgp9y4jO8U8WZKTykl4KO97Pggzr3SeFaQAgfSJarX7ObW0DcTuslYqdp9Xb+Xmak6EjIW1XmXPLQQPxgayVCp1FKnW+l+ZkSARDMTJSaUGBzhw81HP9pSNHpzANAKRPL+cPcpyOLzJ2nL2DseYqpB1pTIaOlDhGIaemw+P7/s4KnRcU0ao0JkMiGIqRsT559LHK87D/8GlHluvofmVpSAQAqRXSThXoJXW1IsxxJMeJq0hPpicYOtGkIj3dZdW+Dz+LncdSnAfd1aPFSK7rXinpTknHSppijHk9GaEASTq6X5m+M3Wabl/0slri8XZrTogO0B1TTktzMgBIjYg2yHHa/3nXnjxnXdsWPPBNWNsVcuo810e0LoVp0BM9XaG/QtIVkn6ZhCzAYU4vH6IHZ16ov615T09v3KC6WIuktoF59qgxOn/4COWFwj6nBIBkSXTCZSL2H39nuaJHQ7ExZqUkua6bnDRAO4b17qN/n3SSvjrxRNU2tyg/HFZRhCuuAeSemIbL2pCkuLxcaxzTyJRnQudaNUhxW9zhhZGH4u8sc7GmGFkj7ITUt6CAgRhAzoprgJp0apcDsbWStSHV24vSEwydKFCDzuuyat868QZ7aYrzoLu6nC5c110gaVA7h24zxjzq9Qu5rjtH0hxJMsYoGo16DrlPJBLp1uuChj55R6+8oU/e0Stv6FMnYtfL1rwmRy3tHrZqu2grXvBRlZWMS2+2DObr91Trl2RrnpVj97R7eP/fWd6Z6tvrdHn6GCBF+G+vY45Nwk0PXNd9XtLNCVxoZysqKhL+OtFoVJWVlQm/Lmjok3f0yhv65B298oY+da5Ar6jU+XaHW7M12LNUbb8pKS+9wTKY399TeXpb/ZxvKOS0Pxg32cmqst+VVXGakx3M7z75oby8XPJwC0g+hwYAIMM06TR9YB9UsR5VSWiBHPuBrM1Tkybvvc3zKeI2z5mlRcer0v5eRfZxFTlPKqxtkiJq1vFqsLPUqNPF2JXZerol22xJP5E0QNITrusuM8acn5RkAAAEWFyDVGu/qMJ+t6mycrvaLgNiEM5kcfVTnT6lOvsp7b8/N39nWaOnu0/MlTQ3SVkAAEC72Hoy+7CXQbbhbwwAAACBx1AMAACAwGMoBgAAQOAxFAMAACDw2BsEAIBOhLVFBVosRw2Kq68aNV1WpX7HApBkDMUAALQjrK3q7dzXNhA7H97oqo+9Tw2aqT32K7Iq8TEhgGRiKAYA4BBhVajMuV5hZ5cOv/Fri4qdJ5WnNdpl7/X9DmUAkoM1xQAAHKKv832FnV2SJOeQey/se5znvKdezq/TnAxAqjAUAwBwgIhWK99Z3s4Z4sMV6Wk5qkt9KAApx1AMAMABCp0XJB1+hrg9IadB+Xo9xYkApANrigEAOICjPQnVhxKsB4IipN0q0MsKqVpx9VKTpiquI/yO1SGGYgAADmDVO6H6uHqlKAmQnRztUR/nfhXqWTlObP/z1obUpNNUY29QXAN8TNg+lk8AAHCARnu6JHlaUxy3hWrWSSlOBGQPR3tU5nxNRc48SbFDjsZV6Lyk/s5XFNIOP+J1iqEYAIADxHSUmu04T2uKG3W+LGeKgf16Oz9XnrNWUsc7t4SdHerr3JPmZF1jKAYA4BDV9htqtW13rTv0jPG+xy12tPbYL6Q5GZC5HFWrSAu6rLNWKnCWKKxNaUjlHUMxAACHaNUQ7bL3q8lObueMcVgNdubeG3dwlhjYp+3uj81d1u37b6pQL6Y4UWK40A4AgHa0aoh22x8pbDeqQIsVcurUavupSacrrv5+xwMyTqI7sYScGsnD2v10YSgGAKATrRqueg3PqF/eQCZKdCeWuE1sp5dUY/kEAAAAeqxJp8javC53btl3vFHTUx8qAQzFAAAA6DGrUjXq7C53bnEcqcmeqFaNSEsurxiKAQAAkBQ19jrF7DBJHe/c0mr7q8bekuZkXWMoBgAAQFJY9dVO+z9qsGfp0DGz7Qzxydpl71erjvQnYCe40A4AAABJY1WqanuH9miHCu1LCjnVitsSNek0tWqI3/E6xFAMAACApItroOp1Rdbs3MLyCQAAAAQeQzEAAAACj+UTAAAEUFibVOTMV0iVsipQs52sJp0qRgMEFd/5AAAEiKNq9XXuVqHzykHPlzh/V6sdqGp7k5p1ik/pAP/0aCh2XfceSZdKapa0VtK1xpiqZAQDAADJ5ahOZc6NynPWyloddpOFsLND/XSrdtvvqVlT/QkJ+KSna4rnSxpnjJkg6T1Jt/Y8EgAASIUS5yHlOWslHT4Q7+M4cfV17lbb+S4gOHp0ptgY88wBDxdJ+mjP4gAAgNRoUrGe8FQZdnar0C5Uo2amOBOQOZK5+8RnJT2VxPcDAABJkqd3FXJqPNcXOK+lMA2Qebo8U+y67gJJg9o5dJsx5tG9NbdJikn6YyfvM0fSHEkyxigajSYeNhLp1uuChj55R6+8oU/e0Stv6JN3yeqV05wn1XqvL8hvVbR39vwd8T3lDX3qmGNtz24z4rruZyR9UdI5xph6jy+zFRUVCX+taDSqysrKhF8XNPTJO3rlDX3yjl55Q5+8S1avInpP0dAcz/X19lLV2Jt6/HXThe8pb4LYp/LycknqYBX9h3q0fMJ13QskfV3SZQkMxAAAIM1iGquYHeq5vsGem8I0QObp6Zri+yX1ljTfdd1lruv+IgmZAABA0jmqs1dKkrr6kLjFjlWLJqQhE5A5err7xJhkBQEAAKnVoEuVZ1eq2Dn8uvh9+xa32qiq7Lfl4dNmIKdwRzsAAALDUY29RTE7WiXOXxV2th9wLE8N9iztsZ9XXAN9Swj4haEYAIBACaleH1W9na08u1xhfSCrQjVrvKxK/Q4H+IahGACAQAqrRRPV4ncMIEMk8+YdAAAAQFZiKAYAAEDgMRQDAAAg8BiKAQAAEHgMxQAAAAg8hmIAAAAEHkMxAAAAAo+hGAAAAIHHUAwAAIDAYygGAABA4DEUAwAAIPAYigEAABB4DMUAAAAIPIZiAAAABB5DMQAAAAIv4ncAAAAAHM5RgyJaLUeNatVAtWq4JMfvWDmLoRgAACCDOKpSL+cPKtLTCjl1+59vsUepzl6lRp3jY7rcxVAMAACQIULaoTLna4o4FQc9b62U57ynUue7qrVrVWvn+JQwd7GmGAAAICNYlTp3HDYQS5Kzd9WEtVIv508q1D/TnC33MRQDAABkgDwtV76zUtZ2XLNvOC52THpCBQhDMQAAQAYocuZL+nDw7Yi1Ur6zSmFtTEOq4GAoBgAAyAAhfeCpbt/QHFZlCtMED0MxAABARihIqNomWI/OMRQDAABkgCZ7oufauO2tFo1NYZrgYSgGABzGUY3y9brytYh1i0CaNGqm4rbYU22DLlSiZ5bRuR7tU+y67nclzZIUl7RD0meMMYfvIwIAyAphbVWJ86CK9E85TvP+55vt8aqzn1CTpvmYDshtVsWqsV9TqfO99o/btvXEMTtUtfbqNKfLfT09U3yPMWaCMWaipMcl3Z6ETAAAH0S0XmXOl1XsPH3QQNx2pfvb6he6TcX6i48JgdzXqPNUFf+m4rbksGOOIzXb8dpl75VVHx/S5bYenSk2xtQc8LBEUic76wEAMldMpc5tCjtVhx058KYBfUI/Vyw+Vs3yvvYRQGIadZ6a7HQV2meV77whR81q1QA12vPUomMldbFnG7qlx7d5dl33LknXSKqWdFaPEwEA0q5AL7V7F60DfXjTgL+qOYELggAkzqpYDbpUDfZSv6MEhmM7u22KJNd1F0ga1M6h24wxjx5Qd6ukQmPMHR28zxxJcyTJGDO5ubm5vbJORSIRxWKxhF8XNPTJO3rlDX3yLlt7FdpzvUItz3dZ1/Ybw1Fr6YtSqLTbXy9b++QHeuUNffImiH3Kz8+XPJxe73Io9sp13WGSnjTGjPNQbisqEr8eLxqNqrKSjaq7Qp+8o1fe0CfvsrVX/Z05ynPe81xfGf9fxTSy218vW/vkB3rlDX3yJoh9Ki8vlzwMxT260M513QM3yJslaVVP3g8A4A+rwpTWA0Cm6+ma4h+4rnu02rZk2yjpSz2PBABItyY7WfnOvzzVxuxgteqIFCcCgPTq6e4TH0lWEACAfxp0sXrZ38txWrusrbezxL2fAOQafqoBABRXVHtsxx/27bv8pNkep3rNSlMqAEifHm/JBgDIDfW6UopH1Mv5pUJO40HHHEdqtNNUbW8Vt5YFkIsYigEA+9VrthrsTBXZ+cpzVshRi1p1pOrthWrVCL/jAUDKMBQDAA5i1Uv1mi3Z2X5HAYC0YU0xAAAAAo+hGAAAAIHHUAwAAIDAYygGAABA4DEUAwAAIPAYigEAABB4DMUAAAAIPIZiAAAABB5DMQAAAAKPoRgAAACBx1AMAACAwGMoBgAAQOBF/A4AAEBXIlqnQud5hVStuErUZKepRcdJcvyOBiBHMBQDADJWSDvU1/m+Cpw3D3q+l/MntdijVG2/oZhG+ZQOQC5h+QQAICOFVKn+zr+pwHlT1h58zFopz3lPZc5XFdF6fwICyCkMxQCAjNTb+anCznZJknPIKol9j0NOrfo496Q5GYBcxFAMAMg4IVWqUC8cdob4UNZK+c47iujd9AQDkLMYigEAGadAS+Q4rYedIT7UvuMFWpT6UAByGkMxACDjOGpIqD7k1KUoCYCgYCgGAGScuEoTq7f9UpQEQFAwFAMAMk6Tpipui7qss1ay1lGjZqQ+FICcxlAMAMg4VsVq0EVt/7uTi+0cR2rSNLXqyDQlA5CrGIoBABlpj/2Cmu34Ti+2i9lhqrY3py8UgJzFUAwAyFCF2mV/qDrrKm5LDjpibYHq7cXaae+XTXD9MQC0h9s8AwAyWIH22OtUq2uVb19XSDWKq0TNmiyr3n6HA5BDkjIUu657k6QfShpgjKlMxnsCALCPVZGadLrfMQDksB4vn3Bdd6ik8yRt6nkcAAAAIP2Ssab4vyV9XVIXN+MEAAAAMlOPhmLXdWdJ2mKMeStJeQAAAIC063JNseu6CyQNaufQbZK+qbalE11yXXeOpDmSZIxRNBpNIGabSCTSrdcFDX3yjl55Q5+8o1fe0Cfv6JU39Mkb+tQxx3a2K3onXNcdL+lZSfV7nxoiqULSFGPMti5ebisqKhL+mtFoVJWVXMfXFfrkHb3yhj55R6+8oU/e0Stv6JM3QexTeXm5JHWy43mbbu8+YYxZLmngvseu626QdBK7TwAAACDbcPMOAAAABF7Sbt5hjBmRrPcCAAAA0qnba4p7iO3bAAAAkC5drin2a/mE050/rusu7e5rg/SHPtEr+kSvMv0PfaJX9Ik+pflPl1hTDAAAgMBjKAYAAEDgZdtQ/Cu/A2QJ+uQdvfKGPnlHr7yhT97RK2/okzf0qQN+XWgHAAAAZIxsO1MMAAAAJF3S9ilON9d1b5L0Q0kDuIve4VzX/a6kWZLiknZI+owxJvF7aweA67r3SLpUUrOktZKuNcZU+Zsq87iue6WkOyUdq7bbub/ub6LM4rruBZLukxSW9BtjzA98jpSRXNd9QNIlknYYY8b5nSdTua47VNLvJR2htm1Mf2WMuc/fVJnJdd1CSS9IKlDbXPOwMeYOf1NlLtd1w5Jel7TFGHOJ33kySVaeKd77w+I8SZv8zpLB7jHGTDDGTJT0uKTb/Q6UweZLGmeMmSDpPUm3+pwnU62QdIXafvngAHt/yfxU0oWSjpP0cdd1j/M3Vcb6naQL/A6RBWKSbjLGHCdpqqSv8D3VoSZJZxtjTpA0UdIFrutO9TlTJrtB0kq/Q2SirByKJf23pK+Lm4B0yBhTc8DDEtGrDhljnjHGxPY+XCRpiJ95MpUxZqUx5l2/c2SoKZLWGGPWGWOaJf2f2j6pwSGMMS9I2uV3jkxnjNlqjHlj7//eo7YhZrC/qTKTMcYaY2r3Pszb+4ffee1wXXeIpIsl/cbvLJko65ZPuK47S22n/N9yXdfvOBnNdd27JF0jqVrSWT7HyRaflfQXv0Mg6wyW9P4BjzdLOsWnLMgxruuOkDRJ0mKfo2SsvZ/WLJU0RtJPjTH0qn33qu2kYm+/g2SijByKXdddIGlQO4duk/RNtS2dCLzO+mSMedQYc5uk21zXvVXS9ZICu8aqq17trblNbR9Z/jGd2TKJlz4BSB/XdXtJ+pukrx3yCSAOYIxplTTRdd1SSXNd1x1njFnhd65M4rruvrX8S13XneF3nkyUkUOxMebc9p53XXe8pJGS9p0lHiLpDdd1pxhjtqUxYkboqE/t+KOkJxXgobirXrmu+xm1XfxzjjEmsB+7JfA9hYNtkTT0gMdD9j4HdJvrunlqG4j/aIx5xO882cAYU+W67nNqW7fOUHywaZIuc133IkmFkvq4rvuQMeZqn3NljIwcijtijFkuaeC+x67rbpB0ErtPHM513bHGmNV7H86StMrPPJls764BX5d0pjGm3u88yEpLJI11XXek2obhj0n6hL+RkM1c13Uk/VbSSmPMj/3Ok8lc1x0gqWXvQFwkaaaku32OlXGMMbdq74Xke88U38xAfLCsGoqRkB+4rnu02rZk2yjpSz7nyWT3q20rn/l7P4FYZIyhX4dwXXe2pJ9IGiDpCdd1lxljzvc5VkYwxsRc171e0jy1bcn2gDHmbZ9jZSTXdf8saYakqOu6myXdYYz5rb+pMtI0SZ+StNx13WV7n/umMeZJHzNlqiMlPbh3XXFIkjHGPO5zJmQh7mgHAACAwMvWLdkAAACApGEoBgAAQOAxFAMAACDwGIoBAAAQeAzFAAAACDyGYgAAAAQeQzEAAAACj6EYAAAAgff/AY8i6xZJvG3kAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(123)\n",
    "X = np.vstack([np.random.randn(10,2) + [2,2],\n",
    "               np.random.randn(10,2) + [-2,2],\n",
    "               np.random.randn(10,2) + [2,-2]])\n",
    "Y = np.repeat(np.eye(3), 10, 0)\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "matplotlib.rcParams['figure.figsize'] = (12,6)\n",
    "plt.scatter(X[:,0], X[:,1], 100, np.argmax(Y, 1), lw=2, cmap=plt.cm.viridis);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LinearMulticlass/W</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>(none)</td>\n",
       "      <td>True</td>\n",
       "      <td>(2, 3)</td>\n",
       "      <td>True</td>\n",
       "      <td>[[-0.7727087142471915, 0.7948626677932181, 0.3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearMulticlass/b</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>(none)</td>\n",
       "      <td>True</td>\n",
       "      <td>(3,)</td>\n",
       "      <td>True</td>\n",
       "      <td>[0.045490080631097156, -0.2330920609844135, -1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<__main__.LinearMulticlass at 0x7f2916e61be0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LinearMulticlass(X, Y)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'\n",
      "  Objective function value: 0.000013\n",
      "  Number of iterations: 26\n",
      "  Number of functions evaluations: 27\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'\n",
      "  Objective function value: 0.000013\n",
      "  Number of iterations: 26\n",
      "  Number of functions evaluations: 27\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "      <th>transform</th>\n",
       "      <th>trainable</th>\n",
       "      <th>shape</th>\n",
       "      <th>fixed_shape</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LinearMulticlass/W</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>(none)</td>\n",
       "      <td>True</td>\n",
       "      <td>(2, 3)</td>\n",
       "      <td>True</td>\n",
       "      <td>[[8.558497428411247, -30.636553276386415, 22.4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearMulticlass/b</th>\n",
       "      <td>Parameter</td>\n",
       "      <td>None</td>\n",
       "      <td>(none)</td>\n",
       "      <td>True</td>\n",
       "      <td>(3,)</td>\n",
       "      <td>True</td>\n",
       "      <td>[11.857844278138266, -12.947434324963158, -0.2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<__main__.LinearMulticlass at 0x7f2916e61be0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt = gpflow.train.ScipyOptimizer()\n",
    "opt.minimize(m)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "xx, yy = np.mgrid[-4:4:200j, -4:4:200j]\n",
    "X_test = np.vstack([xx.flatten(), yy.flatten()]).T\n",
    "f_test = np.dot(X_test, m.W.read_value()) + m.b.read_value()\n",
    "p_test = np.exp(f_test)\n",
    "p_test /= p_test.sum(1)[:,None]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAFpCAYAAAB0/VUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8VOW9BvDnzJLJvk62yb6RjQBhBwVEBDdwbae2rtVbr61tvbbaatW6F6lKpbi0XC8u1KKDS1VciiAoyI4hQBZCAtkXsm+TzGRmzv0jEqEkZIZZzizP9/Px02bmxzlP3kySX868530FURRBREREROTLZFIHICIiIiKSGptiIiIiIvJ5bIqJiIiIyOexKSYiIiIin8emmIiIiIh8HptiIiIiIvJ5bIqJiIiIyOexKSYiIiIin8emmIiIiIh8HptiIiIiIvJ5ConOy72liYiIiMhVhPEKpGqK0djYKNWpPYparUZbW5vUMTwGx8t6HCvbcLysx7GyDcfLehwr23C8hmk0GqvqOH2CiIiIiHwem2IiIiIi8nlsiomIiIjI57EpJiIiIiKfx6aYiIiIiHwem2IiIiIi8nlsiomIiIjI57EpJiIiIiKfx6aYiIiIiHwem2IiIiIi8nlsiomIiIjI5ymkDkBERERE7mGgbxCNx1ogWiyISVEjVB0idSSXYVNMRERE5ONa69qx8cXN2Pn+fhgHjAAAmVyGqUsmYukvFyNtcrLECZ2PTTERERGRD6s5Uo8///hl9HX2n/G4xWzB/s8O4eDmEvz85Vsx/fLJEiV0Dc4pJiIiIvJRg/0GrLx1zVkN8elMQ2a8cvebaKxscWEy12NTTEREROSjdn2wH10t3ePWmYwmbH7taxckkg6bYiIiIiIftWPDXqtrv3lvHyxmixPTSItNMREREZGPam/otLp2sM8Afc+AE9NIi00xERERkY9SqpQ21nvvGg1siomIiIh8VM7sDKtr0yYnQRWocmIaabEpJiIiIvJRF986z/ramy90YhLpOewauFarlQPYD6BBp9MtddRxiYiIiMg50iYl4eJbLsCXb35zzrqc2ZmYe910F6WShiOvFN8DoMyBxyMiIiIiJ7v5yR/g0p9dBEEmjPr81EsL8D+v/wwKP++dTww46EqxVqtNBHAlgKcB/MYRxyQiIiIi55PJZfjJo9diyR0LsO2fu1BdXAuLxYK49FhcdOMcJOclSB3RJRzV8r8A4HcAQhx0PCKSmE6nw5w5c5CUlCR1FCIicgF1YiR+8LsrpY4hGUEURbsOoNVqlwK4QqfT/UKr1V4E4L7R5hRrtdo7AdwJADqdbprRaLTrvL5CoVDAZDJJHcNjcLysN95YxcbG4r777sP999/vwlTui68t63GsbMPxsh7HyjYcr2F+fn4AMPrckNM4oileDuBmACYA/gBCAbyv0+luOsc/ExsbG+06r69Qq9Voa2uTOobH4HhZb7yxmjt3LmbMmIFVq1a5MJX74mvLehwr23C8rMexsg3Ha5hGowGsaIrtnj6h0+keBPAgAJx2pfhcDTEReYC4uDjU1dVJHYOIiMgluE4xEY0qKSkJLS0tUscgIiJyCYeuraHT6bYB2ObIYxKRNDIzM7Fp0yapYxAREbkErxQT0ajy8/Oh1+uljkFEROQS3r0KMxGdt8LCQphMJphMJigU/FFB7q+/W4+WE60QBAGxadEIDA2QOhIReRD+piOiUUVEREAQBJSVlaGgoEDqOERjaqxswScvbcbuD7+FyTi8/JRSpcCsq6di6d2XID4jVuKEROQJOH2CiMYUEBCAgwcPSh2DaEzluyrx2JXPY8eGvSMNMQAMGUzYoduLx658HuW7KyVMSESegk0xEY0pPDwcZWVlUscgGlXXyR68cPv/wtBvGLNmsM+AVbe/iu7WHhcmIyJPxKaYiMYUFxeHqqoqqWMQjWrbWzsx0Ds4bp2+ZwBfrd/tgkRE5MnYFBPRmNLT07mBB7mtHRv2Wl27/Z09TkxCRN6ATTGRmxNFEbW9PSjtaEd9Xy/s3ZrdFhMnTkR7e7vLzkdki/aGThtqO5yYhIi8AVefIHJTZtGCj49X4b2qYzjR0z3yeEZYOH6QOQFXpqZDJoy7lbtdZs6cif7+fqeeg+h8KfwUMA4YrapVqpROTkNEno5XionckMliwSO7vsFzRfvPaIgBoKq7CysO7MXje3bCLFqcmqOgoACiKHIKBbml7FkZTqklIt/EppjIDa05cghfN9afs2ZLfS1eLy1xag6ZTAZ/f3/s2rXLqechOh+Lbr3Q6tqLbaglIt/EppjIzfQPDeGD48esqt1QeRSDJtP4hXaIjIxEUVGRU89BdD4mL8rD1EvH31hm2mWTMGlhrgsSEZEnY1NM5Ga+aqjDgJWNbt/QELaPc0XZXhqNBseOWdekE7mSTCbDz1+6FXOunTZmzdzrZ+CuF2+BTMZfd0R0brzRjsjNNOttu7GtWa93UpJh6enp2LOHy1mRe/LzV+Ku1bdg2a+WYNtbO1Ff3ggASMzVYOGNc6HJipM4IRF5CjbFRG5GKZPbVK+SO/cKWE5ODj799FOnnoPIXgkT4nDj49dJHYOIPBjfTyJyM1PU0TbVT1bHOCnJsClTpkDv5KvRREREUmNTTORmJkapkRkWblVtbkQUsiMinZpn8uTJsFgs6Ovrc+p5iIiIpMSmmMjNCIKAe6dMg3KcG4P8ZDLcM2Wq0/P4+/tDLpfj0KFDTj8XERGRVNgUE7mhydEx+PMF8xHm5zfq8xEqFZ678CJMjFK7JE9gYCBKSpy7JjIREZGUeKOdB6nu6cbBtlYYzWbEBgZiTpwGfnLbbsoizzEjNh7vXXE1ttTX4uuGevQajQj188NFiUlYmJgMlQu/9uHh4Th69KjLzkdERORqbIo9wLGuTqwu/hbftp484/EwPxW0Wdm4KScXcoEX/b2Rv0KBK1PTcWVquqQ5YmJiUFNTI2kGIiIiZ2In5eYOtjTjF9s2n9UQA0C30YD/LTmEp/fthkUUJUhHviIpKQnNzc1SxyAiInIaNsVubMhixj2bPh13d7NNtTX4pPq4i1KRL8rIyEBnZ6fUMYiIiJyGTbEb+7qhHi391u1u9m5lBUReLSYnycvL45JsRETk1dgUu7Gt9XVW11Z1d6Gur9eJaciXTZkyBUNDQ7BYLFJHISIicgo2xW6s22iwrd5gWz2RteLi4iAIAqqqqqSOQkRE5BRcfcKNBSuVNtUHKUdf05bIEVQqFYqKipCVlSV1FJKIxWLBka+O4pt396KjqQtKlQK5c7Mw/4bZCIsOlToeEZFd2BS7sQs1idje2GBVbWJwMFJD+UuJnCcsLAylpaVSxyCJtNa1Y9Xtr6KurPGMx0u2V+CDlZ9D++AyXHbnQonSERHZz+6mWKvV+gP4GoDqu+O9q9PpHrX3uAQsSkzGK0eK0TU4OG7tdRlZkAmCC1KRr4qJicHx41zlxBf1tPXimR++iLb6jlGfNw+Zsf6Jf0GQCbj0vy5ybTgiIgdxxJxiA4CLdTrdZABTAFym1WpnO+C4Ps9focCzFy+BYpxmd2ZsHK7LmOCiVOSrUlNTUVtbK3UMksDHL34xZkN8Ot2fPkZfp3Ur5hARuRu7m2KdTifqdLpTazUpv/uPa4M5yLzkFKyctxCpIWdPjVDKZLguIwvPzJ0PhYz3TJJz5eXl4eTJszeRIe9mGDBiu26PVbUmo8nqWiIid+OQOcVarVYO4ACATAAv6XT8qehIU2NisW7JFShqPYmi1pMwWsyIDQzCosRkhKlUUscjHzF9+nSuVeyD6ssaMdAz/hSuU47u4QolROSZBEdu+KDVasMBfADgVzqd7sh/PHcngDsBQKfTTTMajQ47rzdTKBQwjbOjHX2P42U9W8fKaDQiJCQEDQ0NUKvVTkzmnnz1tXVkexl+v/gpq+unXlKA5Z8/7JNjdb589bV1PjhWtuF4DfPz8wOAcW+8cujqEzqdrkur1W4FcBmAI//x3BoAa777UGxra3Pkqb2WWq0Gx8p6HC/rnc9Y+fn5YePGjVi6dKmTUrkvX31tKULkNtWHxobAZDL55FidL199bZ0PjpVtOF7DNBqNVXV2T0TVarXR310hhlarDQCwGEC5vcclIvcTHh6Ob7/9VuoY5ELqxEjkzrV+bep52llOTENE5DyOuDsrHsBWrVZ7CMA+AF/odLqNDjguEbmZ+Ph4HD16VOoY5GJX/HyRVXXZszKQMTXFyWmIiJzD7ukTOp3uEIBCB2QhIjeXlpaGw4cPSx2DXGzSwlz8+NFrsP7xf41Zk5Adh7v/dhsErpdORB6KO9qRV9CbhlDf24tm0YLAIRNC/bjltTPk5ORg69atUscgCVz2s4VIytHgs79vxeFtZSOPh8eGYsGP5+CyOxciMDRAwoRERPZhU0weram/D28dLcPnNdUYNA/fYasQBCxMTMaN2bnIDI+QOKF3mTRpEpdl82H587KRPy8bXS3d6GzuhtJfibj0GCiUtt2MR0TkjtgUk8c62tmBe7dvRc9/LO9nEkV8UVeDrxvr8dTsCzEn3rq7Tml8hYWFMJvNMBqNp5a4IR8UHhuG8NgwqWMQETkUt0Ejj9Q/NIT7v/nqrIb4dAazGY/s3oHGfl7ZdJTQ0FDIZDIcOXJk/GIiIiIPwqaYPNLnNSfQMTj+LluDZjPerzzmgkS+IyAgAIcOHZI6BhERkUOxKSaP9GnNCatrP6s5Dkfu3OjrwsLCUFFRIXUMIiIih2JTTB6ppb/f6tpuoxEDZm5z6SgxMTGoqamROgYREZFDsSkmj6SU2/bSVcr4UneUxMRENDY2Sh2DiIjIodgpkEearI6xunZipBpKGZeMcpT09HS0t7dLHYOIiMih2BSTR7o2I8vq2msyMp2YxPfk5ORwrWIiIvI6bIrJI02KUuOylNRx66ZGx+CSpBTnB/IhkydPhsFgkDoGERGRQ7EpJo8kCAIemDYLV6ePfRX4wvgEPDN3PhScT+xQycnJAIC6ujqJkxARETkOd7Qjj6WQyXD/1Bm4ISsHHx6vREVXB+QKJRICArAsLQPZEZFSR/RKMpkMKpUKRUVFSEpKkjoOEXm5xmPN+PLNb3BwSwkMeiOCIwIx66qpuOgnc7izIjkUm2LyeEkhIfjl5EIAgFqtRltbm8SJvF9ISAhKS0tx1VVXSR2FiLzYxhe/wLsrPjljrfmetl588Pxn2PjSZty1+mZMv3yyhAnJm/B9ZSKyWUxMDDfwICKn2vLmDmx4ZuOYmy8NDQ7hpZ+/jrKd3LWUHINNsYczmM3oHByEwWyWOgr5kKSkJM4pJiKnMQ4Y8d6fPxm3zmKy4F0r6oiswekTHkgURextacZ7lRXY1dwIEYAMAubEa/CDzCzMiI2XOiJ5uezsbOzfv1/qGETkpfZ9Uoz+Lr1VtZX7T6C2tAHJeQlOTkXejleKPYwoilhdXITf7tiGnd81xABggYhvmhpw7/ZtePlQ0ZhvNxE5QmFhIbq7u6WOQUReqvqIbe9E1Rypd1IS8iVsij3M28fKoas8es6af1aU470qzrEi55k1axZMJhP0euuu5BAR2UK02HZhx9Z6otGwKfYgRrMZ/ygvs6p2XXkJTBaLkxORrwoLC4NCocDu3buljkJEXkiTFevUeqLRsCn2INsbG9BttG4nsfbBQexsanRyIvJloaGhOHDggNQxiMgLzb56GvwC/KyqTcyOR8bUVOcGIp/AptiD1PX22FRfa2M9kS3i4uJQWloqdQwi8kKBoQG4/M6FVtVec+9lEATByYnIF7Ap9iAyG7/pba0nskVycjJqamqkjkFEXuqa316GBT+Zc86aGx+7FjOWTnFRIvJ2XJLNg+RG2rZtcW5klJOSEAE5OTnYu3ev1DGIyEvJZDL8dMWPMO2ySdjyxnYc2loG0SJCqVJi1lWFuOS2eUibnCx1TPIibIo9yLSYOCQGB6O+r2/c2tSQUExRR7sgFfmqgoIC9Pb2Sh2DiLyYIAiYfHEeJl+cB7PJjJDAEPQN9kEm4xvd5Hh8VXkQmSDgFwWFGG9ShADg5wVTOMeKnGr69OkYGhqCyWSSOgoR+QC5Qo7A0EA2xOQ0fGV5mPkJifjDjNlQjvFDQSmT4ZGZc3CBhjv7kHOp1WoIgoCKigqpoxAREdmN0yc80OUpaZgeE4uPjlfh68Y69BqHEOLnh4sSkrAsLR3qgECpI5KP8Pf3R3FxMfLy8qSOQkREZBc2xR4qOiAQd+QX4I78AqmjkA8LDQ1FeXm51DGIiIjsZndTrNVqkwC8CSAWgAhgjU6nW2XvcYnI/UVHR+PEiRNSxyAiIrKbI64UmwD8VqfTfavVakMAHNBqtV/odDqu6k/k5TQaDerr66WOQT6g/mgTvnxzB4o2HcFA3yCCI4Mwa1khFt50AdSJti1XSUQ0GrubYp1O1wSg6bv/36vVassAJABgU0zk5dLS0lBUVCR1DPJyH7/4Bd59ZuMZjw30DmLji5vx+d+34r9W/gRzrp0uUToi8hYOXX1Cq9WmAigEsMeRxyWi8yeKotOOnZuby7WKyam2vLnjrIb4dKYhM/5+zz9waGuZC1MRkTcSHPULU6vVBgP4CsDTOp3u/VGevxPAnQCg0+mmGY1Gh5zX2ykUCq4DawOO17DOgQFsKC/Be+VlqO/phkImQ2FcPG7IK8DitHTIZTKHjFVpaSmmTp2KwcFBByV3X3xtWc9RY2UcNOLWzF+jp238P7wypqRi1a6nPHJ9dr62rMexsg3Ha5ifnx+Acbd5cExTrNVqlQA2Avi3TqdbacU/ERsbG+0+ry9Qq9Voa2uTOobH4HgBpR3tuH/HV+g2GkZ9flpMLJbPmYfk+Hi7x8pisSApKQlFRUWIiYmx61jujq8t6zlqrHZ9sB9/+9U6q+sf3fgbpE9Jsfu8rsbXlvU4VrbheA3TaDSAFU2x3dMntFqtAOD/AJRZ2RATkZM09/fjvh3bxmyIAeDAyRY8tnenQ6ZVyGQyKJVKHDx40O5jEf2n2pIGm+prjthWT0R0OkfMKb4AwM0ALtZqtQe/++8KBxyXiGz09rFy9FgxNWlnUyOKT7Y45JzBwcE4cuSIQ45FdDrb/25z3vx5IvJ+jlh9YgesuCRNRM5lMJvxWbX1awa/XXIY900qtPu8arWaWz2TU2gmxNpWnxXnpCRE5AscuvoEEUnnpF6PftOQ1fXHOjscct7ExETU1NQ45FhEp5u1rBABIf5W1WqyYjFhZrqTExGRN2NTTOQlpLrpPjMzE83NzdKcnLyaKlCFK+++xKraa397uUeuPEFE7oNNMZGXiA4IRLBSaXX9hMgoh5x3ypQp6O7udsixiP7T0rsvwZI7Foz5vCAIuPHx6zBzqf1TgYjItzlim2cicgMquRyXp6RhQ6V183tvyJvokPPOnj0bBoMBJpMJCgV/pJBjnWp6p15agC1v7EDRF4dhMprhH6TCzGWFuOS2eUiZmCh1TCLyAvwNRuRFfjQhB/+urR53BYoL4xMwKSYW7e3tdp8zLi4OMpkMxcXFmDZtmt3HO11fZz8Obi5Bb0cfAkICMGlhLiLjwx16DvIMuXOzkDs3C6IoYshgglKlOO/pEgN9g2g42gyzyYzopChEaviaIiI2xUReJS4wCCsvvAj3f/MVOg2jr1U8MzYOf5w1x6HzL4ODg7Fr1y6HNcUDvYP45+MfYOf7+2Eyfr8bk0wuw7TLJ+Gmx69DeGyYQ85FnkUQBPj5Wz9N6HTtjZ34ePUX2Pn+fhj6v//+KLgoF0vvvgQ5czIdFZOIPBCbYiIvkxMZhbcuvRIbTxzHJ9XH0djfB4Ugw8QoNa7NyMIFGg3kgmNvJ4iJiXHYWsUDvYNY/sPVqDlSf9ZzFrMF+zYexIniWjz8wf8gIo6NMVmn8VgzntG+iO7Ws7eMPrytDEe+Lsftf74B82+YLUE6InIHbIqJvFConwo/yc7FT7JzXXK+lJQUnDhh/RrJ5/L2Ux+O2hCfrq2uA2vvX4/frrvLIeck72YymrDytjWjNsSniBYRa3/3NhKy45FR6HlbRROR/bj6BBHZbcKECWhpsX+HvP4uPb55d69VtYe2lqH5+Em7z0neb/9nh9BaM/78edEiYtOr25wfiIjcEptiIrLbxIkT0dPTY/dxir8sxZDBNH7hd/Z9Wmz3Ocn77dhg3R9awPBrarB/9Pn4ROTd2BQTkd2mTZsGg8EAi8Vi13H6Ovttqu/v1Nt1PvINHU2dVteah8znnGZBRN6LTTER2S0pKQkA7J5XHBgWYFN9QKh1WwCTb1OqbFut4nxXtyAiz8Yb7RzEYDZjW30tDra1wmg2IyYwEJenpCE5JFTqaEQuoVKpcPDgQWRkZJz3MSZdlAu5Ug7zkNmq+mmXFpz3uch35M7JQvWhOqtq49JjEB7Ln9t0biajCQc+P4Rj+07ANGRCTIoac6+bzqUiPRybYgfYUleLlUX70W08cx7auvJSzNMk4A/TZyPEz0+idESuERISgvLycruOEaoOwaxlhdj5/v5xa3NmZyIxR2PX+cg3LLz5Any+ZitEURy39uJbLnDoGt7kfXb96wD++dj76GnrO+PxDSs2Yp52Fm564nq+2+ChPG76ROfgIHY2NeCrhjoc6+q06oecM22qrcaje745qyE+ZXtjA+7dvhUDJutvHiLyRNHR0aiqqrL7OD9+9FrEpcecsyZUHYzbn/ux3eci3xCbqsayXy8ety59SgoW3jjXBYnIU339zm787ZdvntUQA4DFZMFX/9yFv/7X/8Fk5btd5F485kpxQ18v/rfkMLbV18J0WiM8ITwCN+Xk4eLEZJdn6h8awrPf7hu3rryzA+9UlOO2vIkuSEUkDY1Gg4aGBruPExoVjIfe/zVef0CHb/99+Kw/fPMuyMJtK25AbKra7nOR77juvisgV8jx0V83jTo9p+CiXPz8pVvgF8B39Wh0PW29eOPBDePWHd5Whu3v7MbCmy5wQSpyJI9oio91deKer79Ej9F41nMVXZ344+5vUJffi1tz812a6/OaE1ZfAf7wRCVuysmDQuZxF+eJrJKSkoKSkhKHHCtUHYJfv3oHWuvaceCzQ+jt6ENgSAAKl0yEJivOIecg3yIIAq659zJcdONcbH9nNyoPVMM8ZEZMqhrzb5iN1IIkqSOSm/v67d1nbDt/Lptf346LbpzLqTgexu2b4iGLGQ/u3D5qQ3y6/y05hJyISMyKi3dRMmBPS5PVta0DAzjR042s8AgnJiKSTnZ2NtavX+/QY0YnReGyOxc69Jjk28JjQrHsV0ukjkEeqHhrmdW19eVN6GjqQpSGv/M9idtftvyqoR7NeuvWLtUdO+rkNGcatHGeMOcVkzcrKCjA4OCg1DGIiJxisM+2n2+DfdwExtO4fVO8qbba6to9LU3oNLjul3KUv21rqtpaT+RJ8vPzIYoiuru7pY5CRORwoeoQm+pDooKdlIScxe2b4nYbrzx1uPBK1ZLkVKtrJ0aqkRDMbxBbDJpMONTWin0tzahxwBbC5FwKhQIKhQIHDx6UOgoR0TmJooiB3kH0dfbDYrZuJ87ZV0+1+vj587MRyqbY47j9nOIAudy2eoXrPqWZcXFICQlFTe/4DdsPs7JdkMg7dBkMeLOsBJ/WHEff0NDI4zkRkfjxhBwsSkqRMB2dS2BgIA4fPowFCxZIHYWI6CzGASO+fmcPvly3Aw1HmwEM76R54Q9nYvFP5yMmZexVbWYtK4Ru+cfosWIb8CV38GegJ3L7K8XTY62/0zwxOARxgUFOTHMmuSDD8rnzEKk691azw42c65eM80RtA3rctXUTdJVHz2iIgeGl7R7dsxN/P1IsUToaT1RUFCoqKqSOQUR0lp72Pjx13Sqse/jdkYYYAPTdA9j06ld4eMmfUbJ97HuT/AL8cM+r/wX/INU5z7PsV4sxZZFrV8Mix3D7pviqtAworVzG7Nr0TMhcvPxJckgo1ly8BJcmp56VMzUkFA9On4VfFExxaSZPJYoiHtn9Der7zl4U/XTrykuxpa7WRanIFgkJCaiurpY6BhHRGSwWC/76X6+i5nD9mDWGfgNW3fEqmqpaxqzJnJaKh//1P5h8cd5Zy63Fpcfgjud/jB/8fqnDcpNruf30iUj/APxyUiH+cvDAOesmRUXjmowsF6U6U1xQEB6ZOQe/mjwVJe1tMFrMiA0MQm5EJNcotMGR9jYcbm+zqvbtijJefXdD6enp2Lx5s9QxiIjOUPbNMRzbd2LcOoPeiM/XbMNPV/xozJqkXA1+8+Z/o7WufXi9a5MZMclRyJqRzt/5Hs7tm2IAuD5zApQyGV4+fPCst9QBYGFiEh6cNgsqG+cfO1q4SoULNAmSZvBkn9uw0khZZwdqenqQEhrqvEBks4kTJ2LDhvF3fCIicqVt/9xlde3O9/fjJ49dC9U4uxtGJ0UhOinK3mjkRjyiKQaAq9IzsTg5FZvranC4rRVDFgvig4JxRWoaEoNtWyaF3FOrXm9T/ckBPZtiNzN79mwMDAzAYrFAxt0bichNNFedtLrWOGBEV3M3YtOinZiI3JHHNMXA8MoSy9IysCwtQ+oo5AQqhW1X+qV+Z4DOlpaWBkEQUFFRgZycHKnjEBEBAAS5bdMaZHL+Ue+LHPJV12q1a7Va7UmtVnvEEccj3zQtOtbq2mClEhO4ZbbbkclkCAwMxK5d1r9VSUTkbOmTrb8HJVQdjIj4cCemIXflqD+FXgdwmYOORT5qSXIqAq1cZ/rK1HT4u3BNarKeWq1GcTGXzSMi97Hw5gusrl3wk7lQKPlOpC9ySFOs0+m+BtDhiGOR7wpUKnFv4fRx6xKDg3FLDteAdFdJSUmoqqqSOgYR0YiU/ETMvX7GuHWRmnAsuX2+CxKRO+KkGXIrl6ek4aHps8a8YjwxUo3VCxYhTHXuxdNJOhMmTEBTU5PUMYiIznDHszdgzjXTxnw+JkWN3799N0LVvHnfVwmiKDrkQFqtNhXARp1ON3GM5+8EcCcA6HS6aUaj0SHn9XYKhQImk0nqGC7XZzTi42NHsbuhHgazCbFBwbh6Qg4KY+POuQ6kr47X+XDWWL3xxhu455570NXV5fBjS4mvLetxrGx4J73UAAAgAElEQVTD8bKevWMliiLK91Tik79/gbLdx2AeMiMuLQZLbluAC6+fBT//cy/D5mn42hrm5+cHAOPebemypvg/iI2NjQ45r7dTq9Voa7NuQwvieNnCWWN17NgxLFy4EPX1Y+8c5Yn42rIex8o2HC/rcaxsw/EaptFoACuaYk6fICKHysjIgCiKaGhokDoKERGR1Ry1JNt6ALsAZGu12nqtVnuHI45LRJ5HJpPBz88PRUVFUkchIiKymkPWtNLpdD92xHGIyDuEhISgtLQUS5culToKERGRVTh9gogcLioqCsePH5c6BhERkdXYFBORw8XHx6Ourk7qGERERFZjU0xEDpeSkoKTJ09KHYOIiMhqbIqJyOEmTJiA7u5uqWMQERFZjU0xkZcaNJnQbTDALFpcfu6JEydiYGDA5eclIiI6Xw5ZfYKI3IPJYsGWuhp8UFWJIx3DC7YHyBW4JDkFP8ycgPSwcJfkKCgogMVigV6vR2BgoEvOSUREZA9eKSbyEgazCb//5ms8uW/3SEMMAANmEz4+UYWfbv4cm2qrXZLF398fcrkchw8fdsn5iIiI7MWmmMhLrDiwF3tamsZ83iyKeGrfbhS3uuYGuICAABw6dMgl5yKiYcYBIxorW9B4rBnGAaPUcUgiJpOJ93WcB06fIPICdb292FRbM26dRRTxZnkpno+OcXqmiIgIlJeXO/08RAS01rXjs79txTfv7cVgnwEA4B+swgXXz8Tl/70Q0clREickR7FYLGhpaUFJSQnKy8tRVVWFqqoqNDU1obOzEwaDARaLBVOnTsWuXbukjutR2BQTeYGN1VVW1+5paUJzfz/UarUTEw2vVVxdXe3UcxARUPVtNZ67+W/Qd595c+tgnwFb3tiOXf/aj/vW3YWMqanSBCSr6PV67Nu3D4cOHUJ1dTUaGxvR1taGnp4e9PX1YXBwEEajERbL8M3TSqUSgYGBCA8PR0JCAi655BLk5+cjNzcX2dnZCA4Olvgz8jxsiom8QG1vj031dX29mOikLKekp6dj586dTj4LkW/r7ejDytvWnNUQn07fPYCVt63BM9v+gJBINkquZLFY0NTUhBMnTqCurg719fVobm5Ga2sr2tra0N7eju7ubuj1epjNZsjlcgQHByM8PBxRUVFITk5GbGws4uLikJCQgNTUVKSlpSEyMlLqT80rsSkm8gIyQbCpXm5j/fnIy8vDxx9/7PTzEPmyr/65C30d/ePW9XX046t/7sLSXy52QSrf0d3djbKyMlRUVIz8V19fj46ODgwMDMBkMgEA5HI5VCoVAgICEBwcjNDQUKjVahQUFCAtLQ05OTmYPn06r+5KjE0xkRfIiYjEVw31VtUqZTKXLM02c+ZM6PV6p5+HyJd9/c4em2rZFI/PZDLh8OHDKC4uRlVVFWpra9Ha2oquri709fVhYGAARqPxjIbX398fISEh0Gg0mDlzJnJycpCTk4O8vDzExDj/Hg5yDDbFRF7gytR0/F/JYZhEcdzaixKSEK5SOT1Tfn4+RFFEU1MT4uPjnX4+Il/UVtfulFpv1N3djaqqKlRXV6OhoQFNTU1oa2tDS0sL2tvb0dPTg97eXhiNRgiCgMDAQISEhECtViM6OhrZ2dmIj4+HRqNBSkoKUlNTkZCQAJmMC3l5CzbFRF4g0j8AN2bn4Y3yknPWBSmUuC3X2bOJh8lkMvj7+2P79u3QarUuOSeRr5ErFTCbrFt6Ta703l/5g4ODqKysREVFxch0htraWrS3t6Ovrw8Gw/CKHDKZDEqlEv7+/ggKCkJoaCgiIyMxdepUJCUlITc3FzNnzuTVXR/lvd8hRD7mjvwCDJpNeOfY0VGfD1ep8Mzc+UgJDXVZpsjISBQVFbEpJnKSCTPTceQr65Y+zJqR5uQ0jldTU4OioiKUl5fjxIkT6OjowMmTJ9Hb2wu9Xg+DwYChoSGIoghBEODn54fQ0FDExsYiKysLV1xxBbKzszFx4kSkpqbyqi6dE5tiIi8hEwT8avJUXJaShn8dr8T+lmYYzGZEBwTgspQ0XJqSimCln0szJSYm4ujR0Zt0IrLfxTdfYHVTfPHNFzg5jXWMRiNqampQVVWF+vp6NDY2orW1FS0tLWhra0NXVxe6u7thMBggiiJUKhWCg4MRFRWF+Ph45OXljazGkJiYiOTkZKSnp8Pf31/qT408HJtiIi+TFR6B+6fOkDoGACAjIwM7duyQOgaR1ypcPBEFF+Xi8Layc9YVLMjB1CUFTs9jsVhQW1uLiooKHDlyBBUVFaipqcHJkyfR3d2NwcHBkau6SqUSKpUKgYGBCA4ORmRkJLKyskYa32nTpmJCWh8C5d9AQC9EhEAVugxt3UkAnL+CDvkeNsVE5DQTJ07ERx99JHUMIq8lk8vwqzU/xd9//Q8c+Hz0bdWnXlqAu1bfDJn8/KcOdHd3o6ioCIcOHUJlZeXIzWnd3d3o7+/H4OAghoaGzthYIigoCGq1GklJSZg+fTry8vJQUFCAzMzMca/qylGNcOEpKIXKM5/o3YAoIQtd4sMwI+W8Px+i0bApJiKnmTZtGpdlI3IyVaAKv371Dhw/WINtb+1CXVkjACAxJx4Lb5qL9CmjN4+nNpaorKxEXV0dmpqa0NzcPDKNobOzE52dnRgYGIDFYoFCoUBQUBAiIyMRFRWFlJQUxMbGQqPRIDExEUlJScjMzERYWJhdn48ctYgSfgWZ0HvWcyIApXAMUfgV2sWXYEaSXeciOh2bYiJymtzcXIiiiI6ODu7ARORk6VNSzmiAOzs7UVJSgi/+/jkqKipGtg4+tbGE2WwGACgUCvj5+Y1sLBEeHo7Y2FhMnjwZOTk5mDJlCvLz8+Hn55p7EkKFF0ZtiIHvJ03IhB6E4i/oFFe6JBP5BjbFROQ0CoUCCoUCRUVFWLRokdRxiDye0WhEcXExDh48iKqqKjQ1NY3M1z21sYTBYBhpeOVyOQIDAxEREYH4+HjMnTsXOTk5yM/PR05Ojtv9sSpHDVTCtxBF4Fwbb4oioBK+hVys4TQKchg2xUTkVMHBwTh8+DCbYqJz6OzsRGVlJWpra9HY2IjGxsaRebunpjL09/fDZDJBJpMhKCgIYWFhiIiIQHR09MiKDBqNBsnJycjMzERsbKzHLUGmwl4A526IT39ehX3QsykmB2FTTEROFRkZicrKyvELibyU0WhEWVkZjhw5gmPHjo3M4W1tbYVer8fQ0BCA4au6p1ZkCA4ORlhYGCIjIzF9+nRkZmYiLy8PhYWFbnd115EEDNhYz3sWyHHYFBORU8XHx6Ourk7qGEQOZbFYcOzYMezfvx+VlZVoaGhAc3Mzurq6zthYwmQyQRTFkR0eQ0JCEBMTg7y8POTm5iIvLw+zZs1CUFCQx13VdQYLImys994/EMj12BQTkVMlJydzrWLyGHq9HtXV1aipqRnZWKK5uRltbW1obW1FR0fHyLbBgiAgICAAISEhCA8PR3R09MiKDHFxcUhOTkZaWhpSUlLOeZOaWq1GW1ubCz9L9zWIeQgV/wpgaNw5xYASg5jnomTkC9gUE5FTZWVlYePGjVLHIILFYsGJEydQVlaGioqKka2DT548OdLojraxRGhoKMLDw5GTk4PU1FRkZGRg5syZSEnhXFZHExGOASxBoPDJOesEAdCLl0KEfcu/EZ2OTTEROVV+fj7XKianam5uxt69e0d2T2tsbERXVxd6enrO2lhCEAT4+fkhMDBw5MruokWLkJWVhdzcXGRmZrps6TEaXa94NxQ4Dj9h7F36jGIeesW7XZiKfIFDmmKtVnsZgFUA5ABe1el0zzjiuETk+SZNmgSz2Qyj0chmg6xmsVjQ0NCA48ePo66ubmTO7smTJ3Hy5El0dnaip6cHer0eoijCz88PQUFBI6sxZGRkIDY2FrGxsUhKSkJqaipSU1Pt3liCnE9EIDrFlQjCGwjEJ2esWSwKoei3XIE+8TYA594Vj8hWdjfFWq1WDuAlAIsB1APYp9VqP9LpdKX2HpuIPF9oaChkMhlKSkpQWFgodRxyE83NzTh69CjKy8tRWlqK48ePo6mpCT09PRgcHDxrY4nAwMCRFRkSExMxe/ZspKenY/r06cjNzYVCwTc+vYmIAPSJd6EPt8FPLIYMvbAgBKFRi9DX3id1PPJSjvgpMhNApU6nOw4AWq32bQBXA2BTTEQAAH9/fxw+fJhNsZfr6+tDUVERqqurUVxcjKamJrS1taG7uxv9/f0YGBiA0Wg8o+ENCAhAVFQUEhMTcdlllyEzMxP5+fnIzs5GaGioxJ8RSc8fRsz6/kPBHwCbYnIORzTFCQBOX2+pHjj9FUxEvi48PBxlZWPPDyT31tHRgaqqKlRXV6OpqQlNTU1obm4eWY3h1MYSZrMZcrl8ZGOJqKgoxMXFoaCgAHFxcUhMTERiYuLI1AYuQUZE7sRl7zdptdo7AdwJADqdDmq12lWn9mgKhYJjZQOOl/VcOVaJiYmor6/36K+Nt762+vr6UFJSgoMHD+Lw4cM4duwY6uvr0dbWhv7+/jM2lvDz80NAQACCg4MRHh6OmJgYTJo0CVlZWZg5cyZmzJiBwMBAKBQKmEwmiT8zz+Gtry1n4FjZhuNlG0c0xQ0Akk77OPG7x86g0+nWAFjz3Yci12S0DtevtA3Hy3quHKuEhAQUFxd79NfGk15bFosF5eXlKCoqQnl5ORoaGka2Cu7r6xtzY4mIiAjExcWhsLAQmZmZyM3NRX5+PmJiYqy6qqvX66HX6z1qrNwBx8t6HCvbcLyGaTQaq+oc0RTvA5Cl1WrTMNwM3wDgJw44LhF5iZycHGzZskXqGB5Pr9fj+PHjIxtLNDc3o6mpCSdPnkRHRwfa29vR29uLoaGhkY0lQkNDERkZiaioKKSkpCAuLg4ajQYJCQnIyMhASkoKb1IjIoIDmmKdTmfSarW/BPBvDC/Jtlan05XYnYyIvMb06dPR398vdQy3ZjKZcOLECRw6dAhlZWWorKxEXV0dWltbRxrd0zeW8Pf3R0BAAMLCwhAZGYmcnBxkZGSgoKAAU6dORUxMjNSfEpFVBPTCH19CITRAhBxDYj4MmA1upUCu5pBXnE6n+xTAp444FhF5n6lTp8JsNqOzsxMRERFSx3GphoYG7Nu3D6WlpWhoaEBLS8vIFd3+/n4YDIYzNpZQqVQIDQ1FdHQ0UlNTsXjxYuTl5SE7OxtpaWlc65m8iAnBwv8hCO9DEAzfPywAZjEaPeLdMOAiydKR7+GfYUTkdH5+flAqldi5cyeuvPJKqePYzWQyoba2FtXV1WhoaEBjYyMaGxvR2tqK1tZWtLW1oaenZ2TbYJVKhZCQEISHhyMqKgoTJkxATEwM4uPjR6YxpKenIzAwUOpPjchFLAgT/oQA4ctRn5ULrYgQHkO35fcYwOUuzka+ik0xEblEeHg4Dhw44BFNcVNTE4qLi1FeXo7KykocP34cJ0+eRFdXFwwGAywWC4DhO7tVKhUCAgIQEhKCiIgIJCYmYt68ecjJycHkyZORlZXFpceI/oM/vkSA8CVEERCEsetChedhEGfCgijXhSOfxaaYiFwiISEB5eXlkp2/s7MTBw4cQGlpKWpqakY2lujp6Rl1YwmlUomgoCBERkYiOTkZ8+fPR0pKCrKzs5GZmcmNJYjsECj8C8C5G+Lh500IED9FP252QSrydWyKicglMjIycODAAYce02KxoK2tDSdOnEBtbe3IigwtLS0j0xhO7aZmsVigUCgQHByM0NBQREVFQaPRoLCwELGxsdBoNEhNTUV6ejrUavVZV3e5tBGRYwjohp9wZNyrxKeohJ3oF9kUk/OxKSYilygoKMAnn3yC1tZWREdHW/Vvurq6UFJSgqNHj6KsrAxVVVVoaGhAV1cXBgcHRzaIkMvlUCqVIxtLhIWFQa1WY9KkScjIyMDEiRMxdepU3qRG5AZk0AOwriEerue2zuQabIqJyCVKSkqQnZ2NBQsWYPHixdBoNGdsFdzd3Y2+vj4MDAzAYDDAbDZDFEXI5fKRjSUSEhIwf/58ZGdnIysrC7m5uVx6jMjDWBACURQAiFY1xhaEOT0TEcCmmIicrLe3Fy+88AK2bNmCO+64AwcOHMBnn30Gk8mEyMhIqNVqqNXqkRUZEhISkJSUhJSUFG4sQeSFRATDiBlQCXutqh8UL3ZyIqJh/G1DRE7R19eHv/71r3jttdcgiiJiY2PR3t6OH/zgB3jmmWeg0WggWPv+KRF5lX7xeqiEvePOK7aIQRjAEtcFI5/GppiIHOqzzz7DqlWrUFJSApVKhWuvvRY/+9nPMGHCBKmjEZGbMGIW+sQfI1hYf9ZzpxplUVSiS3wMIoIlSEi+iE0xEdmts7MTK1aswL/+9S/09/ejsLAQr732GhYtWsSrwUQ0qj7xTphFDYKFf0AutIw8LgiAUSxAr3gXhpAvYULyNWyKiei87dmzB0888QSKi4sRERGBG2+8Effeey+Cg3llh4jGI2AAyzAgXgE/sQhyNACQYwh5MCFd6nDkg9gUE5FNjEYjXnzxRbz++uvo6OhAQUEBNmzYgDlz5kgdjYg8khxGTAcwXeog5OPYFBORVY4fP44nnngCX375Jfz8/LBs2TI88sgjiIyMlDoaERGR3dgUE9GYLBYL1q5dizVr1qChoQFJSUlYvnw5brzxRqmjERERORSbYiI6y+7du/H8889j7969EAQBS5YswYYNG5CSkiJ1NCIiIqdgU0xEAICenh48//zzeO+999DV1YWcnBw899xzuP766yGTyaSOR0RE5FRsiol83L59+/DEE0+gqKgIoaGhuOaaa3D//fcjIiJC6mhEREQuw6aYyAeZTCY8/fTTePnll9He3o68vDysX78e8+bNkzoaERGRJNgUE/mQmpoaPP7449iyZQuUSiWWLl2KP/7xj1xBgoiIfB6bYnK5boMBn9Ycx5d1deg2GhCsVOKC+ARclZ6B6IBAqeN5HYvFgnfeeQerV69GTU0NEhMT8dRTT+Gee+5BR0eH1PGIiIjcApticqmdTQ14dM9ODJhMZzxe0dWJdeUl+O3UGViWliFROu9SXl6OZ555Bl9//TVMJhMWLlyIdevWISNjeHx58xwREdH32BSTyxS3nsQfdm6HSRRHfd4kilhxYC8CFApcksSlv86HXq/H6tWr8fbbb+PkyZNISUnBgw8+iJ/+9KdQKPjtTkRENBb+liSXefnwwTEb4tO9dKgIFyUkQcErmVYrKirCk08+iX379iEwMBCXX345HnjgAcTFxUkdjYiIyCOwKSaXONbViZKOdqtqWwcGsLOpEfMTEp2cyrOZTCa8/PLLWLt2Ldra2pCdnY3XX38dixYtkjoaERGRx2FTTC5RamVD/H19G5viMdTV1eGJJ57AF198AZlMhmXLluGRRx6BWq2WOhoR2cSAAHwJec+/ESOcgAg5hpADvXgVjJgJgO+WuSs5GhAofAQVtkOGXlgQAgPmQS9eBTMSpI5H54lNMbmE2YppE/bU+4L3338fK1euxIkTJ6DRaPDoo4/i1ltv5Q1zRB5IjkZECL+DQqgHTIAgnHp8J/yFnTCIM9AlPg4RXJHH3QTgY4QKf4EgWEYek6EXCryDQGxAj3gvBrBMwoR0vtgUk0skBQfbWB/ipCSepaGhAcuXL8e///1vGAwGXHjhhXj11VeRk5MjdTQiOk8CehEh/BYKoWnU50URUAn7EI7H0Sk+A0BwbUAakwrbECZ7fsznBcGCMOF5WCwhMOAi1wUjh2BTTC4xNSYW8YFBaNL3j1vrL1dgkQ+vPmEymbBmzRqsW7cOtbW1iIuLwy9+8Qvcfffd8PPzkzoeEdkpEB+N2RADw1eNhxvjPfATD8KIQhemo7FZECL83arKEGENDOJ8cAqMZ2FTTC4hF2S4JTcfKw7sHbdWm5WNIKXSBancy5EjR/DEE09g9+7dUCqVWLBgAV5//XVkZ2dLHY2IHEZEoPDRuFWnplMECB/CKLIpdgd+2H/OP2ZOpxAa4Sd+CyOmOzkVOZJdTbFWq/0hgMcA5AKYqdPp9jsiFHmnZWkZaO7vxxvlJWPWXJaSijvyJ7owlbQsFgteeuklvPbaa2hpaUFmZiZWr16Nq6++WupoROQEAnogF1qsrleiwolpyBZKHLOxvoJNsYex90rxEQDXAbDu/QTyeT+bOAlTomPwXmUFdjY1woLhG+oKo2NwXUYWLkpIgiB4//y5hoYGPP7449i0aRNkMhmuuOIKPPzww1xXmMjr2XoTMW86dh/82nk7u5pinU5XBgBardYxacgnzIiNw4zYOAyYTOg1GhGkVPrMdImNGzfi2WefRWVlJTQaDR5++GHcfvvtXEGCyEeICIFZDIdc6LKq3gTfvb/C3dj6teDXzvNwTjFJJkChQIAPbD3c1taG5cuXY+PGjdDr9ZgzZw5eeeUV5OXlSR2NiFxOjgFciWC8ZVX1gMilvdyFAXNgFiMhFzrGrTWLahgw2wWpyJHG7Ui0Wu1mAKO9p/uQTqf70NoTabXaOwHcCQA6nY4bDVhJoVBwrGzgLuNlsViwbt06rFq1CqWlpVCr1bjrrrvw0EMPITDQPdYddZex8hQcL+txrMZhuQNi96cQxM5zlonyfIREXIkQwfsvHlhL8tfW4C8A/VMQMfpCeSOPB/0can/pp8NJPl4eRhAdsEmCVqvdBuA+G260ExsbG+0+ry9Qq9Voa2uTOobHkHq8jh07hieffBJff/01RFHErFmz8Pvf/x7Tpk2TLNNYpB4rT8Pxsh7HanwKlCNCeGDMaRRDYgY6xWdhQaSLk7k36V9bIoKF1xAsvDlmRZ94C/rEn8Id1peWfrzcg0ajAaz4gvDPTyI7WSwW/O1vf8PatWvR1NSElJQUPPnkk7jxxhs5V5iIRmVCDtrFtQgQP0aw4lMIluEVKYbEDOjFqzCASwH4SxuSRiGgT7wdBnEGAoUP4I/tEIQhiKISg5gHvXgthlAgdUg6T/YuyXYtgNUAogF8otVqD+p0uksdkozIzTU1NeGxxx7Dpk2bAACXXnopHnroISQlJUmcjIg8gQWR6MetCAj/LdraGjC80YNv3HTs6YZQgG6xAN2wQBANEKECN+rwfPauPvEBgA8clIXII3z++edYsWIFKioqEBcXhwceeAA/+9nPeFWYiOygkjoAnRcZRARIHYIchNMniKzQ3d2NFStW4IMPPkBfXx9mzZqFzz//HAUFfJuMiIjIG7ApJjqHTZs2YdWqVSguLkZYWBh+9KMf4b777kNwcLDU0YiIiMiB2BQT/Ye6ujo8/fTT2Lx5MwwGAyZPnoz169dj3rx5UkcjIiIiJ2FTTIThFSTWrl2LV199FXV1dYiPj8c999yD//7v/4afn5/U8YiIiMjJ2BSTT2tubsaTTz6Jzz77DBaLBYsWLcJbb72FjIwMqaMRkYvIUQM/lAIwwwwNjJgCriRA5HvYFJNP2rx5M5YvX46jR48iNjYWv/nNb3DXXXdB4QPbThPRMAXKESL8HSqh6IzHTWI8+sUbMYAr4Q4bMBCRa7ADIJ/R19eH5557DjqdDr29vZgxYwY++eQTTJ48WepoRORifihChPAABMEAUQSE03pfhdCEMOE5yMVG9Il3SheSiFyKTTF5vT179mDFihXYu3cvQkJCcP311+N3v/sdQkNDpY5GRBIQoEe48EcIgmH441EuBosiECz8E0ZxEoyY7eKERCQFNsXklTo6OrBixQp89NFH6O3tRU5ODtauXYslS5ZIHY2IJOaPLyATes9Zc6pRDhLeg1FkU0zkC9gUk9ewWCx49dVX8fzzz+P48eOIiorCDTfcgHvvvZdXhYlohL+w1ao6UQRUwj4IYg9E8GcIkbdjU0wer7m5GX/605/w6aefwmQy4cILL8TLL7/M3eaIaFQydFlVd+pqsQw9MLMpJjonOeqhRCkEmGGCBkMogKet4sKmmDzWli1bsHz5cpSXlyM6Ohq//OUv8cc//hE9PT1SRyMiNyYiyLq6727AExHo5EREnkuBCoQIa6AS9p/xuElMQL94EwZwuUTJbMemmDyKXq/HypUrsX79enR3d2P69On4+OOPUVhYCADcaIOIxmUQ58BPKBm3ThAAo5gDCyJdkIrI8yhRjAjhd5CNuopLA8KEFd+t4nKHdCFtwKaYPEJxcTH+9Kc/YefOnQgKCsK1116LBx98kHOFichmA7gCweIbEIShMWtO/YLXi9e4MBmR5xCgR4TwCGTjruKyDkZxIoyY5eKEtmNTTG5Lr9fjhRdewNtvv42Ojg5kZGTglVdewdKlS6WORkQezIJI9Ii/QZiwYswaQQAGxfkYxGIXJiPyHP7YAplw7umK36/i8j6MIptiIpt9+OGHeOmll1BaWorAwEBcddVVeOCBB6BWq6WORkReYgCXQ7T4I0R4CXKh7YznRFEJPa5Cr/hzAHJpAhK5OX/hS6vqhldx2QNB7IWIECensg+bYnILbW1tePrpp/HJJ59gcHAQ06dPx1tvvYUFCxZIHY2IvNQgFmJQnAeVuPO7OcYmmMUEDGARRIRJHY/IrcnQaVXd96u4dMPMpphobNu2bcPTTz+NsrIyREVF4Y477sA999wDf39/qaMRkU9QwID5MIjzpQ5C5FGsXZXFk1ZxYVNMLjc4OIiVK1firbfeQnd3NwoLC/Hhhx9i2rRpUkcjIiIiKwyv4lI6bp0gAENitkes4sKmmFzm6NGjePzxx7F9+3YEBATguuuuwx/+8AeuIEFERORhvl/FxTRmjaet4sKmmJzKZDLhlVdeweuvv47m5makpaVh9erVuOYaz/gGISIiorNZEIUe8X8QJjw36vOnGuJB8QIMYImL050fNsXkFNu2bcOqVatw4MABKBQKXHHFFXjooYcQHx8vdTQiIiJygAEsPW0Vl/+88U6JfnEpesVfwFNWcWzIOcwAABM+SURBVGFTTA7T2dmJFStW4MMPP0Rvby8mTpyIF198EVdddZXU0YiIiMgJBnEJBsUF8Bd3QHnGKi6LISJc6ng2YVNMdtuxYweefPJJlJSUIDw8HD/60Y9w3333ITg4WOpoRERE5HTK75Y4XCh1ELuwKabzYjQa8Ze//AXr1q1DV1cXJk+ejA8++AAzZsyQOhoRERGRzdgUk02qqqrw2GOP4auvvoJKpRpZQSIsjAvdExERkediU0zjslgsWLt2LdasWYOGhgakpKTg+eefxw9/+EOpoxER0ZhEKFEMPxyBIAzBLMZhEAs8YhMFIimwKaYxFRcX49lnn8U333wDURSxePFibNiwASkpKVJHIyKic/DDAYQIq6EUqr9/UABCxL9Cj2vRJ94OtgBEZ7LrO0Kr1T4LYBkAI4AqAD/V6XRdjghG0ujr68PKlSvx7rvvoqOjAxkZGXj88cdx0003QSaTSR2PiIjGocJOhAuPQBDMZz0nEwYQjH9CjkZ0i38EwJ/rRKfY+93wBYCJOp1uEoAKAA/aH4mksGfPHlx55ZXIycnB+vXrcemll+LQoUP46quvcMstt7AhJiLyAAL0CBOeHrUhPkUUgQBhGwLwuQuTEbk/u64U63S6Tad9uBvAD+yLQ65kNBrxwgsv4B//+Ac6OjqQn5+P9evXY968eVJHIyKi8+CPLyAT+s9ZIwjD/xsovI8B8XIAgvODEXkAR04ouh3AOw48HjlJdXU1Hn30UWzduhUqlQpXX301HnroIUREREgdjYiI7OAv/H979x4kVXmncfx7ZgYYrooSNAibaElBLBKjWF7KGOKyIRoSWdf1ja5uomZrkqqVrFkVCl0vBBTIUGpiSCEJKImUyRuDhVFRI1lN/nGjKFldY/AS2UREC5DrAEMzZ//owUUFppuZ6dM95/uporC7z3Q//BhmHk+/857flXRcmkKv5FXq0rdp4+huTiXVhg5LcQjhCdjvv5jrY4zL2o+5HigASw7yPE1AE0CMkSFDhhxS4LxpaGjoklnt3UGiubmZN954g2OPPZZ58+Zx+eWXd0HK6tFV88oDZ1Ue51U6Z1WerpxX/eadcOCVE+/Ze7b4iMMboKF2/q783CqP8ypPkqZpp54ghHAZ8A1gfIyxpcQPS9euXdup182LIUOGsH79+kP++DfeeINZs2bxxBNPUCgUOPvss7nxxhs57rjjujBl9ejsvPLEWZXHeZXOWZWnK+c1OLmWPskzHR6XpsVi/E5bpI2hXfLaleDnVnmcV9GwYcOghHVCnd194hxgCjCujEKsbtba2soPf/hDlixZwtq1axk+fDjf/va3aWpqonfv3lnHkyR1k53puJJKcZJAa/qJmirEUnfr7JriHwB9gF+HEACejjF+s9OpdEief/55pk+fzsqVK2lsbGT8+PHcf//97issSTmxk/EMTO+iLtl6wGP2niVuSc+vYDKp+nV294njuyqIDk2hUOD222/n3nvvZcOGDYwePZof/ehHnHPOOVlHkyRVWEpfNqU3MZhpJMnu9wrwvpIEdqRfYCefzyakVKW8nE2N2ruDxJNPPkmvXr2YNGkS06ZNc0G9JOVcK6ewMb2dgfyA3snL73usLR3E9vRCtnMJbsUmvZ+luMbEGLnjjjtYs2YNI0aM4NZbb+Xiiy/24hqSpPfsZgwb0/k0pH+iNy+QsJsCH2UXZ1Bc9SjpgyzFNWDdunXceuutLF++nF27djFu3DgWL17MyJEjs44mSapiBUZRYFTWMaSaYCmuUm1tbdx7773cfffdrF69mqFDh9LU1MTkyZNpbGzMOp4kSVKPYimuMi+99BLTp0/n6aefpq6ujs9+9rPMmzePE044IetokiRJPZaluAoUCgXuvPNOfvrTn/L2229z/PHHM3fuXC644AKGDh3qxtuSJEndzFKcoTVr1nDzzTfzm9/8hoaGBiZOnMh1113H0Ud7HXpJkqRKshRnYOnSpdx22238+c9/ZsSIEcyYMYNLL73UHSQkSZIyYimukI0bNzJr1iyWLVvGzp07+cxnPsPChQsZNcqfCpYkScqapbibPfTQQ3z/+9/npZde4ogjjuBrX/saV199tTtISJIkVRFLcTd4/fXXmTFjBk899RSFQoGTTz6ZZcuWMXbs2KyjSZIkaT8sxV2kUCgwf/58Fi9ezNq1axkxYgRTp07l61//Og0NjlmSJKma2dY6ac2aNXznO99hxYoV1NXVMWHCBO6//34+9rGPZR1NkiRJJbIUH6IHH3yQuXPn8tprr3HMMcdwww03cPnll7uDhCRJUg2yFJdh8+bNzJ49m6VLl9LS0sKZZ57JggULGD16dNbRJEmS1AmW4hI89dRTNDc3s2rVKg477DAuueQSrrnmGvr165d1NEmSJHUBS/EBrFu3jtmzZ7N8+XK2b9/OmDFjuO+++zjrrLOyjiZJkqQuZineR1tbG4sWLWLRokWsWbOGoUOHcsUVVzB58mTPCkuSJPVglmLgzTffZMaMGTz++OOkacq4ceO4++67vdqcJEmqgJQ6NgG7aeMwoE/WgXIp16X44Ycf5rvf/S6vvvoqw4YNY8qUKTQ1NbmDhCRJqoCd9OMh+iXLaEj+AkCa9mEH42lJ/5ECx2WcL19yV4q3bdvGnDlz+MUvfsH27ds544wzuOuuu9xBQpIkVUzCZo5IrqVXsvr99ye76Mcj9OVxNqX/wS4+l03AHMpNKV65ciUzZ87kmWeeYdCgQVx00UVMmTLFtcKSJKnCUg5Pbv5QId5XkhQ4nJlsSI+mgCfuKqFHl+ItW7bQ3NzMAw88wKZNmxg1ahSLFy9m/PjxWUeTJEk51YsX6ZM8T5pCkhz4uCQp0J+fszm9qXLhcqzHleK2tjZijCxYsIDVq1czcOBAzj//fK699loGDx6cdTxJkpRzfZOHgYMXYoA0hUZ+yxY2k3JYBZLlW48pxW+99Ra33HILy5cvp1AocNpppzFr1ixOO+20rKNJkiS9p4E3SzquWJr3UJ++TcFS3O1qvhQ/+uijzJkzh9WrV3PUUUcxefJkrrzyShoaav6PJkmSeqCU+jI/wk5TCTU55W3bttHc3EyMkW3btnHqqafy2GOPMWbMmKyjSZIkHdRuPkEfVpV0bFs6kALHdHMiQY2V4hdffJGbbrqJ3//+9wwYMIAQAlOnTnUHCUkqS6H9QgEpbQymxr4VSDVvR/pl+vMzIO1wXfEOzsWLeVRGp74ShhBmAJOANuAd4LIY49quCLZXa2srd9xxB0uWLGHDhg2MHDmShQsXMmHChK58GUnq8erYQL/kAfryMPXJuwC0pYfRwhdpSS+gjSEZJ5TyYQ/DaOFC+ifxoDtQ7EmHsD39SmXD5VhnTw80xxhvAAghfAu4Efhmp1NRXCs8b948Vq1aRWNjI+eddx5Tp05l6NChXfH0kpQrDbzO4OQa6pON77u/LtnMAO6jL8t5N22mwMiMEkr5sjX9BlCgf7J0v48X0mG8m86mjSMrGyzHOlWKY4xb9rnZH0g783zr169n1qxZ/OpXv2LHjh2ceOKJLFiwgHPPPbczTytJuZawjcHJlA8V4n3VJ5sYzFTWp/eQMqiC6aS8qmdr+i12pF+ib/IgvfkDCbvZw9HsSM9lJ2cBvbMOmSudXkgWQrgF+CqwGTj7UJ5jxYoVzJw5k1deeYUjjzySK664gquuuorGxsbOxpOk3OvLo9Qn6zs8rj7ZSN/0EVq4qAKpJAEUOI6t6VVZxxCQpOnBT+6GEJ4Ajt7PQ9fHGJftc9w0oDHGuN/LroQQmoAmgBjj2E2bNjF9+nTuuecetmzZwumnn87cuXMZO3bsIf9heqKGhgYKhULWMWqG8yqdsypPLc+rfvP5JHteKenYtO7j7Dn8oU69Xi3PKgvOq3TOqjzOq6h3794AHfxIYwmluFQhhL8BHokxlrIvWlpfX0///v258MILmTp1KgMGDOiSHD3NkCFDWL++4zM8KnJepXNW5anleR2VTCBJWks6Nk3reDtdQQnfPw6olmeVBedVOmdVHudVNGzYMCjhi1pnd58YGWPce/phEvByqR87f/58Jk6c2JmXlySVIKW+jIpb7kUFJKln6Oya4tkhhFEUt2RbQxk7T1iIJakyihcKeK7kYztzlliSalVnd5+4oKuCSJK6R0s6iT7JcwfdD3XvYy3ppMqGk6QqUZd1AElS99rFmexKTyJJiuX3g/YW4tb0k+xkXOUDSlIVsBRLUo/XwKZ0JrvSU/Z7pjhJYFd6Eu+mt+IlnyXllV/9JCkHUvrzbtpM73QVfZMH6cXrQMpujmNH+mVaORnXEkvKM0uxJOVGQisn0ZqelHUQSao6Lp+QJElS7lmKJUmSlHuWYkmSJOWepViSJEm5ZymWJElS7lmKJUmSlHuWYkmSJOWepViSJEm5ZymWJElS7lmKJUmSlHuWYkmSJOWepViSJEm515B1AElS3qX04mXq+QtQz25GsYfhWYeSlDOWYklSZvrwJAOSn9Aref199+9KT2Fr+i8UGJ1RMkl5YymWJGWiHz9jUN38D92fptAneZbe/DfvprNoZWwG6STljWuKJUkV14sXGFQ3nzT98GNJsvf3Vg5PbiRha2XDScolS7EkqeL6Jb8E/r8AH0hdsp2+PFaBRJLyzlIsSaqwXTTyu/2eJd6fxmRF98aRJFxTLEmqsDq2kSR7Sj6+no3dmEaSijxTLEmqqJS+xd9LPFPcRr9uTCNJRZZiSVJFpfSjNf1kh+uJ99rF6d0bSJKwFEuSMtCS/j3Q8dniNK1jR3peBRJJyjtLsSSp4nZyNjvTcSTJwYvx1vQb7OGjlQsmKbcsxZKkDNSxKb2B7ekFQK8PPdqWDmRz27/TwlcqH01SLrn7hCQpIw1sTSeznUtpTB+nIfkrKfXsTkezk7OBPlkHlJQjXVKKQwhXA3OBj8QY13fFc0qS8qGNwcUzwiXuRiFJ3aHTyydCCCOACcD/dj6OJEmSVHldsab4dmAK/j++JEmSalSnSnEIYRLwZozxD12UR5IkSaq4DtcUhxCeAI7ez0PXA9dRXDrRoRBCE9AEEGNk2LBhZcTMN2dVHudVOmdVHudVOmdVHudVOmdVHudVuiQt9TqbHxBC+CSwAmhpv2s4sBY4Nca4roOPfTbGeMohvXDOOKvyOK/SOavyOK/SOavyOK/SOavyOK/yHPLuEzHGF4Che2+HEN4ATnH3CUmSJNUaL94hSZKk3Ouyi3fEGD9exuELuup1c8BZlcd5lc5Zlcd5lc5Zlcd5lc5Zlcd5leGQ1xRLkiRJPYXLJyRJkpR7XbZ84lB5ieiOhRBmAJOANuAd4LIY49psU1WvEEIz8GWgFXgNuDzGuCnbVNUphHAhcDPwCYo7xzybbaLqE0I4B/geUA/8OMY4O+NIVSuEsAj4EvBOjHFM1nmqWfvVYH8CHEXx4lcLYozfyzZV9QohNAK/BfpQ7C73xxhvyjZVdQsh1APPUryexJeyzlMLMj1T7CWiS9YcY/xUjPHTwEPAjVkHqnK/BsbEGD8FrAamZZynmr0I/APFbzb6gPZvKvOAc4ETgItDCCdkm6qq3QOck3WIGlEAro4xngCcDvyrn1sHtQv42xjjicCngXNCCKdnnKna/Rvwx6xD1JKsl094iegSxBi37HOzP87roGKMj8cYC+03n6a4h7b2I8b4xxjjn7LOUcVOBV6NMb4eY2wFfkbxXRvtR4zxt8DGrHPUghjjWzHG59r/eyvF8nJMtqmqV4wxjTFua7/Zq/2X3wsPIIQwHJgI/DjrLLUks+UT+14iOoSQVYyaEUK4BfgqsBk4O+M4teQK4OdZh1DNOgb4yz63/wqcllEW9VAhhI8DJwH/lXGUqtb+zs1K4HhgXozReR3YHRRPOg7MOkgt6dZS3FWXiM6Dg80qxrgsxng9cH0IYRpwJZDrtVQdzav9mOspvkW5pJLZqk0ps5KUjRDCAOCXwFUfeFdQHxBj3AN8OoRwOPBACGFMjPHFrHNVmxDC3nX9K0MIn8s6Ty3p1lIcY/y7/d3ffonoY4G9Z4mHA8+FEDq8RHRPdaBZ7ccS4BFyXoo7mlcI4TKKP/AzPsaY67fYyvjc0oe9CYzY5/bw9vukTgsh9KJYiJfEGJdmnadWxBg3hRD+k+L6dUvxh50JnBdC+CLQCAwKIdwbY7w041xVL5PlE14iujwhhJExxlfab04CXs4yT7Vr3y1gCjAuxtiSdR7VtGeAkSGEYymW4YuAf8o2knqCEEICLAT+GGO8Les81S6E8BFgd3sh7gt8HpiTcayqFGOcRvsPmLefKb7GQlyazLdkU0lmhxBGUdySbQ3wzYzzVLsfUNy259ft70Q8HWN0ZvsRQjgfuBP4CPBwCGFVjPELGceqGjHGQgjhSuAxiluyLYox/k/GsapWCOE+4HPAkBDCX4GbYowLs01Vtc4E/hl4IYSwqv2+62KMj2SYqZp9FFjcvq64DogxxocyzqQexivaSZIkKfey3pJNkiRJypylWJIkSblnKZYkSVLuWYolSZKUe5ZiSZIk5Z6lWJIkSblnKZYkSVLuWYolSZKUe/8H1rZMPP4F5VMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12, 6))\n",
    "for i in range(3):\n",
    "    plt.contour(xx, yy, p_test[:,i].reshape(200,200), [0.5], colors='k', linewidths=1)\n",
    "plt.scatter(X[:,0], X[:,1], 100, np.argmax(Y, 1), lw=2, cmap=plt.cm.viridis);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That concludes the new model example and this notebook. You might want to convince yourself that the `LinearMulticlass` model and its parameters have all the functionality demonstrated above. You could also add some priors and run Hamiltonian Monte Carlo using the HMC optimizer `gpflow.train.HMC` and its `sample` method. See the sparse_MCMC notebook for details of running the sampler."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}