gluonts_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">GluonTS</span> Deep Learning for Forecasting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. What is GluonTS?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GluonTS is a Python toolkit for probabilistic time series modeling, built around Apache MXNet (incubating).\n",
    "\n",
    "GluonTS is especially suited for working with multiple time series datasets. It provides utilities for loading and iterating over time series datasets, state of the art models ready to be trained, as well as building blocks to define your own models and quickly experiment with different solutions.\n",
    "\n",
    "The toolkit targets scientists and engineers who want to tweak algorithms or build and experiment with their own models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Web page: https://gluon-ts.mxnet.io/\n",
    "* GitHub repo: https://github.com/awslabs/gluon-ts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Exercise 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install --upgrade --quiet gluonts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gluonts\n",
    "gluonts.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Package overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Main modules:\n",
    "* `gluonts.dataset` -- abstractions and utilities to manipulate datasets\n",
    "* `gluonts.model` -- pre-implemented models, plus abstractions and utilities to help you define your own\n",
    "* `gluonts.evaluation` -- model evaluation tools, i.e., to compute accuracy metrics and compare models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other useful modules:\n",
    "* `gluonts.distribution` -- several types of parametric probability distributions, to help you define probabilistic models\n",
    "* `gluonts.kernels` -- kernel functions, to support developing e.g. Gaussian-processes-based models\n",
    "* `gluonts.trainer` -- provides a default `Trainer` class, exposing several training options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Quick start guide"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Third-party imports\n",
    "%matplotlib inline\n",
    "import mxnet as mx\n",
    "from mxnet import gluon\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Download a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GluonTS comes with a number of publicly available datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.dataset.repository.datasets import get_dataset, dataset_recipes\n",
    "from gluonts.dataset.util import to_pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Available datasets: {list(dataset_recipes.keys())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download one of the built-in datasets, simply call `get_dataset` with one of the above names. GluonTS can re-use the saved dataset so that it does not need to be downloaded again: simply set `regenerate=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = get_dataset(\"m4_hourly\", regenerate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the datasets provided by GluonTS are objects that consists of three main members:\n",
    "\n",
    "- `dataset.train` is an iterable collection of data entries used for training. Each entry corresponds to one time series\n",
    "- `dataset.test` is an iterable collection of data entries used for inference. The test dataset is an extended version of the train dataset that contains a window in the end of each time series that was not seen during training. This window has length equal to the recommended prediction length.\n",
    "- `dataset.metadata` containts metadata of the dataset such as the frequency of the time series, a recommended prediction horizon, associated features, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(dataset.train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are these datasets made of?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the first time series in the training set\n",
    "train_entry = next(iter(dataset.train))\n",
    "pprint(train_entry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the first time series in the test set\n",
    "test_entry = next(iter(dataset.test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the timeseries to pandas series objects to make plotting easier\n",
    "test_series = to_pandas(test_entry)\n",
    "train_series = to_pandas(train_entry)\n",
    "\n",
    "fig, ax = plt.subplots(2, 1, sharex=True, sharey=True, figsize=(10, 7))\n",
    "\n",
    "train_series.plot(ax=ax[0])\n",
    "ax[0].grid(which=\"both\")\n",
    "ax[0].legend([\"train series\"], loc=\"upper left\")\n",
    "\n",
    "test_series.plot(ax=ax[1])\n",
    "ax[1].axvline(train_series.index[-1], color='r') # end of train dataset\n",
    "ax[1].grid(which=\"both\")\n",
    "ax[1].legend([\"test series\", \"end of train series\"], loc=\"upper left\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Length of forecasting window in test dataset: {len(test_series) - len(train_series)}\")\n",
    "print(f\"Recommended prediction horizon: {dataset.metadata.prediction_length}\")\n",
    "print(f\"Frequency of the time series: {dataset.metadata.freq}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Train an existing model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GluonTS comes with a number of pre-built models. All the user needs to do is configure some hyperparameters. The existing models focus on (but are not limited to) probabilistic forecasting. Probabilistic forecasts are predictions in the form of a probability distribution, rather than simply a single point estimate.\n",
    "\n",
    "We will begin with GluonTS's pre-built feedforward neural network estimator, a simple but powerful forecasting model. We will use this model to demonstrate the process of training a model, producing forecasts, and evaluating the results.\n",
    "\n",
    "GluonTS's built-in feedforward neural network (`SimpleFeedForwardEstimator`) accepts an input window of length `context_length` and predicts the distribution of the values of the subsequent `prediction_length` values. In GluonTS parlance, the feedforward neural network model is an example of `Estimator`. In GluonTS, `Estimator` objects represent a forecasting model as well as details such as its coefficients, weights, etc.\n",
    "\n",
    "In general, each estimator (pre-built or custom) is configured by a number of hyperparameters that can be either common (but not binding) among all estimators (e.g., the `prediction_length`) or specific for the particular estimator (e.g., number of layers for a neural network or the stride in a CNN).\n",
    "\n",
    "Finally, each estimator is configured by a `Trainer`, which defines how the model will be trained i.e., the number of epochs, the learning rate, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator\n",
    "from gluonts.trainer import Trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code block below creates the feedforward network estimator. There are many more available options than the ones we have used here, which are described in the documentation. The `SimpleFeedForwardEstimator` documentation is available at:   \n",
    "https://gluon-ts.mxnet.io/api/gluonts/gluonts.model.simple_feedforward.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = SimpleFeedForwardEstimator(\n",
    "    # required\n",
    "    prediction_length=dataset.metadata.prediction_length,\n",
    "    freq=dataset.metadata.freq,\n",
    "    # optional\n",
    "    num_hidden_dimensions=[10],    \n",
    "    context_length=100, \n",
    "    trainer=Trainer(\n",
    "        ctx=\"cpu\", \n",
    "        epochs=10, \n",
    "        learning_rate=1e-3,\n",
    "        batch_size=32,\n",
    "        num_batches_per_epoch=100,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After specifing our estimator with all the necessary hyperparameters we can train it using our training dataset `dataset.train` by invoking the `train` method of the estimator. The training algorithm returns a fitted model (or a `Predictor` in GluonTS parlance) that can be used to construct forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = estimator.train(dataset.train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a predictor in hand, we can now predict the last window of the `dataset.test` and evaluate our model's performance.\n",
    "\n",
    "GluonTS comes with the `make_evaluation_predictions` function that helps in the process of prediction and model evaluation. Roughly, this function performs the following steps:\n",
    "\n",
    "- Removes the final window of length `prediction_length` of the `dataset.test` that we want to predict\n",
    "- The estimator uses the remaining data to predict (in the form of sample paths) the \"future\" window that was just removed\n",
    "- The module outputs the forecast sample paths and the `dataset.test` (as python generator objects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.evaluation.backtest import make_evaluation_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_it, ts_it = make_evaluation_predictions(\n",
    "    dataset=dataset.test,  # test dataset\n",
    "    predictor=predictor,  # predictor\n",
    "    num_eval_samples=100,  # number of sample paths we want for evaluation\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the generators to lists\n",
    "forecasts = list(forecast_it)\n",
    "tss = list(ts_it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can examine the first element of these lists (that corresponds to the first time series of the dataset). Let's start with the list containing the time series, i.e., `tss`. We expect the first entry of `tss` to contain the (target of the) first time series of `dataset.test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(tss[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tss[0].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entries in the `forecast` list are a bit more complex. They are objects that contain all the sample paths in the form of `numpy.ndarray` with dimension `(num_samples, prediction_length)`, the start date of the forecast, the frequency of the time series, etc. We can access all these information by simply invoking the corresponding attribute of the forecast object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of sample paths: {forecasts[0].num_samples}\")\n",
    "print(f\"Dimension of samples: {forecasts[0].samples.shape}\")\n",
    "print(f\"Start date of the forecast window: {forecasts[0].start_date}\")\n",
    "print(f\"Frequency of the time series: {forecasts[0].freq}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Forecast` objects have a `plot` method that can summarize the forecast paths as the mean, prediction intervals, etc. The prediction intervals are shaded in different colors as a \"fan chart\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecasts[0].plot(color='g')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way you can of course plot predictions alongside actual data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tss[0][-150:].plot(figsize=(10,5))  # plot the time series\n",
    "forecasts[0].plot(color='g')  # plot the predicted distribution\n",
    "plt.grid(axis='both', which='both')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do calculations to summarize the sample paths, such computing the mean or a quantile for each of the 48 time steps in the forecast window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Mean of the forecast:\\n {forecasts[0].mean}\")\n",
    "print(f\"0.5-quantile (median) of the forecast:\\n {forecasts[0].quantile(0.5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also evaluate the quality of our forecasts numerically. In GluonTS, the `Evaluator` class can compute aggregate performance metrics, as well as metrics per time series (which can be useful for analyzing performance across heterogeneous time series)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.evaluation import Evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])\n",
    "agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(dataset.test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate metrics aggregate both across time-steps and across time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(agg_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual metrics are aggregated only across time-steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_metrics.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_metrics.plot(x='MSIS', y='MASE', kind='scatter')\n",
    "plt.grid(which=\"both\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary**: "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# download a dataset\n",
    "dataset = get_dataset(\"m4_hourly\", regenerate=False)\n",
    "\n",
    "# choose a pre-built model\n",
    "estimator = SimpleFeedForwardEstimator(\n",
    "    prediction_length=dataset.metadata.prediction_length,\n",
    "    freq=dataset.metadata.time_granularity,\n",
    "    num_hidden_dimensions=[10],    \n",
    "    context_length=100, \n",
    "    trainer=Trainer(ctx=\"cpu\", \n",
    "                    epochs=5, \n",
    "                    learning_rate=1e-3, \n",
    "                    num_batches_per_epoch=100\n",
    "                   )\n",
    ")\n",
    "\n",
    "# train the model on the train dataset\n",
    "predictor = estimator.train(dataset.train)\n",
    "\n",
    "# create forecasts\n",
    "forecast_it, ts_it = make_evaluation_predictions(\n",
    "    dataset=dataset.test,  # test dataset\n",
    "    predictor=predictor,  # predictor\n",
    "    num_eval_samples=100,  # number of sample paths we want for evaluation\n",
    ")\n",
    "\n",
    "# evaluate the forecasts\n",
    "evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])\n",
    "agg_metrics, item_metrics = evaluator(ts_it, forecast_it, num_series=len(dataset.test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use a different model:\n",
    "- Replace the `SimpleFeedForwardEstimator` with a `DeepAREstimator` \n",
    "- Configure (at least) the required hyperparameters (use the same `Trainer` as in `SimpleFeedForwardEstimator`)\n",
    "- Set `solved = True` and check the result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DeepAREstimator` documentation:  \n",
    "https://gluon-ts.mxnet.io/api/gluonts/gluonts.model.deepar.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# place your code here\n",
    "from gluonts.model.deepar import DeepAREstimator\n",
    "\n",
    "estimator = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_solution(predictor, test_ds, plot_idx=0):\n",
    "    forecast_it, ts_it = make_evaluation_predictions(\n",
    "        dataset=test_ds,  # test dataset\n",
    "        predictor=predictor,  # predictor\n",
    "        num_eval_samples=100,  # number of sample paths we want for evaluation\n",
    "    )\n",
    "\n",
    "    forecasts = list(forecast_it)\n",
    "    tss = list(ts_it)\n",
    "    \n",
    "    tss[plot_idx][-150:].plot(figsize=(10,5))  # plot the time series\n",
    "    forecasts[plot_idx].plot(color='g')  # plot the predicted distribution\n",
    "    plt.grid(axis='both', which='both')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "solved = False\n",
    "\n",
    "if solved:\n",
    "    predictor = estimator.train(dataset.train)\n",
    "    evaluate_solution(predictor, dataset.test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to underfit `DeepAREstimator`:\n",
    "- Add a large `dropout_rate` \n",
    "- Decrease significantly the network size (`num_layers` and `num_cells`)\n",
    "- Decrease `context_length`\n",
    "- Set `solved = True` and check the result\n",
    "\n",
    "Try to plot more time series: use the `plot_idx` argument of `evaluate_solution`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# place your code here\n",
    "estimator = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "solved = False\n",
    "\n",
    "if solved:\n",
    "    predictor = estimator.train(dataset.train)\n",
    "    evaluate_solution(predictor, dataset.test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Custom datasets and models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Datasets "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, a dataset should satisfy some minimum format requirements to be compatible with GluonTS. In particular:\n",
    "- it should be an iterable collection of data entries (dictionaries) \n",
    "- each entry corresponds to one time series\n",
    "- each entry should have at least a `target` field, which contains the actual values of the time series, and a `start` field, which denotes the starting date of the time series\n",
    "- there are optional fields that define possible features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's revisit the `m4_hourly` dataset that we have already downloaded. We can examine the first entry of the dataset and see what is the underlying structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_entry = next(iter(dataset.train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(train_entry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_entry.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start field: starting date of the time series\n",
    "train_entry['start']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# target field: contains the time series values\n",
    "train_entry['target'][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feat_static_cat: contains time independent (static) categorical features\n",
    "train_entry['feat_static_cat']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets provided by GluonTS are already in the appropriate format and they can be used without any further processing steps. However, custom datasets need to be converted. Fortunately this is an easy task since the only requirements is that it is iterable and that is has a `target` and a `start` field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, suppose your dataset is in the form of a `numpy.array` where the initial time stamp is given as a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_series = 100  \n",
    "period = 24\n",
    "num_steps = 10 * period  \n",
    "prediction_length = 24\n",
    "freq = \"1H\"\n",
    "\n",
    "pattern = np.sin(np.tile(np.linspace(-np.pi, np.pi, period), int(num_steps / period)))\n",
    "noise = np.random.normal(loc=1, scale=0.3, size=(num_series, num_steps))\n",
    "\n",
    "target = pattern + noise\n",
    "\n",
    "start = \"01-01-2019 00:00:00\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(target[0])\n",
    "plt.grid(which=\"both\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can split your dataset and make it available to GluonTS with just two lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.dataset.common import ListDataset\n",
    "\n",
    "# train dataset: cut the last window of length \"prediction_length\", add \"target\" and \"start\" fields\n",
    "train_ds = ListDataset(\n",
    "    data_iter=[{'target': x, 'start': start} for x in target[:, :-prediction_length]],\n",
    "    freq=freq\n",
    ")\n",
    "\n",
    "# test dataset: use the whole dataset, add \"target\" and \"start\" fields\n",
    "test_ds = ListDataset(\n",
    "    data_iter=[{'target': x, 'start': start} for x in target],\n",
    "    freq=freq\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! Now we can simpy use `train_ds` and `test_ds` instead of `dataset.train` and `dataset.test`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Exercise 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following publicly available time series\n",
    "* https://raw.githubusercontent.com/numenta/NAB/master/data/realTraffic/occupancy_6005.csv\n",
    "* https://raw.githubusercontent.com/numenta/NAB/master/data/realTraffic/occupancy_t4013.csv\n",
    "\n",
    "Try to:\n",
    "1. Read in the data (hint: you can use `pandas.read_csv`)\n",
    "2. Create a `ListDataset` composed of both time series (what is the frequency of the data?)\n",
    "3. Slice out a training portion (you can choose where in time to do that)\n",
    "4. Train a model with `prediction_length=48`\n",
    "5. Plot the forecasts & evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Probabilistic forecasting with a feedforward neural network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For creating our own forecast model we need the following basic components:\n",
    "- a training network\n",
    "- a prediction network\n",
    "- an estimator that specifies any data processing and uses the networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training network can be arbitrarily complex but it should follow some basic rules:\n",
    "- It should have a `hybrid_forward` method that defines what should happen when the network is called    \n",
    "- Its `hybrid_forward` should return a **loss** based on the prediction and the true values. The loss for probabilistic forecasting is usually the negative log-probability of the chosen distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How can our model learn a distribution?\n",
    "\n",
    "In order to learn a distribution we need to learn its parameters. For example, in the simple case where we assume a Gaussian distribution, we need to learn the mean and the variance that fully specify the distribution.\n",
    "\n",
    "Each distribution that is available in GluonTS is defined by the corresponding `Distribution` class (e.g., `Gaussian`). This class defines -among others- the parameters of the distribution, its (log-)likelihood and a sampling method (given the parameters). \n",
    "\n",
    "However, it is not straightforward how to connect a model with such a distribution and learn its parameters. For this, each distribution comes with a `DistributionOutput` class (e.g., `GaussianOutput`) that makes this connection possible. \n",
    "\n",
    "The main usage of `DistributionOutput` is to take the output tensor of the model and use its last dimension as features which maps to the parameters of the distribution. For example, if the output of the network is a tensor of dimension `(a,b)`, there will be `b` features that are going to be projected and create `a` different `Distribution` objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Design choices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Create a simple training network that defines a neural network which takes as input a window of length `context_length` and outputs the subsequent window of dimension `prediction_length`\n",
    "- We need to output a `Distribution` for each time step, i.e., `prediction_length` distribution objects. Therefore the network should output `prediction_length * num_features` parameters (where `num_features` can be a hyperparameter)\n",
    "- The `DistributionOutput` should take as input a tensor of shape `(prediction_length, num_features)` to create `prediction_length` distributions. Therefore, we need to reshape the network output to `(prediction_length, num_features)` \n",
    "- We can choose to fit Gaussian distributions, i.e., use the `GaussianOutput`\n",
    "- The `hybrid_forward` method of the training network returns the negative log-probability as loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in all the tensors that we handle, there is an initial dimension that refers to the batch, e.g., the actual output dimension of our network will be `(batch_size, prediction_length * num_features)`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.distribution.distribution_output import DistributionOutput\n",
    "from gluonts.distribution.gaussian import GaussianOutput"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyProbNetwork(gluon.HybridBlock):\n",
    "    def __init__(self, \n",
    "                 prediction_length, \n",
    "                 distr_output, \n",
    "                 num_cells, \n",
    "                 num_sample_paths=100, \n",
    "                 **kwargs\n",
    "    ) -> None:\n",
    "        super().__init__(**kwargs)\n",
    "        self.prediction_length = prediction_length\n",
    "        self.distr_output = distr_output\n",
    "        self.num_cells = num_cells\n",
    "        self.num_sample_paths = num_sample_paths\n",
    "        self.proj_distr_args = distr_output.get_args_proj()\n",
    "\n",
    "        with self.name_scope():\n",
    "            # Set up a 2 layer neural network that its ouput will be projected to the distribution parameters\n",
    "            self.nn = mx.gluon.nn.HybridSequential()\n",
    "            self.nn.add(mx.gluon.nn.Dense(units=self.num_cells, activation='relu'))\n",
    "            self.nn.add(mx.gluon.nn.Dense(units=self.prediction_length * self.num_cells, activation='relu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyProbTrainNetwork(MyProbNetwork):\n",
    "    def hybrid_forward(self, F, past_target, future_target):\n",
    "        # compute network output\n",
    "        net_output = self.nn(past_target)\n",
    "\n",
    "        # (batch, prediction_length * nn_features)  ->  (batch, prediction_length, nn_features)\n",
    "        net_output = net_output.reshape(0, self.prediction_length, -1)\n",
    "\n",
    "        # project network output to distribution parameters domain\n",
    "        distr_args = self.proj_distr_args(net_output)\n",
    "\n",
    "        # compute distribution\n",
    "        distr = self.distr_output.distribution(distr_args)\n",
    "\n",
    "        # negative log-likelihood\n",
    "        loss = distr.loss(future_target)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The prediction network should be identical to the training network. Further, it should also comply to the following rule:\n",
    "- The prediction network's `hybrid_forward` should return the predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Design choices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We want the prediction network to output sample paths for each time series. To achieve this we can repeat each time series as many times as the number of sample paths and do a standard forecast for each of them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyProbPredNetwork(MyProbTrainNetwork):\n",
    "    # The prediction network only receives past_target and returns predictions\n",
    "    def hybrid_forward(self, F, past_target):\n",
    "        # repeat past target: from (batch_size, past_target_length)  \n",
    "        # to (batch_size * num_sample_paths, past_target_length)\n",
    "        repeated_past_target = past_target.repeat(\n",
    "            repeats=self.num_sample_paths, axis=0\n",
    "        )\n",
    "        \n",
    "        # compute network output\n",
    "        net_output = self.nn(repeated_past_target)\n",
    "\n",
    "        # from (batch * num_sample_paths, prediction_length * nn_features)  \n",
    "        # to (batch * num_sample_paths, prediction_length, nn_features)\n",
    "        net_output = net_output.reshape(0, self.prediction_length, -1)\n",
    "       \n",
    "        # project network output to distribution parameters domain\n",
    "        distr_args = self.proj_distr_args(net_output)\n",
    "\n",
    "        # compute distribution\n",
    "        distr = self.distr_output.distribution(distr_args)\n",
    "\n",
    "        # get (batch_size * num_sample_paths, prediction_length) samples\n",
    "        samples = distr.sample()\n",
    "        \n",
    "        # reshape from (batch_size * num_sample_paths, prediction_length) to \n",
    "        # (batch_size, num_sample_paths, prediction_length)\n",
    "        return samples.reshape(shape=(-1, self.num_sample_paths, self.prediction_length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimator should comply with the following structure:\n",
    "- It should include a `create_transformation` method that defines all the possible feature transformations and how the data is split during training\n",
    "- It should include a `create_training_network` method that returns the training network configured with any necessary hyperparameters\n",
    "- It should include a `create_predictor` method that creates the prediction network, and returns a `Predictor` object "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Predictor` defines the `predictor.predict` method of a given predictor. This method takes the test dataset, it passes it through the prediction network, and yields the predictions. You can think of the `Predictor` object as a wrapper of the prediction network that defines its `predict` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.model.estimator import GluonEstimator\n",
    "from gluonts.model.predictor import Predictor, RepresentableBlockPredictor\n",
    "from gluonts.core.component import validated\n",
    "from gluonts.trainer import Trainer\n",
    "from gluonts.support.util import copy_parameters\n",
    "from gluonts.transform import ExpectedNumInstanceSampler, Transformation, InstanceSplitter, FieldName\n",
    "from mxnet.gluon import HybridBlock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyProbEstimator(GluonEstimator):\n",
    "    @validated()\n",
    "    def __init__(\n",
    "            self,\n",
    "            prediction_length: int,\n",
    "            context_length: int,\n",
    "            freq: str,\n",
    "            distr_output: DistributionOutput,\n",
    "            num_cells: int,\n",
    "            num_sample_paths: int = 100,\n",
    "            trainer: Trainer = Trainer()\n",
    "    ) -> None:\n",
    "        super().__init__(trainer=trainer)\n",
    "        self.prediction_length = prediction_length\n",
    "        self.context_length = context_length\n",
    "        self.freq = freq\n",
    "        self.distr_output = distr_output\n",
    "        self.num_cells = num_cells\n",
    "        self.num_sample_paths = num_sample_paths\n",
    "\n",
    "    def create_transformation(self):\n",
    "        return InstanceSplitter(\n",
    "            target_field=FieldName.TARGET,\n",
    "            is_pad_field=FieldName.IS_PAD,\n",
    "            start_field=FieldName.START,\n",
    "            forecast_start_field=FieldName.FORECAST_START,\n",
    "            train_sampler=ExpectedNumInstanceSampler(num_instances=1),\n",
    "            past_length=self.context_length,\n",
    "            future_length=self.prediction_length,\n",
    "        )\n",
    "\n",
    "    def create_training_network(self) -> MyProbTrainNetwork:\n",
    "        return MyProbTrainNetwork(\n",
    "            prediction_length=self.prediction_length,\n",
    "            distr_output=self.distr_output,\n",
    "            num_cells=self.num_cells,\n",
    "            num_sample_paths=self.num_sample_paths\n",
    "        )\n",
    "\n",
    "    def create_predictor(\n",
    "            self, transformation: Transformation, trained_network: HybridBlock\n",
    "    ) -> Predictor:\n",
    "        prediction_network = MyProbPredNetwork(\n",
    "            prediction_length=self.prediction_length,\n",
    "            distr_output=self.distr_output,\n",
    "            num_cells=self.num_cells,\n",
    "            num_sample_paths=self.num_sample_paths\n",
    "        )\n",
    "\n",
    "        copy_parameters(trained_network, prediction_network)\n",
    "\n",
    "        return RepresentableBlockPredictor(\n",
    "            input_transform=transformation,\n",
    "            prediction_net=prediction_network,\n",
    "            batch_size=self.trainer.batch_size,\n",
    "            freq=self.freq,\n",
    "            prediction_length=self.prediction_length,\n",
    "            ctx=self.trainer.ctx,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = MyProbEstimator(\n",
    "    prediction_length=prediction_length,\n",
    "    freq=freq,\n",
    "    context_length=2*prediction_length,\n",
    "    distr_output=GaussianOutput(),\n",
    "    num_cells=40,\n",
    "    num_sample_paths=100,\n",
    "    trainer=Trainer(\n",
    "        ctx=\"cpu\", \n",
    "        epochs=5, \n",
    "        learning_rate=1e-3, \n",
    "        hybridize=False,\n",
    "        batch_size=32,\n",
    "        num_batches_per_epoch=100,\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimator can be trained using our training dataset `train_ds` just by invoking its `train` method. The training returns a predictor that can be used to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "predictor = estimator.train(train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_it, ts_it = make_evaluation_predictions(\n",
    "    dataset=test_ds,  # test dataset\n",
    "    predictor=predictor,  # predictor\n",
    "    num_eval_samples=100,  # number of sample paths we want for evaluation\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_entry = list(forecast_it)[0]\n",
    "ts_entry = list(ts_it)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 Exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the probabilistic feedforward model to a point forecast model:\n",
    "- The output of the network should be `prediction_length` parameters\n",
    "- We do not need a distribution\n",
    "- We need to define a loss between the predictions and the true target for the training network\n",
    "- The prediction network should output directly the predictions\n",
    "- Set `solved = True` and check the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyPointNetwork(gluon.HybridBlock):\n",
    "    pass\n",
    "\n",
    "class MyPointTrainNetwork(MyPointNetwork):    \n",
    "    pass\n",
    "\n",
    "class MyPointPredNetwork(MyPointTrainNetwork):\n",
    "    pass\n",
    "\n",
    "class MyPointEstimator(GluonEstimator):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "solved = False\n",
    "\n",
    "if solved:\n",
    "    predictor = estimator.train(train_ds)\n",
    "    evaluate_solution(predictor, test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">WARNING:</span> YOU ARE TOO CLOSE TO THE EXERCISE SOLUTIONS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.1 Solution to Exercise 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.model.deepar import DeepAREstimator\n",
    "\n",
    "estimator = DeepAREstimator(\n",
    "    # required\n",
    "    prediction_length=dataset.metadata.prediction_length,\n",
    "    freq=dataset.metadata.freq,\n",
    "    # optional\n",
    "    context_length=100,\n",
    "    num_layers=2,\n",
    "    num_cells=20,\n",
    "    trainer=Trainer(\n",
    "        ctx=\"cpu\", \n",
    "        epochs=5, \n",
    "        learning_rate=1e-3, \n",
    "        batch_size=32,\n",
    "        num_batches_per_epoch=100\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.2 Solution to Exercise 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = DeepAREstimator(\n",
    "    # required\n",
    "    prediction_length=dataset.metadata.prediction_length,\n",
    "    freq=dataset.metadata.freq,\n",
    "    # optional\n",
    "    context_length=10,\n",
    "    num_layers=1,\n",
    "    num_cells=5,\n",
    "    dropout_rate=0.9,\n",
    "    trainer=Trainer(\n",
    "        ctx=\"cpu\", \n",
    "        epochs=3, \n",
    "        learning_rate=1e-3, \n",
    "        batch_size=32,\n",
    "        num_batches_per_epoch=100\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.3 Solution to Exercise 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.read_csv(\"https://raw.githubusercontent.com/numenta/NAB/master/data/realTraffic/occupancy_6005.csv\")\n",
    "df2 = pd.read_csv(\"https://raw.githubusercontent.com/numenta/NAB/master/data/realTraffic/occupancy_t4013.csv\")\n",
    "\n",
    "dataset = ListDataset(\n",
    "    [\n",
    "        {\"start\": df1.timestamp[0], \"target\": df1.value},\n",
    "        {\"start\": df2.timestamp[0], \"target\": df2.value},\n",
    "    ],\n",
    "    freq=\"30min\"\n",
    ")\n",
    "\n",
    "dataset_train = ListDataset(\n",
    "    [\n",
    "        {\"start\": df1.timestamp[0], \"target\": df1.value[:-48]},\n",
    "        {\"start\": df2.timestamp[0], \"target\": df2.value[:-48]},\n",
    "    ],\n",
    "    freq=\"30min\"\n",
    ")\n",
    "\n",
    "from gluonts.model.deepar import DeepAREstimator\n",
    "\n",
    "estimator = DeepAREstimator(\n",
    "    prediction_length=48,\n",
    "    freq=\"30min\",\n",
    "    trainer=Trainer(epochs=10),\n",
    ")\n",
    "\n",
    "predictor = estimator.train(dataset_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.4 Solution to Exercise 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyPointNetwork(gluon.HybridBlock):\n",
    "    def __init__(self, prediction_length, num_cells, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.prediction_length = prediction_length\n",
    "        self.num_cells = num_cells\n",
    "    \n",
    "        with self.name_scope():\n",
    "            # Set up a 3 layer neural network that directly predicts the target values\n",
    "            self.nn = mx.gluon.nn.HybridSequential()\n",
    "            self.nn.add(mx.gluon.nn.Dense(units=self.num_cells, activation='relu'))\n",
    "            self.nn.add(mx.gluon.nn.Dense(units=self.num_cells, activation='relu'))\n",
    "            self.nn.add(mx.gluon.nn.Dense(units=self.prediction_length, activation='softrelu'))\n",
    "\n",
    "class MyPointTrainNetwork(MyPointNetwork):    \n",
    "    def hybrid_forward(self, F, past_target, future_target):\n",
    "        prediction = self.nn(past_target)\n",
    "        # calculate L1 loss with the future_target to learn the median\n",
    "        return (prediction - future_target).abs().mean(axis=-1)\n",
    "\n",
    "\n",
    "class MyPointPredNetwork(MyPointTrainNetwork):\n",
    "    # The prediction network only receives past_target and returns predictions\n",
    "    def hybrid_forward(self, F, past_target):\n",
    "        prediction = self.nn(past_target)\n",
    "        return prediction.expand_dims(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyPointEstimator(GluonEstimator):\n",
    "    @validated()\n",
    "    def __init__(\n",
    "        self,\n",
    "        prediction_length: int,\n",
    "        context_length: int,\n",
    "        freq: str,\n",
    "        num_cells: int,\n",
    "        trainer: Trainer = Trainer()\n",
    "    ) -> None:\n",
    "        super().__init__(trainer=trainer)\n",
    "        self.prediction_length = prediction_length\n",
    "        self.context_length = context_length\n",
    "        self.freq = freq\n",
    "        self.num_cells = num_cells\n",
    "            \n",
    "    def create_transformation(self):\n",
    "        return InstanceSplitter(\n",
    "                    target_field=FieldName.TARGET,\n",
    "                    is_pad_field=FieldName.IS_PAD,\n",
    "                    start_field=FieldName.START,\n",
    "                    forecast_start_field=FieldName.FORECAST_START,\n",
    "                    train_sampler=ExpectedNumInstanceSampler(num_instances=1),\n",
    "                    past_length=self.context_length,\n",
    "                    future_length=self.prediction_length,\n",
    "                )\n",
    "    \n",
    "    def create_training_network(self) -> MyPointTrainNetwork:\n",
    "        return MyPointTrainNetwork(\n",
    "            prediction_length=self.prediction_length,\n",
    "            num_cells = self.num_cells\n",
    "        )\n",
    "\n",
    "    def create_predictor(\n",
    "        self, transformation: Transformation, trained_network: HybridBlock\n",
    "    ) -> Predictor:\n",
    "        prediction_network = MyPointPredNetwork(\n",
    "            prediction_length=self.prediction_length,\n",
    "            num_cells=self.num_cells\n",
    "        )\n",
    "\n",
    "        copy_parameters(trained_network, prediction_network)\n",
    "\n",
    "        return RepresentableBlockPredictor(\n",
    "            input_transform=transformation,\n",
    "            prediction_net=prediction_network,\n",
    "            batch_size=self.trainer.batch_size,\n",
    "            freq=self.freq,\n",
    "            prediction_length=self.prediction_length,\n",
    "            ctx=self.trainer.ctx,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = MyPointEstimator(\n",
    "    prediction_length=prediction_length,\n",
    "    freq=freq,\n",
    "    context_length=2*prediction_length,\n",
    "    num_cells=40,\n",
    "    trainer=Trainer(\n",
    "        ctx=\"cpu\", \n",
    "        epochs=5, \n",
    "        learning_rate=1e-3, \n",
    "        hybridize=False, \n",
    "        num_batches_per_epoch=100\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "gluonts-workshop",
   "language": "python",
   "name": "gluonts-workshop"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}