18_reinforcement_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 18 – Reinforcement Learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code in chapter 18_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/18_reinforcement_learning.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU was detected. CNNs can be very slow without a GPU.\n"
     ]
    }
   ],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    !apt update && apt install -y libpq-dev libsdl2-dev swig xorg-dev xvfb\n",
    "    !pip install -q -U tf-agents-nightly pyvirtualdisplay gym[atari]\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# To get smooth animations\n",
    "import matplotlib.animation as animation\n",
    "mpl.rc('animation', html='jshtml')\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"rl\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to OpenAI gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will be using [OpenAI gym](https://gym.openai.com/), a great toolkit for developing and comparing Reinforcement Learning algorithms. It provides many environments for your learning *agents* to interact with. Let's start by importing `gym`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's list all the available environments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_values([EnvSpec(Copy-v0), EnvSpec(RepeatCopy-v0), EnvSpec(ReversedAddition-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(Reverse-v0), EnvSpec(CartPole-v0), EnvSpec(CartPole-v1), EnvSpec(MountainCar-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(Pendulum-v0), EnvSpec(Acrobot-v1), EnvSpec(LunarLander-v2), EnvSpec(LunarLanderContinuous-v2), EnvSpec(BipedalWalker-v3), EnvSpec(BipedalWalkerHardcore-v3), EnvSpec(CarRacing-v0), EnvSpec(Blackjack-v0), EnvSpec(KellyCoinflip-v0), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(FrozenLake-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(CliffWalking-v0), EnvSpec(NChain-v0), EnvSpec(Roulette-v0), EnvSpec(Taxi-v3), EnvSpec(GuessingGame-v0), EnvSpec(HotterColder-v0), EnvSpec(Reacher-v2), EnvSpec(Pusher-v2), EnvSpec(Thrower-v2), EnvSpec(Striker-v2), EnvSpec(InvertedPendulum-v2), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(HalfCheetah-v2), EnvSpec(HalfCheetah-v3), EnvSpec(Hopper-v2), EnvSpec(Hopper-v3), EnvSpec(Swimmer-v2), EnvSpec(Swimmer-v3), EnvSpec(Walker2d-v2), EnvSpec(Walker2d-v3), EnvSpec(Ant-v2), EnvSpec(Ant-v3), EnvSpec(Humanoid-v2), EnvSpec(Humanoid-v3), EnvSpec(HumanoidStandup-v2), EnvSpec(FetchSlide-v1), EnvSpec(FetchPickAndPlace-v1), EnvSpec(FetchReach-v1), EnvSpec(FetchPush-v1), EnvSpec(HandReach-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v1), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v1), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v1), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HandManipulateBlock-v0), EnvSpec(HandManipulateBlockTouchSensors-v0), EnvSpec(HandManipulateBlockTouchSensors-v1), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v1), EnvSpec(HandManipulateEggFull-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(HandManipulateEggTouchSensors-v0), EnvSpec(HandManipulateEggTouchSensors-v1), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v1), EnvSpec(HandManipulatePenFull-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(HandManipulatePenTouchSensors-v0), EnvSpec(HandManipulatePenTouchSensors-v1), EnvSpec(FetchSlideDense-v1), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(FetchReachDense-v1), EnvSpec(FetchPushDense-v1), EnvSpec(HandReachDense-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v1), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v1), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(HandManipulateEggDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v1), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v1), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v1), EnvSpec(Adventure-v0), EnvSpec(Adventure-v4), EnvSpec(AdventureDeterministic-v0), EnvSpec(AdventureDeterministic-v4), EnvSpec(AdventureNoFrameskip-v0), EnvSpec(AdventureNoFrameskip-v4), EnvSpec(Adventure-ram-v0), EnvSpec(Adventure-ram-v4), EnvSpec(Adventure-ramDeterministic-v0), EnvSpec(Adventure-ramDeterministic-v4), EnvSpec(Adventure-ramNoFrameskip-v0), EnvSpec(Adventure-ramNoFrameskip-v4), EnvSpec(AirRaid-v0), EnvSpec(AirRaid-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(AirRaidDeterministic-v4), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(AirRaid-ram-v4), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(Alien-v4), EnvSpec(AlienDeterministic-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(AlienNoFrameskip-v0), EnvSpec(AlienNoFrameskip-v4), EnvSpec(Alien-ram-v0), EnvSpec(Alien-ram-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(Amidar-v0), EnvSpec(Amidar-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(Assault-v0), EnvSpec(Assault-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(AssaultDeterministic-v4), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(Assault-ram-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(Asterix-v0), EnvSpec(Asterix-v4), EnvSpec(AsterixDeterministic-v0), EnvSpec(AsterixDeterministic-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Asterix-ram-v0), EnvSpec(Asterix-ram-v4), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(Asteroids-v0), EnvSpec(Asteroids-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(Atlantis-v0), EnvSpec(Atlantis-v4), EnvSpec(AtlantisDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(BankHeist-v4), EnvSpec(BankHeistDeterministic-v0), EnvSpec(BankHeistDeterministic-v4), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BankHeist-ram-v0), EnvSpec(BankHeist-ram-v4), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(BattleZone-v0), EnvSpec(BattleZone-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(BattleZone-ram-v0), EnvSpec(BattleZone-ram-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(BeamRider-v4), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(BeamRider-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Berzerk-v0), EnvSpec(Berzerk-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(BerzerkDeterministic-v4), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(Bowling-v0), EnvSpec(Bowling-v4), EnvSpec(BowlingDeterministic-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Bowling-ram-v4), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(Boxing-v0), EnvSpec(Boxing-v4), EnvSpec(BoxingDeterministic-v0), EnvSpec(BoxingDeterministic-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Boxing-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(Breakout-v0), EnvSpec(Breakout-v4), EnvSpec(BreakoutDeterministic-v0), EnvSpec(BreakoutDeterministic-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(Breakout-ram-v0), EnvSpec(Breakout-ram-v4), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(Carnival-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Centipede-v0), EnvSpec(Centipede-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(CentipedeDeterministic-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(Centipede-ram-v0), EnvSpec(Centipede-ram-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(ChopperCommand-v0), EnvSpec(ChopperCommand-v4), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(CrazyClimber-v0), EnvSpec(CrazyClimber-v4), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(CrazyClimber-ram-v0), EnvSpec(CrazyClimber-ram-v4), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(Defender-v0), EnvSpec(Defender-v4), EnvSpec(DefenderDeterministic-v0), EnvSpec(DefenderDeterministic-v4), EnvSpec(DefenderNoFrameskip-v0), EnvSpec(DefenderNoFrameskip-v4), EnvSpec(Defender-ram-v0), EnvSpec(Defender-ram-v4), EnvSpec(Defender-ramDeterministic-v0), EnvSpec(Defender-ramDeterministic-v4), EnvSpec(Defender-ramNoFrameskip-v0), EnvSpec(Defender-ramNoFrameskip-v4), EnvSpec(DemonAttack-v0), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(DemonAttack-ram-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(DoubleDunk-v0), EnvSpec(DoubleDunk-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorAction-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(ElevatorAction-ram-v0), EnvSpec(ElevatorAction-ram-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(EnduroDeterministic-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(Enduro-ram-v0), EnvSpec(Enduro-ram-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(FishingDerby-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(FishingDerby-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Freeway-v0), EnvSpec(Freeway-v4), EnvSpec(FreewayDeterministic-v0), EnvSpec(FreewayDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Freeway-ram-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Frostbite-v0), EnvSpec(Frostbite-v4), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(Frostbite-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(Gopher-v4), EnvSpec(GopherDeterministic-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Gopher-ram-v4), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(Gravitar-v0), EnvSpec(Gravitar-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(GravitarDeterministic-v4), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Gravitar-ram-v0), EnvSpec(Gravitar-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Hero-v0), EnvSpec(Hero-v4), EnvSpec(HeroDeterministic-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(Hero-ram-v0), EnvSpec(Hero-ram-v4), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(IceHockey-v0), EnvSpec(IceHockey-v4), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(Jamesbond-v0), EnvSpec(Jamesbond-v4), EnvSpec(JamesbondDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Jamesbond-ram-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(JourneyEscape-v0), EnvSpec(JourneyEscape-v4), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Kangaroo-v0), EnvSpec(Kangaroo-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(KangarooDeterministic-v4), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(KangarooNoFrameskip-v4), EnvSpec(Kangaroo-ram-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(Krull-v0), EnvSpec(Krull-v4), EnvSpec(KrullDeterministic-v0), EnvSpec(KrullDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Krull-ram-v0), EnvSpec(Krull-ram-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(KungFuMaster-v0), EnvSpec(KungFuMaster-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(KungFuMaster-ram-v4), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(MsPacman-v0), EnvSpec(MsPacman-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(NameThisGame-v0), EnvSpec(NameThisGame-v4), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(NameThisGame-ram-v0), EnvSpec(NameThisGame-ram-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(Phoenix-v0), EnvSpec(Phoenix-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Pitfall-v0), EnvSpec(Pitfall-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(Pitfall-ram-v4), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Pong-v0), EnvSpec(Pong-v4), EnvSpec(PongDeterministic-v0), EnvSpec(PongDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(PongNoFrameskip-v4), EnvSpec(Pong-ram-v0), EnvSpec(Pong-ram-v4), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Pooyan-v0), EnvSpec(Pooyan-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Pooyan-ram-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(PrivateEye-v0), EnvSpec(PrivateEye-v4), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(PrivateEye-ram-v0), EnvSpec(PrivateEye-ram-v4), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Qbert-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Qbert-ram-v0), EnvSpec(Qbert-ram-v4), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(Riverraid-v0), EnvSpec(Riverraid-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(Riverraid-ram-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(RoadRunner-v0), EnvSpec(RoadRunner-v4), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(RoadRunner-ram-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(Robotank-v0), EnvSpec(Robotank-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(Robotank-ram-v0), EnvSpec(Robotank-ram-v4), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Seaquest-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(Seaquest-ram-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(Skiing-v0), EnvSpec(Skiing-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(SkiingDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(Skiing-ram-v0), EnvSpec(Skiing-ram-v4), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Solaris-v0), EnvSpec(Solaris-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Solaris-ram-v0), EnvSpec(Solaris-ram-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(SpaceInvaders-v0), EnvSpec(SpaceInvaders-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(StarGunner-v0), EnvSpec(StarGunner-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(StarGunner-ram-v0), EnvSpec(StarGunner-ram-v4), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(Tennis-v0), EnvSpec(Tennis-v4), EnvSpec(TennisDeterministic-v0), EnvSpec(TennisDeterministic-v4), EnvSpec(TennisNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v4), EnvSpec(Tennis-ram-v0), EnvSpec(Tennis-ram-v4), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(TimePilot-v0), EnvSpec(TimePilot-v4), EnvSpec(TimePilotDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(Tutankham-v0), EnvSpec(Tutankham-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(TutankhamDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Tutankham-ram-v4), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(UpNDown-v0), EnvSpec(UpNDown-v4), EnvSpec(UpNDownDeterministic-v0), EnvSpec(UpNDownDeterministic-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(UpNDown-ram-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(Venture-v0), EnvSpec(Venture-v4), EnvSpec(VentureDeterministic-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(VentureNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(Venture-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(VideoPinball-v0), EnvSpec(VideoPinball-v4), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(VideoPinball-ram-v0), EnvSpec(VideoPinball-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(WizardOfWor-v0), EnvSpec(WizardOfWor-v4), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(YarsRevenge-v0), EnvSpec(YarsRevenge-v4), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-v0), EnvSpec(Zaxxon-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(Zaxxon-ram-v0), EnvSpec(Zaxxon-ram-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(CubeCrash-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(MemorizeDigits-v0)])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gym.envs.registry.all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Cart-Pole is a very simple environment composed of a cart that can move left or right, and pole placed vertically on top of it. The agent must move the cart left or right to keep the pole upright."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make('CartPole-v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize the environment by calling is `reset()` method. This returns an observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observations vary depending on the environment. In this case it is a 1D NumPy array composed of 4 floats: they represent the cart's horizontal position, its velocity, the angle of the pole (0 = vertical), and the angular velocity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.01258566, -0.00156614,  0.04207708, -0.00180545])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An environment can be visualized by calling its `render()` method, and you can pick the rendering mode (the rendering options depend on the environment)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: some environments (including the Cart-Pole) require access to your display, which opens up a separate window, even if you specify `mode=\"rgb_array\"`. In general you can safely ignore that window. However, if Jupyter is running on a headless server (ie. without a screen) it will raise an exception. One way to avoid this is to install a fake X server like [Xvfb](http://en.wikipedia.org/wiki/Xvfb). On Debian or Ubuntu:\n",
    "\n",
    "```bash\n",
    "$ apt update\n",
    "$ apt install -y xvfb\n",
    "```\n",
    "\n",
    "You can then start Jupyter using the `xvfb-run` command:\n",
    "\n",
    "```bash\n",
    "$ xvfb-run -s \"-screen 0 1400x900x24\" jupyter notebook\n",
    "```\n",
    "\n",
    "Alternatively, you can install the [pyvirtualdisplay](https://github.com/ponty/pyvirtualdisplay) Python library which wraps Xvfb:\n",
    "\n",
    "```bash\n",
    "python3 -m pip install -U pyvirtualdisplay\n",
    "```\n",
    "\n",
    "And run the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import pyvirtualdisplay\n",
    "    display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()\n",
    "except ImportError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we will set `mode=\"rgb_array\"` to get an image of the environment as a NumPy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(800, 1200, 3)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img = env.render(mode=\"rgb_array\")\n",
    "img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_environment(env, figsize=(5,4)):\n",
    "    plt.figure(figsize=figsize)\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    plt.imshow(img)\n",
    "    plt.axis(\"off\")\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAADhCAYAAACa/D2AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAA4tJREFUeJzt3dFNwlAYgFFrWMI5WMM5cCacwzWcwzHqm1EC0fhRbyvnJCT0oeR/aL7chsKd5nm+A+D37kcPALB1QgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUC0Gz3ABf4AAFjCtMSHWpECREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUC0Gz0AFK/PT1+O94fjoEm4ZVakAJGQAkRCChAJKUAkpACRkAJEQgoQCSmbdfoMKYwipACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKT8G/vDcfQI3CghBYiEFCASUoBISAEiIWWT7CDKmggpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKf+CHUQZSUgBIiEFiIQUIBJSgEhI2Rw7iLI2QsqqTNP07eua58E17EYPAL/x8nb4eP/48DxwErAiZYM+R/TcMfw1IQWIhJTNOb2Vd2vPaNM8z6NnOGeVQ7G8Jb8UWum1zt9a5AKzIgWIhBQgElKASEgBIiEFiIQUIBJSgMhv7VkVz3qyRVakAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChAJKUAkpACRkAJEQgoQCSlAJKQAkZACREIKEAkpQCSkAJGQAkRCChDtRg9wwTR6AICfsiIFiIQUIBJSgEhIASIhBYiEFCASUoBISAEiIQWIhBQgElKASEgBIiEFiIQUIBJSgEhIASIhBYiEFCASUoBISAEiIQWIhBQgElKASEgBIiEFiIQUIHoHOogjx4cLz8kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how to interact with an environment. Your agent will need to select an action from an \"action space\" (the set of possible actions). Let's see what this environment's action space looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(2)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yep, just two possible actions: accelerate towards the left or towards the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the pole is leaning toward the right (`obs[2] > 0`), let's accelerate the cart toward the right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.01261699,  0.19292789,  0.04204097, -0.28092127])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "action = 1  # accelerate right\n",
    "obs, reward, done, info = env.step(action)\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the cart is now moving toward the right (`obs[1] > 0`). The pole is still tilted toward the right (`obs[2] > 0`), but its angular velocity is now negative (`obs[3] < 0`), so it will likely be tilted toward the left after the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure cart_pole_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAADqCAYAAACcPZ9GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAA8JJREFUeJzt3dFpwmAUgNGmZAnnaMdwDp2pztE1OkfHSN+KUIU+GL/feA4IIij3IX5cJCbTsiwvANzfaz0AwLMSYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYIDIXA9whf9HA6OZbv2BNmCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAAJG5HgDW8nU6/nnt7fARTAKX2YABIgIMEBFggIgAA0QEGCAiwAARAQaICDBARIABIgIMEBFggIgAA0QEGCAiwDwNV0JjNAIMEBFggIgAA0QEGCAiwAARAWaTLt0PDkYjwAARAQaICDBARIABIgIMEBFggIgAA0QEGCAiwAARAQaICDBARIABIgIMEBFggIgAA0QEGCAiwDwFt6RnRAIMEBFggIgAA0QEGCAiwAARAWZz3JKeRyHAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgHkY0zT963Hr98Ja5noAWMPn9+H3+X53CieB62zAbM55fGFkAszmCTKjEmA2z08QjEqA2Zzz4O53p5f3owAzpmlZlnqGS4YcitY9zlIY9PvAGG5+ANqAASICDBARYICIAANEBBggIsAAEQEGiLgYDw/DObpsjQ0YICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYIDLXA1wx1QMArM0GDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYIDID1HWJEWNmP1BAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "save_fig(\"cart_pole_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like it's doing what we're telling it to do!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment also tells the agent how much reward it got during the last step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the game is over, the environment returns `done=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, `info` is an environment-specific dictionary that can provide some extra information that you may find useful for debugging or for training. For example, in some games it may indicate how many lives the agent has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence of steps between the moment the environment is reset until it is done is called an \"episode\". At the end of an episode (i.e., when `step()` returns `done=True`), you should reset the environment before you continue to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "if done:\n",
    "    obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now how can we make the poll remain upright? We will need to define a _policy_ for that. This is the strategy that the agent will use to select an action at each step. It can use all the past actions and observations to decide what to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A simple hard-coded policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's hard code a simple strategy: if the pole is tilting to the left, then push the cart to the left, and _vice versa_. Let's see if that works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "\n",
    "def basic_policy(obs):\n",
    "    angle = obs[2]\n",
    "    return 0 if angle < 0 else 1\n",
    "\n",
    "totals = []\n",
    "for episode in range(500):\n",
    "    episode_rewards = 0\n",
    "    obs = env.reset()\n",
    "    for step in range(200):\n",
    "        action = basic_policy(obs)\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        episode_rewards += reward\n",
    "        if done:\n",
    "            break\n",
    "    totals.append(episode_rewards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41.718, 8.858356280936096, 24.0, 68.0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(totals), np.std(totals), np.min(totals), np.max(totals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, as expected, this strategy is a bit too basic: the best it did was to keep the poll up for only 68 steps. This environment is considered solved when the agent keeps the poll up for 200 steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize one episode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "\n",
    "frames = []\n",
    "\n",
    "obs = env.reset()\n",
    "for step in range(200):\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    action = basic_policy(obs)\n",
    "\n",
    "    obs, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now show the animation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_scene(num, frames, patch):\n",
    "    patch.set_data(frames[num])\n",
    "    return patch,\n",
    "\n",
    "def plot_animation(frames, repeat=False, interval=40):\n",
    "    fig = plt.figure()\n",
    "    patch = plt.imshow(frames[0])\n",
    "    plt.axis('off')\n",
    "    anim = animation.FuncAnimation(\n",
    "        fig, update_scene, fargs=(frames, patch),\n",
    "        frames=len(frames), repeat=repeat, interval=interval)\n",
    "    plt.close()\n",
    "    return anim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly the system is unstable and after just a few wobbles, the pole ends up too tilted: game over. We will need to be smarter than that!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Policies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural network that will take observations as inputs, and output the action to take for each observation. To choose an action, the network will estimate a probability for each action, then we will select an action randomly according to the estimated probabilities. In the case of the Cart-Pole environment, there are just two possible actions (left or right), so we only need one output neuron: it will output the probability `p` of the action 0 (left), and of course the probability of action 1 (right) will be `1 - p`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "n_inputs = 4 # == env.observation_space.shape[0]\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(5, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular environment, the past actions and observations can safely be ignored, since each observation contains the environment's full state. If there were some hidden state then you may need to consider past actions and observations in order to try to infer the hidden state of the environment. For example, if the environment only revealed the position of the cart but not its velocity, you would have to consider not only the current observation but also the previous observation in order to estimate the current velocity. Another example is if the observations are noisy: you may want to use the past few observations to estimate the most likely current state. Our problem is thus as simple as can be: the current observation is noise-free and contains the environment's full state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may wonder why we plan to pick a random action based on the probability given by the policy network, rather than just picking the action with the highest probability. This approach lets the agent find the right balance between _exploring_ new actions and _exploiting_ the actions that are known to work well. Here's an analogy: suppose you go to a restaurant for the first time, and all the dishes look equally appealing so you randomly pick one. If it turns out to be good, you can increase the probability to order it next time, but you shouldn't increase that probability to 100%, or else you will never try out the other dishes, some of which may be even better than the one you tried."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a small function that will run the model to play one episode, and return the frames so we can display an animation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_policy_net(model, n_max_steps=200, seed=42):\n",
    "    frames = []\n",
    "    env = gym.make(\"CartPole-v1\")\n",
    "    env.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    obs = env.reset()\n",
    "    for step in range(n_max_steps):\n",
    "        frames.append(env.render(mode=\"rgb_array\"))\n",
    "        left_proba = model.predict(obs.reshape(1, -1))\n",
    "        action = int(np.random.rand() > left_proba)\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "    env.close()\n",
    "    return frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at how well this randomly initialized policy network performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yeah... pretty bad. The neural network will have to learn to do better. First let's see if it is capable of learning the basic policy we used earlier: go left if the pole is tilting left, and go right if it is tilting right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make the same net play in 50 different environments in parallel (this will give us a diverse training batch at each step), and train for 5000 iterations. We also reset environments when they are done. We train the model using a custom training loop so we can easily use the predictions at each training step to advance the environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 4999, Loss: 0.094"
     ]
    }
   ],
   "source": [
    "n_environments = 50\n",
    "n_iterations = 5000\n",
    "\n",
    "envs = [gym.make(\"CartPole-v1\") for _ in range(n_environments)]\n",
    "for index, env in enumerate(envs):\n",
    "    env.seed(index)\n",
    "np.random.seed(42)\n",
    "observations = [env.reset() for env in envs]\n",
    "optimizer = keras.optimizers.RMSprop()\n",
    "loss_fn = keras.losses.binary_crossentropy\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    # if angle < 0, we want proba(left) = 1., or else proba(left) = 0.\n",
    "    target_probas = np.array([([1.] if obs[2] < 0 else [0.])\n",
    "                              for obs in observations])\n",
    "    with tf.GradientTape() as tape:\n",
    "        left_probas = model(np.array(observations))\n",
    "        loss = tf.reduce_mean(loss_fn(target_probas, left_probas))\n",
    "    print(\"\\rIteration: {}, Loss: {:.3f}\".format(iteration, loss.numpy()), end=\"\")\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "    actions = (np.random.rand(n_environments, 1) > left_probas.numpy()).astype(np.int32)\n",
    "    for env_index, env in enumerate(envs):\n",
    "        obs, reward, done, info = env.step(actions[env_index][0])\n",
    "        observations[env_index] = obs if not done else env.reset()\n",
    "\n",
    "for env in envs:\n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like it learned the policy correctly. Now let's see if it can learn a better policy on its own. One that does not wobble as much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train this neural network we will need to define the target probabilities `y`. If an action is good we should increase its probability, and conversely if it is bad we should reduce it. But how do we know whether an action is good or bad? The problem is that most actions have delayed effects, so when you win or lose points in an episode, it is not clear which actions contributed to this result: was it just the last action? Or the last 10? Or just one action 50 steps earlier? This is called the _credit assignment problem_.\n",
    "\n",
    "The _Policy Gradients_ algorithm tackles this problem by first playing multiple episodes, then making the actions in good episodes slightly more likely, while actions in bad episodes are made slightly less likely. First we play, then we go back and think about what we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating a function to play a single step using the model. We will also pretend for now that whatever action it takes is the right one, so we can compute the loss and its gradients (we will just save these gradients for now, and modify them later depending on how good or bad the action turned out to be):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, obs, model, loss_fn):\n",
    "    with tf.GradientTape() as tape:\n",
    "        left_proba = model(obs[np.newaxis])\n",
    "        action = (tf.random.uniform([1, 1]) > left_proba)\n",
    "        y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32)\n",
    "        loss = tf.reduce_mean(loss_fn(y_target, left_proba))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    obs, reward, done, info = env.step(int(action[0, 0].numpy()))\n",
    "    return obs, reward, done, grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `left_proba` is high, then `action` will most likely be `False` (since a random number uniformally sampled between 0 and 1 will probably not be greater than `left_proba`). And `False` means 0 when you cast it to a number, so `y_target` would be equal to 1 - 0 = 1. In other words, we set the target to 1, meaning we pretend that the probability of going left should have been 100% (so we took the right action)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create another function that will rely on the `play_one_step()` function to play multiple episodes, returning all the rewards and gradients, for each episode and each step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn):\n",
    "    all_rewards = []\n",
    "    all_grads = []\n",
    "    for episode in range(n_episodes):\n",
    "        current_rewards = []\n",
    "        current_grads = []\n",
    "        obs = env.reset()\n",
    "        for step in range(n_max_steps):\n",
    "            obs, reward, done, grads = play_one_step(env, obs, model, loss_fn)\n",
    "            current_rewards.append(reward)\n",
    "            current_grads.append(grads)\n",
    "            if done:\n",
    "                break\n",
    "        all_rewards.append(current_rewards)\n",
    "        all_grads.append(current_grads)\n",
    "    return all_rewards, all_grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Policy Gradients algorithm uses the model to play the episode several times (e.g., 10 times), then it goes back and looks at all the rewards, discounts them and normalizes them. So let's create couple functions for that: the first will compute discounted rewards; the second will normalize the discounted rewards across many episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rewards(rewards, discount_rate):\n",
    "    discounted = np.array(rewards)\n",
    "    for step in range(len(rewards) - 2, -1, -1):\n",
    "        discounted[step] += discounted[step + 1] * discount_rate\n",
    "    return discounted\n",
    "\n",
    "def discount_and_normalize_rewards(all_rewards, discount_rate):\n",
    "    all_discounted_rewards = [discount_rewards(rewards, discount_rate)\n",
    "                              for rewards in all_rewards]\n",
    "    flat_rewards = np.concatenate(all_discounted_rewards)\n",
    "    reward_mean = flat_rewards.mean()\n",
    "    reward_std = flat_rewards.std()\n",
    "    return [(discounted_rewards - reward_mean) / reward_std\n",
    "            for discounted_rewards in all_discounted_rewards]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say there were 3 actions, and after each action there was a reward: first 10, then 0, then -50. If we use a discount factor of 80%, then the 3rd action will get -50 (full credit for the last reward), but the 2nd action will only get -40 (80% credit for the last reward), and the 1st action will get 80% of -40 (-32) plus full credit for the first reward (+10), which leads to a discounted reward of -22:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-22, -40, -50])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_rewards([10, 0, -50], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To normalize all discounted rewards across all episodes, we compute the mean and standard deviation of all the discounted rewards, and we subtract the mean from each discounted reward, and divide by the standard deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([-0.28435071, -0.86597718, -1.18910299]),\n",
       " array([1.26665318, 1.0727777 ])]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_and_normalize_rewards([[10, 0, -50], [10, 20]], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iterations = 150\n",
    "n_episodes_per_update = 10\n",
    "n_max_steps = 200\n",
    "discount_rate = 0.95"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.01)\n",
    "loss_fn = keras.losses.binary_crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(5, activation=\"elu\", input_shape=[4]),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 149, mean rewards: 191.3"
     ]
    }
   ],
   "source": [
    "env = gym.make(\"CartPole-v1\")\n",
    "env.seed(42);\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    all_rewards, all_grads = play_multiple_episodes(\n",
    "        env, n_episodes_per_update, n_max_steps, model, loss_fn)\n",
    "    total_rewards = sum(map(sum, all_rewards))                     # Not shown in the book\n",
    "    print(\"\\rIteration: {}, mean rewards: {:.1f}\".format(          # Not shown\n",
    "        iteration, total_rewards / n_episodes_per_update), end=\"\") # Not shown\n",
    "    all_final_rewards = discount_and_normalize_rewards(all_rewards,\n",
    "                                                       discount_rate)\n",
    "    all_mean_grads = []\n",
    "    for var_index in range(len(model.trainable_variables)):\n",
    "        mean_grads = tf.reduce_mean(\n",
    "            [final_reward * all_grads[episode_index][step][var_index]\n",
    "             for episode_index, final_rewards in enumerate(all_final_rewards)\n",
    "                 for step, final_reward in enumerate(final_rewards)], axis=0)\n",
    "        all_mean_grads.append(mean_grads)\n",
    "    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Markov Chains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "States: 0 0 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 ...\n",
      "States: 0 0 3 \n",
      "States: 0 0 0 1 2 1 2 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "transition_probabilities = [ # shape=[s, s']\n",
    "        [0.7, 0.2, 0.0, 0.1],  # from s0 to s0, s1, s2, s3\n",
    "        [0.0, 0.0, 0.9, 0.1],  # from s1 to ...\n",
    "        [0.0, 1.0, 0.0, 0.0],  # from s2 to ...\n",
    "        [0.0, 0.0, 0.0, 1.0]]  # from s3 to ...\n",
    "\n",
    "n_max_steps = 50\n",
    "\n",
    "def print_sequence():\n",
    "    current_state = 0\n",
    "    print(\"States:\", end=\" \")\n",
    "    for step in range(n_max_steps):\n",
    "        print(current_state, end=\" \")\n",
    "        if current_state == 3:\n",
    "            break\n",
    "        current_state = np.random.choice(range(4), p=transition_probabilities[current_state])\n",
    "    else:\n",
    "        print(\"...\", end=\"\")\n",
    "    print()\n",
    "\n",
    "for _ in range(10):\n",
    "    print_sequence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Markov Decision Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define some transition probabilities, rewards and possible actions. For example, in state s0, if action a0 is chosen then with proba 0.7 we will go to state s0 with reward +10, with probability 0.3 we will go to state s1 with no reward, and with never go to state s2 (so the transition probabilities are `[0.7, 0.3, 0.0]`, and the rewards are `[+10, 0, 0]`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "transition_probabilities = [ # shape=[s, a, s']\n",
    "        [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]],\n",
    "        [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]],\n",
    "        [None, [0.8, 0.1, 0.1], None]]\n",
    "rewards = [ # shape=[s, a, s']\n",
    "        [[+10, 0, 0], [0, 0, 0], [0, 0, 0]],\n",
    "        [[0, 0, 0], [0, 0, 0], [0, 0, -50]],\n",
    "        [[0, 0, 0], [+40, 0, 0], [0, 0, 0]]]\n",
    "possible_actions = [[0, 1, 2], [0, 2], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-Value Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q_values = np.full((3, 3), -np.inf) # -np.inf for impossible actions\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state, actions] = 0.0  # for all possible actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.90  # the discount factor\n",
    "\n",
    "history1 = [] # Not shown in the book (for the figure below)\n",
    "for iteration in range(50):\n",
    "    Q_prev = Q_values.copy()\n",
    "    history1.append(Q_prev) # Not shown\n",
    "    for s in range(3):\n",
    "        for a in possible_actions[s]:\n",
    "            Q_values[s, a] = np.sum([\n",
    "                    transition_probabilities[s][a][sp]\n",
    "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
    "                for sp in range(3)])\n",
    "\n",
    "history1 = np.array(history1) # Not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18.91891892, 17.02702702, 13.62162162],\n",
       "       [ 0.        ,        -inf, -4.87971488],\n",
       "       [       -inf, 50.13365013,        -inf]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimal policy for this MDP, when using a discount factor of 0.90, is to choose action a0 when in state s0, and choose action a0 when in state s1, and finally choose action a1 (the only possible action) when in state s2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try again with a discount factor of 0.95:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q_values = np.full((3, 3), -np.inf) # -np.inf for impossible actions\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state, actions] = 0.0  # for all possible actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.95  # the discount factor\n",
    "\n",
    "for iteration in range(50):\n",
    "    Q_prev = Q_values.copy()\n",
    "    for s in range(3):\n",
    "        for a in possible_actions[s]:\n",
    "            Q_values[s, a] = np.sum([\n",
    "                    transition_probabilities[s][a][sp]\n",
    "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
    "                for sp in range(3)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[21.73304188, 20.63807938, 16.70138772],\n",
       "       [ 0.95462106,        -inf,  1.01361207],\n",
       "       [       -inf, 53.70728682,        -inf]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 1])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the policy has changed! In state s1, we now prefer to go through the fire (choose action a2). This is because the discount factor is larger so the agent values the future more, and it is therefore ready to pay an immediate penalty in order to get more future rewards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q-Learning works by watching an agent play (e.g., randomly) and gradually improving its estimates of the Q-Values. Once it has accurate Q-Value estimates (or close enough), then the optimal policy consists in choosing the action that has the highest Q-Value (i.e., the greedy policy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to simulate an agent moving around in the environment, so let's define a function to perform some action and get the new state and a reward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step(state, action):\n",
    "    probas = transition_probabilities[state][action]\n",
    "    next_state = np.random.choice([0, 1, 2], p=probas)\n",
    "    reward = rewards[state][action][next_state]\n",
    "    return next_state, reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need an exploration policy, which can be any policy, as long as it visits every possible state many times. We will just use a random policy, since the state space is very small:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exploration_policy(state):\n",
    "    return np.random.choice(possible_actions[state])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's initialize the Q-Values like earlier, and run the Q-Learning algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "Q_values = np.full((3, 3), -np.inf)\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state][actions] = 0\n",
    "\n",
    "alpha0 = 0.05 # initial learning rate\n",
    "decay = 0.005 # learning rate decay\n",
    "gamma = 0.90 # discount factor\n",
    "state = 0 # initial state\n",
    "history2 = [] # Not shown in the book\n",
    "\n",
    "for iteration in range(10000):\n",
    "    history2.append(Q_values.copy()) # Not shown\n",
    "    action = exploration_policy(state)\n",
    "    next_state, reward = step(state, action)\n",
    "    next_value = np.max(Q_values[next_state]) # greedy policy at the next step\n",
    "    alpha = alpha0 / (1 + iteration * decay)\n",
    "    Q_values[state, action] *= 1 - alpha\n",
    "    Q_values[state, action] += alpha * (reward + gamma * next_value)\n",
    "    state = next_state\n",
    "\n",
    "history2 = np.array(history2) # Not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18.77621289, 17.2238872 , 13.74543343],\n",
       "       [ 0.        ,        -inf, -8.00485647],\n",
       "       [       -inf, 49.40208921,        -inf]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1) # optimal action for each state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure q_value_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAEYCAYAAABBfQDEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecXFX5x/HPk82md5JAEpIAMSEISvwRaVKigAqioKGJKIJ0UIoogpRIEWlSBJRqkCJNECMo0gIoLaEkkACBBEgnPdmWtnl+f5w77N2Z2TK7s3tnZ7/v1+u+ZuaeuXees7Nz55lzzznX3B0REREREQk6JB2AiIiIiEghUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVERKRgmNlWZuZmNjbpWKT9UoIsRcPMLjWzt5KOIylmtm/0pdIn6VhEpHWY2RAzu9XM5pvZejNbYGa3mdmWDWw32cxubK04czQPGAS02+O5JE8JsuRFMw7S083s9jrKDogSvlEtE3XTmNlxZraqrsetFMN8MzsjbfULhC+V1a0Zi4gkw8y2BqYCOwBHA58DjgK2B6aY2VaJBZeFmXVqzPPcvdrdF7v7xpaOSaQuSpCl2Zp5kL4DONzMumcpOxZ40d1n5TXgAmVBaVO3d/f10ZeKLo8p0j7cBGwC9nX3Z9x9rrs/B+wbrb+pqTuOGj3uN7OV0fK4mY2MlY8ws8fMbLGZVZjZG2Z2YNo+PjazCWZ2Z9SIcG+s+8R4M3vKzCrNbKaZ7RfbrlYXCzMbFz3ex8xejbaZamb/l/Z6x5rZ3Kh8kpmdYmY6HkqTKEGWfGjOQfpuoBQ4LL7SzAYA3wFujx6XRgfZj8ysysxmmdnZZmZ17djM7jGzv6ety+iGEbUAv2tma83sfTP7WX37Tdt2X+A2oHd0AHczOz8q62xmV0Wt6RVm9lr0/M+2jZ7/TTObCqwD9jGzkWb2DzP71MzKzex1M9s/tt1/gSHAtdH2G9P21yf23EPM7B0zWxd9cZwbr1vUEn2umd1uZmvMbJ6ZndWYuotIcsysH/BN4CZ3r4yXRY9vBvY3s75N2Hc34DlgLbA3sBuwCHg6KgPoAfwL2A/YEfgb8IiZjU7b3VnAe8BY4LzY+suAG6JtpwD3m1mPBkK7HPgV8H/AckLCbVHMuxG+L24CxgD/AH6TU8VFYpQgS7M09yDt7iuAvxNai+N+CFQBD0ePS4C5hER6O+DCaPlRM+M/GbgYOD/a7y+j+yc2chcvAD8H1hC6NwwCro3K/gJ8BTgC+CJwL/C4me2Qto8rgHOB0YSW+J7A44QfGF8CHgMei7XefIfwZXVh9HpD6qjbzsCD0fIF4NdR3U5Ke+rZwBuEL53fA9dE24pI4RoJGPBuHeUzo/KRdZTX54ho22Pcfbq7v0c4JvYADgRw92nu/id3f9vdP3T3ywjHkUPS9vW8u18ZPeeD2Ppr3X1StO48oB8hsa3PBe7+XBTPxYRjZur49zPgP+5+hbvPcvfbgEebUHcRQAmyNF8+DtK3A3tY7b7GxwL3pZJud1/r7hPcfYq7f+zu9wO3At9vZvznAz9397+5+0fu/hhwJXBKYzZ29/WE5Nij7g2L3b0iqsuhwCHu/qK7z3b364GngBPSdnOhuz/l7nPcfZm7v+Hut0RfPB+4+8XAdGB89JorCC3zZdHrfVpHeD8Hnnb3i6MvjLsJyfs5ac97wt1vjr7ArgU+Br7WmPqLSOLq6kKQOlO0X3QmKrXs2Yh97gRsDZSltiOMbegLjAAws+5mdmXUPWJl9JyxwLC0fU2t4zWmx+4vjG4HNhBXfduMBl5Le/6rDexPpE4dkw5AikZDB+n1ZvYD4JZY2f7u/iLwDPARISn+lZntQui/fHStHZmdChwDDAe6ErpmzG5qwGY2CBgM3GFmt8WKOgLVTd1vZCdC3Wel9dboDPwn7bm1vkCi04wTgG8RWog7Al3IPPg3ZDvCac+4/wK/NrNusRb/6WnPWUjDX1QikqwPCMfd7Qln4dJtB2wEbgQeiK1f0Ih9dyDMIHFElrIV0e3VhLOHZ0exVBLOmqUPxKuo4zU2pO64u0fHyYYa7TbE7qe+c1LbGHV/D4nkTAmyNFdjD9IfEZLZ+C/6BfDZwfHPwMlm9mvgJ8A0d3899cQoub6a0Cr6CqHV9mfAAfXEtomaBD0lPggudWA9nsyWhuYeaDsQkuydyEy2K9Mep3+BXEtowf0F8GH0/HvJ/OJpSF1fGOnrNmQp19klkQLm7ivM7N/AKWZ2bbyLW9RP+FTgUXdfTe4z27xBODu3zN3rmqFnD+Av7v636DW7EFqXkxpU/S6Q3jVMXcWkyZQgS7PkeJAGKKtjV38mtJoeSmi1OC+tfA/gJXe/Obb/zzUQ3lLCabe4eB+3hcCnwDbufm8D+6rPekIf6bg3onUDo1byXOwBTHT3R+Czv+M2wNsNvGa6mdG+0vf9SXp/cRFpk04FXiYMnjuf0GAxgjAAbgOhEaE+/c0svd/vEsIP8rMJYx8uJIz/GAocBPwp6jc8C/iumT0WvdZFhDNdSbkB+K+Z/YLQWLMX8N0E45E2Tq1Ekg+nEpK1p83sa2Y21MzGEfrbNuYgjbvPB54kDOorJRyg42YBY83sG9EsDxMIA+Dq82y0zdFm9jkzOxfYJfaaTkjKzzWz081sWzPbIXp+ej/d+nwM9Ijq3t/Murr7u4TTmn+xMJ3R1mb2ZTP7pZkd3MD+ZgHfM7MvmVlqcF/nLK+5l4WpmDarYz/XAPua2QVmNsrMfgicSehjLSJtnLt/ROj3O4MwI9DHhNknNgFj3H1xA7s4HHgzbTkr+gG9FzAHeIgwC8VdhD7IK6NtzyIk0y8SZrN4JbqfCHd/mXA28GeEbmMHEwZAr00qJmnj3F2LlmYvwJaE6c4WELoUOOFg2TeHfXwv2u7eLGWdCa3MqwgH6NsIU/h8GHvOpcBbadtdAiwmnGK8kXDATH/OUYQvhrWE/nUvAofVE+dxwKrYYyMMGFwWxX9+tL4TYaT1HEKL7yLCjBRfisr3jZ7fJ23/WxGS+0rCFaXOBP4N3B57zlcILcrrgI117Y8wovyd6PXnEmbLsFj5fOCMtNf/L3Bd0v9TWrRoyX0Bfhp93g9KOpakF0J3tbeTjkNL21zMXX3aJf/M7KeEFsxDPcwMISIircDMDiN0tbjO3auSjqe1RN0rngLKCQ0G1wLneZidRyQnSpClxbTXg7SIiLQ+M3sAGAf0JgwMvwW43pXoSBMoQRYRERERidEgPRERERGRmKKa5q1///6+1VZbJR2GiEizvP7668vcfUC+96tjpIgUg5Y6RsYVVYK81VZbMXVqXVe1FBFpG8zsk5bYr46RIlIMWuoYGacuFiIiIiIiMUqQRURERERilCCLiIiIiMQoQRYRERERiVGCLCIiIiISowRZRERERCRGCbKIiIiISIwSZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVEREREYpQgi4iIiIjEKEEWEREREYlRgiwiIiIiEqMEWUREREQkRgmyiIiIiEiMEmQRERERkRglyCIiIiIiMUqQRUREErZpE6xdG+5v3AgrVoT7ZWXw4YfJxSXSXnVMOgAREZH2zB0GDYIlS+p+zpQpMHYszJ0LCxbANtvA5pu3Xowi7Y1akEVERFrJ7ruDWVieeCKsmzSp/uQY4MtfDtsMHx72scUW4fH3vw+VlS0ft0h7owRZRESkFbzzDrz8cs3jb30rJLkHHdT0fd5/P3TvDvPnNz8+EamhBFlERKSFrVkD559f/3MeeCD0RXYPS1kZzJkT1vfqVf+2Q4eGbUQkPxJJkM2ss5ndYWafmFmZmb1pZvvHyvcxs/fMrNLMnjOz4UnEKSIi0hzXXx9aiXv3hsceC+tuvBFeeqn28x55BA47LDw3pUcP2HrrsH716pAAL1oUkui5c+G552r3Q+7QAfbdV10uRPIhqUF6HYF5wN7AXOAA4EEz+wJQDjwCHAdMAi4BHgB2TSZUERGRxquqgj/8ARYvhmuvzSw/8kjo2xeWL4fTT4cDD4Tvfrdx+95ii3A7dGhYFi0KiXHKM8+ELhdlZSHBFpGmSSRBdvcKYEJs1T/N7CNgJ2AzYIa7PwRgZhOAZWY22t3fa+1YRUREGuuJJ0Lf4rpMmBCSY4B+/eDuu5v3emawfj106lR7fc+eIVneYw+48EI4/ngYMaLu/TzzTOgf/ctfZu5LpD0qiD7IZrY5MAqYAWwPTEuVRcn07Gi9iIhIXk2aBIMHh2Tz1VfDujPOCInlpk2N20dlJfziF9mT48suCy297nDRRfmLO6W0NMT57rvwwx/WrN9nH+jcGa64Aj73OXj22TBbxqOPwimnQEVFeN7f/x66ZlxwAXTrBg8/3Ph6ixQr84R79ZtZKfAvYLa7n2hmdwBL3f1Xsef8D7jN3Sdm2f4E4ASAYcOG7fTJJ5+0TuAiIi3EzF5397F52peOkfV4+mnYb7/a626/HY47rubxW2/BjjvWvY+NG8Mguqqq2uunTYOPP4bvfCdv4TZow4bcWoBLS8M26UaNgvffz19cIvmUz2NkXRJtQTazDsDdwHrgtGh1OZA+XrcXUJZtH+5+q7uPdfexAwYMaLFYRUTaIh0jM334ITz4YGjRPeGEzPJ4cgwwZgy89lrd+9t779rJsVlogf3iF1s3OYaQ8LqHxDylvguKZEuOAWbNCvU46aT8xPXqq3DvvfnZl0hrSCxBNjMD7gA2B8a7e+pjOgPYMfa87sCIaL2IiEizjBwJhx8eBrd99FFYt+229W+zyy4hYUy1qq5bF/r+nnpqzYwU220X+iBXVdWejSIJw4eH5HfTpjBY8LXXQqKfzU47hVkyKitDd5O4W24JdVm4sPb6l18OXTXiVq+GPfesuRBKfNl1VzjqqJrHxx0HH3wAP/lJmMv52mtDH+n07b7+9drdPaqrwwDEs88OyXt6q71IviTWxcLM/gSMAfZ19/LY+gHAh8CxwOPAb4C93b3BWSzGjh3rU6dObaGIRURaR0udPmyPx0j3kGi5w8UXw4oVcMMNtZ8zcGDoI7xuXeiDC3DmmXD11VBS0vjXmj07XAK6kC1YEOpbWhpadJcsCf2t4wn94sXh0tfZDBwYBhmmd7/45jfh3/9uubjrs912cNNNob/1NtvAJZfAzJmhL/VmmyUTk7Ss1uhikcgsFtG8xicC64DFVvPJPNHd7zWz8cCNwD3Aq8ARScQpIiJtV6p/cefOYcaIRYtql5eWwsknwzXXhNbkrl1DIl1dXZMYu4dp2B5/vP7Xuuuuwk+OAYYMqbn/gx9kf84WW4R6z50bWqLjlizJflnshpLj00+HPn3gN7/JLd7GePdd+NrXMtf371/78XvvNXymoDnKy2HGDOjSpf4+69I2JNLFwt0/cXdz9y7u3iO23BuVP+3uo929q7uPc/ePk4hTRETartTgu3XrMpPj558PXSSuvx46pjUVpbca//OfMG9e3a9TUQE/+lHz4y00w4aFRPmf/8wsiyfacb/6Vc2VAOPLddeFKe7cw6DGo4+GceNg2bLQLaW8PHObbBc86dQJ9t8/zDP91FO51Wf06JquGy+8kNu2ELpzLF6cuf6++2D8+DC13q67hq4sqdd5991Q3zff1JUO25rEZ7HIp/Z4+lBEio+6WDTfzJmwfR2Tg26xRehq0CHHJqKXXw6ty+vXh32bhZbp9AS7WLmHvsiDBtX+2338cbji3157hR8eLWHTprrfrzlzwnv697+HH0KnnRZauQ84AN55p+F9/+53Yf7n1Mns+fPhz38OU/L94Q9hf3XZe+/c6/zCC6Hf94YN4SzG3/8eYu3TJ7f9tIS1a0O//Ntvh3POCd1pSkpy/6y0tNboYqEEWUSkwChBbr7DDoOHHqq9bocdQjLTq1f7SWoleP11GNvAJ6pDh+bN/7zHHmHg4pQpIeHNVdeuobV5xowwkLQhS5eGbbp3r0nun3gidAk67riQ4NfV0p/NJ5+EVv34DCjpzML84IceWvcP0NagBDlHPXv29J122qnWusMOO4xTTjmFyspKDsjyH/vjH/+YH//4xyxbtoxDDjkko/zkk0/m8MMPZ968efwwPgN75Oc//znf/va3ef/99znxxBMzys8//3z23Xdf3nrrLc4444yM8t/+9rfsvvvuvPTSS5x33nkZ5ddddx1jxozh6aef5tJLL80ov+WWW9h2222ZNGkS11xzTUb53XffzRZbDGXixEe5/faH2bixO9XVndm0qQubNnXmtNPOoaSkJ8899yqvvPIm7qVs2lSKe0c2bSrlW986GPdS3n77XT76aC7uJbh3iG5L2G23PaiuhtmzP2LJkuVAB9zDT80OHTry+c/vwKZNMG/efFavXgMYYLgbHTt2ZOutR0StEgupqEidTwuf9I4dOzF06FAAFi1aRFXV2lp169SpM4MHDwbC9uvXr69V3qVLFzbffIto/wvYuHFjrfKuXbsyYMBAABYsmE91dfVnZe5G9+7d2Gyz0Ilt/vx5bNpU+7PSo0d3+vULI0Dmzp2b8bfv1asnffr0xX0T8+bNzyjv3bs3vXv3prp6IwsWLMwo79u3Dz179mLjxg0sXLgoo7xfv3706NGD9evXsXjxpxnl/ftvRrdu3Vm3bi2ffprZaXDAgP507dqNqqpKli5dllG++eYD6dy5C5WVFSxbtjyjfIstNqdTp86Ul5ezYsWKjPLBgwfRsWMpZWVrWLlyVUb5kCGDKSnpyOrVq1m9enVG+dChW2LWgVWrVrJmTeYsj8OGDQNgxYrllJdX1Crr0MHYcsvwv7N8+bLY/1ZQUlLCkOibY+nSJRn/W6WlHRk0KPxvLVnyKWvXrqtV3qlTJ7aIrvm7ePHiLP97nRk4MMyttWjRQjZsSP/f68KAAQM54gh4++3xLF9e++/7/PPPt8jBvz0cI8Op+a2YOnUiABdc8ClHH705l1wykw8//DkdO9ae9uDuu+9m6NChPPDAA/zxj3/M2P/DDz9M//79mThxIhMnTswof+KJJ+jWrRs333wzDz74YEb55MmTAbj66qv5Z1pfha5du/Kvf/0LgEsuuYRnnnmmVvlmm23G3/72NwDOPfdcXn755VrlW265Jffccw8AZ5xxBm+99Vat8lGjRnHrrbcCcMIJJzBr1qxa5WPGjOG6664D4KijjmL+/NrHqd12243LL78cgPHjM/9P99lnHy644AIA9t9/f6rSppQ48MADOfvsswEYN24c6ZL636usHMqUKY2/hOHWW1dSXf0wm232CiUlFXzwwVmsXh06Ge+4I2yzzVyWLj2OkpLax4FrrrmNSZNGsnr1m7z88m9ZsOB7zJ///Ua95oABzzFz5hfo378/d9xxF3fe+RClpeWUl4/g9dfvaHTsAIcf/gSvvTaf+fMPYYst/s2gQf+ke3fnmWf+ihmccspt3HrrD6iu7pbTflN69lzB2rVQXd2VTZu6AtCnz0K+/vXBVFXB9OlvsnHjHDp3XsLatYNYsuSrdOmyiS9/uS/dusHChffRs+cdVFVtQXn5KHr3ns64caO4/PLLqaqCww47mrKy2vO3t9QxMk6/odsod2PdugG89lpX3ngDnn56G2bPPpn16/uxfv1mrF/fl40bezB69GZRP67vRkttJ5+curdLtNQWHVuB7aKltprj/dbRUtv//pe6t2XWerz5Zure4KzlNXlT5pDqykpY9Vnelbl9ZWUYsR5k/oyurISa431mfFVVoX9cMDRr+dKlqUfDspZ/+imErv7Zy0N/to51lgelWcsXLEjd65y1vKbPZJes5TU5fbes5TWtCN2jpbbU9FjQI1pqmz07da8XmVObh7log97RUlvNd3nfaKmtZhT9ZtFSV3n/jLLa5QMzyqqqYM2a1KPMSWSrquL/m1tkLV+5MvUo83+zqir8b36a+btGmqGsbCRvvHFbrXX77VfGiBGbM378bK65RnOCCXTrNo+99x732Y+zq676iEsuWUtZWfiOGzHiRnr3nsbvf38se+21Gy+//BbnnXfnZ9uPGXM6EP9xNotLL12f8To9emxiwgSYNGk+b765lBEjbmHLLf/GRx8dz8aN3fn977dl6NABvPzyU1x00S5s3FhznFy69KsMGBDOerzzztHA0U2u7wMP1Pz4mDfvSObNOxKId5s4/rPyXr3eYYcdzqVfvz785jd3M2UK/OMfD/P22+uZP/9Q+vV7jbKybVm/vua4WlbWL+M1V60aTM3vxS9FS43ycnjuudSjI6OlxvLlM5g5Mwy0ray8i9LSFWza1ImuXRfQtesCYJ9c/ww5K6oW5GI8fege5oqcNi109n/vvbC8/372AQzZdOgQ+jb16QO9e4fTMd261V66dg0jvTt3DoMgOnUK90tLw/2SknBKMrWkHqf6JqXfdugQTsXE7zdmgcz7KXXdj8t1fX2SnsdUil+fPqHvZDp1sWi81ICuRYtgyyy/w4voK07agYoK6JHZ3pDhq18N3Tj23LPm/q9/HfqC9+kTGjB22in+Q71hF14YBlI25rtv6dJwhcmpUyH9xM4pp4R5twcNCvNql5SEvvsQBli++mrILZrXQFCk07xJ3dxD8jt5cugrN3ly9lGzEK6ONGJE6GM0aFDNMnhwKOvbN3xQevRQsicixSd+WeUsvTe47LLWjUekubp3D41fRx4ZBu9B+BF9xx3hO71Xr8y+v9kay7beOpyl2rQJHnkE9t035AOhK0QYiHjggeFM33/+A/vsk1ueMGBAmCVmv/3g3HMbfn59gyznzw/9rvfaKzTWPfAA3HhjeP6pp4aBsHfdFR6/+27o490a1IJcANxDIvznP4d/1PRfVQMHhqs4bbddWEaPDnM59s086ywiRUAtyI1zxhlhmrZsNm7M7SIfItJ2FO2FQiT49FOYODFMp1LTHzO0/u69dxhNOm5czdyNIiISuGdPjk8+OcxgoeRYRJpDCXIrcw+dzm+5BR57LLRyQOg7d+yxcMQRSohFRLKpqAhdzrbZJvS7zObmm1s3JhEpTkqQW9HHH4f+NE88ER6XlMB3vgMnnBCuY68WDxGR7MrKQv/LbP70JzjpJLjpptaNSUSKlxLkVrBxY7jM5kUXhc70vXvD2WeHFuPB2Wc3ExGRmN13z77+6afDAKNsg/RERJpKCXILmzIFjj8+TNMGcPjhIVnONrWTiIjUVl0d5uzOdsng884LybGISL4V2NW1i0dlJfzsZ2H2iWnTYPjw0LXi/vuVHIuINEZZWZhvfZttatbFL8iW5cJ6IiJ5oRbkFlBRAd/+drhKTEkJnHVW6F7RPfNiZCIiUodrrslc9+yzYfaf3XfXMVVEWo4S5DyrqAiTb0+eHFqKH38c/u//ko5KRKTtefHF2o+/9a0ww88xxyQTj4i0HzknyGbWGRgMdAWWuvvSvEfVRsWT40GDQgvyttsmHZWISNvz5JOhtRjCVbSGDAnzw4uItIZG9UE2s55mdrKZvQCsBj4E3gEWm9k8M7vNzL7ckoEWuoqK0Lqh5FhEpPH+8Y8wRuPZZ2HmTNhhh3CV0G9+s+Y5u+4aBuN11DlPEWklDR5uzOxM4HxgDvAP4DJgIVAF9AN2APYEnjKzV4CfuvsHLRZxASovD8nxCy+Eadueew5GjUo6KhGRwrZmDRx0ULi/zz7wxS/CjBmZzxs6tHXjEhFpzO/x3YG93T3LJDsAvAbcaWYnAT8B9gbaTYKs5FhEJDfV1WEA82mn1V4/fXrmc885B7p2bZ24RERSGkyQ3f3QxuzI3dcB7e4in6ecUpMcT54MI0cmHZGISOG68074yU8a99x774Ujj2zZeEREslGPrmZ46im4+27o0gWeeUbJsYhIfRYubFxy/Lvfwfe/D8OGtXxMIiLZNDlBNrPNgQOBIYAT+iU/7u6L8xRbQaushJNOCvcnTIDRoxMNR0Sk4A0Zkn39kUfCffeF+zfdFM7MiYgkqUlX0jOzk4Fnga2AJcBSYGvgGTNrF4e2iy+GOXPCoJKzzko6GhGRtqlbtzCNm4hIIWlqC/IZwBh3r4qvNLPLgGkUeV/kadPg6qvDhPW33QalpUlHJCJS2Natq7l/9tnhGArwm9+E6dseeADuuQeOPz6Z+ERE4pqaIDvQlzDVW1y/qKxoVVeHA3h1Nfz0p7DzzklHJCJSeGbPhgEDoFev8HjZspqyq66CQw+F+fNrpnk77LCwiIgUgqYmyGcBk81sBjAvWjcM+DxwZj4CK1Q33QRTpsCWW8JllyUdjYhI4Zk5E7bfvubxmWfWJMJf+EK43XlnNTCISOFqUoLs7k+Y2ZPAzoTLThuwAHjN3avzGF9BmTsXzjsv3L/pJujZM9l4REQKzYYNNd0nUq69NiwQWpVFRApdTgmymXUm9D8eDcwH3gLecvfZLRBbQXEPk9pXVMD48fCd7yQdkYhIYXnqKfj61+t/zqBBrROLiEhz5NqC/CdgP+Bx4BxgLdDdzMqA6e6+V57jKxiPPAKTJoX+dDfckHQ0IiKFo7IS3nyz4eQYas7CiYgUslynefsW8CN3PxFYB3wZOBaoBF7Jc2wFwx0uvDDcv/zycNU8EREJjjgC9tij7vLjjqu5rznjRaQtyLUFuSvwQXR/PdDB3e8ys57A5/IaWQF56aUw6GTzzTUFkYhIukmTsq/32JxGv/sd9OkDHZo0+76ISOvK9VA1h3DlPAiD8raM7v8LOCJfQRWa224Lt8ccozmPRUTibq5j1vsuXWo/3mwzKClp+XhERPIh1wT5QSDVy2wy8JPo/heALtk2aOtWrgwT2EPt04QiIu3dI4/AqafWXjd2bJgG8403kolJRCQfcupi4e7xmX+vBKaY2QqgB3BLPgMrFPfeC2vXwr77wogRSUcjIpK8f/4TRo0KM/rE/ec/sN9+ycQkIpJPTe4N5u7zge2B04GD3f2nuWxvZqeZ2VQzW2dmE2PrtzIzN7Py2HJBU+NsDne49dZw/4QTkohARKSwTJ0K3/42bLvOwCVNAAAe70lEQVRt7fX9+ys5FpHi0dQr6QHg7iuAu5u4+ULgUuAbhMF/6fq4+8amxpYPr74Kb78dJrZPXQVKRKQ9mzEjc90uu4RWZRGRYtGsBLk53P0RADMbS81gv4KSaj0+5hjo1CnZWERECkF5eea6V4p2kk8Raa8KecKdT8xsvpn92cz6t/aLr14N998f7mtwnohIsDHtvF56P2QRkWJQiAnyMsIFSIYDOwE9gXvrerKZnRD1ZZ66dOnSvAVx771QVQVf+xqMHJm33YqItKp8HyPLymrun3gi/PGPzd6liEjByVuCbGbDzKzZ+3P3cnef6u4b3f1T4DTg62bWq47n3+ruY9197IABA5r78tE+4ZZoTg4NzhORtizfx8gLoiHTV14Jf/pTGKMhIlJs8tmC/DEwzcz2yuM+AVLXYrI877dOU6bA9OlhVPbBB7fWq4qIFJ5Vq2DJknB/ypSa9W+/nUw8IiKtIZ+D9I4FtgauAnZp6Mlm1jF6/RKgxMy6ABsJ3SpWES5p3Re4AZjs7qvzGGu9UoPzfvxj6Ny5tV5VRKSwuEPfvuF+RQU8/3xN2TbbJBOTiEhryFuC7O4To7sXNXKT89OeexTwG+B94LfAQGAN8BTw/fxE2bA1a+Cvfw33jz++tV5VRKSwuMP++9c8vvhi6BA753j66a0fk4hIa2lygmxmmwMHAkMI3SAWAo+7++LGbO/uE4AJdRT/talxNdd990FlJYwbF64UJSLSHk2fDk8+WfP4iitq+h8ffHBNy7KISDFqUh9kMzsZeBbYClgCLCV0r3jGzE7JW3QJuPPOcKvBeSLSno0Zk7nupZfC7bhxrRqKiEira2oL8hnAGHeviq80s8uAacDNzQ0sCStWhMuoduqkK+eJSPvlnrnu6KNDqzJAly6tG4+ISGtraoLshAF0VWnr+1Ez60Sb8+KL4Yth112hW7ekoxERScb69Znr7rqr5v4uDQ7DFhFp25qaIJ8FTDazGcC8aN0w4PPAmfkILAmTJ4dbnT4UkfasKtb08cc/wskn1y4fPrx14xERaW1NSpDd/QkzexLYGRhMmKN4AfCau1fnMb5WlUqQv/rVRMMQEUlUKkEeOBBOOikzQdYAPREpdjklyGb2EPBUdGWmajNbAWxJmKc4f9d5TsCKFTBtWpj3eNddk45GRCQZBx8Mc+aE+127JhuLiEhScm1B3otwIRDMbDPgVULr8Toz28fd2+y1lV54oab/sQagiEh7VF0Njz1W87ikJLlYRESSlOs0bz2BRdH98YTLS28G3AZclr+wWp/6H4tIe1dRUftxqiX5179u/VhERJKUa4I8FxgR3T8E+Iu7bwQmAm26Y4L6H4tIe5eeIKdccknN/TfeaJ1YRESSlGsXizuBm8zsCeCrwEmx/bTZidFWrAjze3burOmLRKT9Wb8eysvh8MOzl5tlnxtZRKRY5ZQgu/uVZgbwDeBsd49OwLEz8EmeY2s1zz8fDv677ab+xyLS/my9NSxcmLk+PvexiEh7kvM0b+5+JXBl2urNgfvzElEC1P9YRNqzbMkxwI9+1LpxiIgUiqZeKKSWKGlus9T/WETaqxUrsq9Xciwi7VmDg/TMbOvG7syCoc0LqXUtXx76H3fpAjvvnHQ0IiKta/787Ouvuqp14xARKSSNmcXiZTO7w8x2q+sJZtbXzE4GZgIH5S26VvD88+FW/Y9FpD2aPbv240suCbNZDByYTDwiIoWgMV0sRgO/Bh43s2rgdcJcyGuBvsDnge2A14Az3P3JFoq1Raj/sYi0Z++9V3N/p53g/POTi0VEpFA02ILs7qvc/RfAEOBk4D2gD7A1sBG4C/iSu3+lrSXHoP7HItK+bdoUbseNgylTEg1FRKRgNHqQnrtXAQ9HS1FYtgzeflv9j0Wk/Vq/PtzuuWeY71hERHK/kl5RSfU/3n33cJEQEZH2Zu3acNutzV7qSUQk/3JOkM1sfzN73MxmpmasMLPjzGyf/IfXstT/WETau3Xrwq0aCUREauSUIJvZD4AHgVmEPsilUVEJ8Mv8htbylCCLSHt3/fXhVgmyiEiNXFuQfwkc7+5nEgbopbwCjMlbVK1g6VJ45x3o2lX9j0VEVq1KOgIRkcKRa4I8Eng5y/pyoFfzw2k96n8sIiIiItnkmiAvBEZlWb8XMDvL+oKl7hUi0t5VV9fc798/uThERApNrgnyrcANZvaV6PFQMzsauBL4Y14ja2FKkEWkvduwoeb+kUcmF4eISKFp9DzIAO5+pZn1Bp4CugDPAeuAq939phaIr0WUl8OMGdCpk/ofi0j7lUqQe/QIi4iIBDklyADu/mszu4xwiekOwEx3L897ZC0odWnVUaNCkiwi0h6lLhJSWlr/80RE2pucEmQz+0cd6wFw9+/kIaYWl0qQR49ONg4RkSSlWpCVIIuI1JZrC/LytMelwI7AUOCRvETUCt59N9xut12ycYiIJCnVgqwzaSIiteXaB/mYbOvN7BqgLC8RtQK1IIuIwJo14Xb+/GTjEBEpNDlfaroOtwCn5GlfLU4tyCIicPHFSUcgIlKY8pUgb5un/bS4DRvgww/D/VHZZnQWEWknXnst6QhERApTroP0bkhfBQwC9gfuzFdQLWnOnJAkDx8O3bsnHY2ISHJ6tanrn4qItJ5cB+l9Ie3xJmApcCZtJEFW/2MRkeCQQ+Dtt6FLl6QjEREpLLkO0vtqSwXSWlIJsvofi0h7V1ISbs88M9k4REQKTb76IOfMzE4zs6lmts7MJqaV7WNm75lZpZk9Z2bD8/W6qQF6akEWkfZu+vRwqxZkEZHaGmxBruviINnkeKGQhcClwDeArrHX60+YU/k4YBJwCfAAsGsO+66TWpBFRIKnngq3qZZkEREJGtPFIv3iIHnh7o8AmNlYYMtY0feAGe7+UFQ+AVhmZqPd/b3mvaZakEVEUnr1gpUrYf/9k45ERKSwNJgg13VxkBa0PTAt9voVZjY7Wp+RIJvZCcAJAMOGDat3x4sXh4nx+/aFAQPyGrOISEGq7xiZupLeFlu0dlQiIoUtsT7I9egBrE5btxrome3J7n6ru49197EDGsh64xcIMWt+oCIiha6+Y+S6deFWl5oWEakt12neMLOOwM7AMKDWYdXd/5KHmMqB9Nk5e5GHS1lrijcRkRqpFmQlyCIiteV6oZDRhIFzWxMuElId7WMDsA7IR4I8Azg69prdgRHR+mbRJaZFRGqkEuTOnZONQ0Sk0OTaxeI64HWgN1AJbAeMBd4CxueyIzPraGZdgBKgxMy6RK3TjwI7mNn4qPxCYHpzB+iBWpBFRFLcaxLk0tJkYxERKTS5JshfBi519wrCVfQ6uvsbwC+Ba3Lc1/lAFfAr4Kjo/vnuvpSQbF8GrAR2AY7Icd9ZqQVZRCTYsCHclpZCh0IcjSIikqBc+yAboeUYwiWmhwDvA/OBz+WyI3efAEyoo+xpIK/tvGVlsGBBOJW41Vb53LOISNuzOhoKrf7HIiKZck2Q3wF2BOYArwHnmFk1cDzwYZ5jy6tU94pRozQpvojIrFnhtqIi2ThERApRrgnyZUD36P75wD+B54BlwGF5jCvv1P9YRKRGqovFbrslG4eISCFqVIJsZvu4+zPu/mRqnbvPAT5vZv2Ale7uLRVkPugS0yIiNVID9Hr0SDYOEZFC1NihGU+Z2Rwz+7WZDY4XuPuKQk+OQZeYFhGJ0xzIIiJ1a2yCvD3wCPBT4BMze9zMDjazNtObVy3IIiI1lCCLiNStUQmyu7/r7mcDWwKHAw48BCwwsyvMbNsWjLHZNmyADz4Il5ceNSrpaEREkqcEWUSkbjnNfunuG939EXc/EBgO3AB8D5hpZi+0RID5MGcObNwIw4dDt25JRyMikrzHHw+377yTbBwiIoWoydPDu/tC4GZCkrwK+Eq+gso39T8WEantnnvC7YwZycYhIlKIcp3mDQAz2xc4FjgYWAv8Fbg9j3HllaZ4ExEREZHGanSCbGbDgGOAHxO6V7wAnAA87O5rWyS6PNElpkVEauvWDSor4XM5XQNVRKR9aFQXCzN7inD1vBOB+4FR7j7O3e8p9OQY1IIsIpLu4IPD7UUXJRuHiEghamwLchVhMN7j7l7dgvHknbtakEVE0q1bF247d042DhGRQtSoBNndv5O+zsy+Akx193V5jyqPFi2CsjLo1w/69086GhGRwqAEWUSkbk2exQL4FzAkX4G0lHjrsVmysYiIFIr//S/cKkEWEcnUnAS5TaSb6n8sIpJp5Mhwq4YDEZFMzUmQ2wRdYlpEJNPaaHj15psnG4eISCFqToJ8IvBpvgJpKbpIiIhIplSC3KVLsnGIiBSinC8UYma9gZHAO7SBFmi1IIuIZFqzJtwqQRYRydToBNfMhpnZJGA58CrwJrDMzP5qZgNjzyuYIR9r1sCCBWEQyvDhSUcjIlI4Fi8OtxqkJyKSqVEtyGY2BHgF2ARcCMwkDNL7PHAK8IqZfQnYK1p3RYtEm6OPPw6322wDJSWJhiIiUlA6dw5TvfXpk3QkIiKFp7FdLC4CPgL2dfeq2PpHzexa4D/AP4BdgKPyG2LTLVwYbocU/GR0IiKtx71mHuROnZKNRUSkEDU2QT4A+EFacgyAu1ea2fnAs8DP3P3hfAbYHIsWhdtBg5KNQ0SkkGzcGG47doQOBT+SRESk9TX20DgAmF1P+YdAtbvf2PyQ8ifVgjx4cLJxiIgUErUei4jUr7EJ8hLgc/WUjwQWNz+c/FILsohIpk+jCTorK5ONQ0SkUDU2Qf4XcGm2GSrMrAtwCfBEPgPLB7Ugi4hk+uEPk45ARKSwNbYP8gRgKvChmd0IvAc4sD1hFosS4LCWCLA51IIsIpLp5ZeTjkBEpLA1KkF294VmtjtwM/BbwhRvEJLkfwOnuvvClgmx6VItyEqQRURERKSxGn0lPXf/GDjAzPoS+hwDfODuK1sisOZyr5kIXwmyiEimX/0q6QhERApTzpeajhLi11oglrxasQLWr4fevaFbt6SjEREpDKtW1dwfMya5OEREClnRzoCpAXoiIpkmTKi5379/YmGIiBS0ok2QNUBPRCTT9dfX3N977+TiEBEpZEWbIKsFWUSkfh1z7mQnItI+FG2CrBZkEREREWmKok2Q1YIsIlK3v/wl6QhERApXwSbIZjbZzNaaWXm0vJ/L9mpBFhHJNGpUuN1552TjEBEpZAWbIEdOc/ce0bJtLhuqBVlEJNPGjeG2pCTZOEREClmhJ8hNphZkEZFMc+aEWw3QExGpW6EnyJeb2TIz+5+Zjcv2BDM7wcymmtnUpUuXAuEqerrMtIhI7WPkp58u/2y95kAWEalbISfI5wDbAEOAW4FJZjYi/Unufqu7j3X3sQMGDABg5cpwFb1evaB791aNWUSkoMSPkf36bQaE7hU9eiQcmIhIASvYBNndX3X3Mndf5+53Af8DDmjMtup/LCKSadOmcLvllsnGISJS6Ao2Qc7CAWvME9X/WEQkk3u47dQp2ThERApdQSbIZtbHzL5hZl3MrKOZ/QDYC3iyMdurBVlEJFMqQe7cOdk4REQKXaGOYy4FLgVGA9XAe8DB7t6ouZDVgiwikkktyCIijVOQCbK7LwW+3NTt1YIsIpJpw4Zwu3ZtsnGIiBS6guxi0VxqQRYRyWTRKI6qqmTjEBEpdEWZIKsFWUQkU6qLxfbbJxuHiEihK8oEWS3IIiKZVq4Mt+qDLCJSv6JLkHUVPRGR7JYvr30rIiLZFV2CvGoVrFsHPXvqSlEiItk8/3zSEYiIFLaiS5DV/1hEREREmqPoEmT1PxYRqd/NNycdgYhIYSu6BFktyCIi2aWmeTv22GTjEBEpdEWXIKsFWUQku9Q0byUlycYhIlLoii5B1gwWIiL1U4IsIlK/okuQUy3I6mIhIpJdqquFiIhkV3QJslqQRUTqptZjEZGGFV2CrBZkEZG6KUEWEWlY0SXIakEWEalbh6I76ouI5F9RHSqrq2Ht2nAFvZ49k45GRKTwqAVZRKRhRZUgb9gQbtV6LCKSXUVF0hGIiBS+okyQ1f9YRERERJqqKBNktSCLiIiISFMVVYK8fn24VQuyiEh2AwcmHYGISOErqgRZLcgiIvU788ykIxARKXxFmSCrBVlEJDsdH0VEGlaUCbJakEVEshszJukIREQKX1EmyGohERHJrmPHpCMQESl8RZkgqwVZRCS70tKkIxARKXxFlSBv2gTdu+sqeiIidVGCLCLSsKJKkCG0HpslHYWISGFSgiwi0rCiS5DV/1hEpG7qgywi0rCiS5DV/1hEpG5qQRYRaVjRJchqQRYRqVuvXklHICJS+IouQVYLsohI3dTFQkSkYUWXIKsFWUQkuw5Fd8QXEWkZRXe4VAuyiEh2muFHRKRxii5BVguyiEh2SpBFRBqn6BJktSCLiIiISHMUbIJsZv3M7FEzqzCzT8zsyIa26dBBI7RFROqiFmQRkcYp5PHMNwHrgc2BMcDjZjbN3WfUtUFpqb4ARETqouOjiEjjFGQLspl1B8YDF7h7ubv/F/gH8MP6ttME+CIidVOCLCLSOIXagjwKqHb3WbF104C9059oZicAJ0QP15nZO60QXyHpDyxLOogEtMd6q87tx7b52pGOke32f6g91lt1bj/ydoysS6EmyD2A1WnrVgM905/o7rcCtwKY2VR3H9vy4RWO9lhnaJ/1Vp3bDzObmq996RjZ/uoM7bPeqnP7kc9jZF0KsosFUA6kD7frBZQlEIuIiIiItCOFmiDPAjqa2cjYuh2BOgfoiYiIiIjkQ0EmyO5eATwCXGxm3c3sK8BBwN0NbHpriwdXeNpjnaF91lt1bj9aqt7t8e/ZHusM7bPeqnP70eL1Nndv6ddoEjPrB9wJ7AcsB37l7vclG5WIiIiIFLuCTZBFRERERJJQkF0sRERERESSogRZRERERCSmKBJkM+tnZo+aWYWZfWJmRyYdU76Z2WlmNtXM1pnZxLSyfczsPTOrNLPnzGx4QmHmlZl1NrM7ove0zMzeNLP9Y+XFWu97zGyRma0xs1lmdlysrCjrnGJmI81srZndE1t3ZPQ/UGFmf4/GJxQFM5sc1bc8Wt6PleWt3sVwjGzO8SDa9s7oM7XYzM5K23fBf65y/Ww09J63hc+VmR1hZu9GMc42sz2j9UX5XpvZVmb2hJmtjGK/0cw6RmVjzOz1KO7XzWxMbDszsyvMbHm0XGlWc93M+rZtbdbEXKY572tD29bJ3dv8AvwVeIBwgZE9CBcV2T7puPJcx+8BBwN/BCbG1veP6nso0AW4Cngl6XjzVOfuwARgK8KPuQMJc2FvVeT13h7oHN0fDSwGdirmOsfq/h/gReCe2N+iDNgr+nzfB9yfdJx5rO9k4Lg6/gfyVu9iOEY253gAXB79X/UFtos+U9+MytrE5yrXz0Z973lb+FwRBuh/Auwavd9DoqVo32vgCWBiFNsWwNvAz4BO0d/iTKBztO4ToFO03YnA+8CW0d9oJnBSVFbvtgnUsUm5THPe1/q2rTfWpP8h8vDH7g6sB0bF1t0N/C7p2Fqovpem/VOdALyU9veoAkYnHWsL1X86ML691JtwOc1FwGHFXmfgCOBBQhKUSgJ+C9wXe86I6PPeM+l481TnyWRPkPNW72I+Rjb2eAAsAL4eK7+EKCFsC5+rXD8bDb3nbeFzBbwE/CTL+qJ9r4F3gQNij68CbgG+HtXLYmVzqUkQXwJOiJX9hChBbGjbBOuaUy7TnPe1vm3rW4qhi8UooNrdZ8XWTSP8Qm4PtifUF/hsDunZFGH9zWxzwvs9gyKvt5ndbGaVwHuEBPkJirjOZtYLuBj4eVpRep1nE33xt150Le5yM1tmZv8zs3HRunzWuyiPkY09HphZX2BwvJza9S/oz1UTPxsNvecF/bkysxJgLDDAzD40s/lRd4OuFPF7DVwPHGFm3cxsCLA/8G9CfNM9yu4i06mjXmTWub5tC0WLvK+N2LZOxZAg9yA0rcetJvyKbg/aRf3NrBS4F7jL3d+jyOvt7qcQ6rIn4aI56yjuOl8C3OHu89LWF3OdAc4BtiGcFr0VmGRmI8hvvYvub5jj8aBH7HF6GQ1sWwia8tloqE6FXufNgVLgEMIxcAzwJeB8ivu9fp6QuK0B5gNTgb+T+/u5GugR9UMu9DqntNT72tC2dSqGBLkc6JW2rhehf1V7UPT1N7MOhNOD64HTotVFX293r3b3/xL6lZ1MkdY5GjCyL3BtluKirHOKu7/q7mXuvs7d7wL+BxxAfutdVH/DJhwPymOP08sa2jZRzfhsNFSngq1zpCq6/YO7L3L3ZcDvafiz0Zbf6w7Ak4QGke6EfrV9gSvI/f3sBZRHrcYFW+c0LfW+NrRtnYohQZ4FdDSzkbF1OxJOu7UHMwj1BcDMuhP6kxVF/aNfwHcQWhTGu/uGqKio652mIzV1K8Y6jyMMtJprZouBs4HxZvYGmXXehjDQZFbmboqCA0Z+6100x8imHA/cfSWhm9KOsV3F61/In6txNO2z0dB7XtCfq+g9m0/4PKQr1ve6HzAUuDH6wbwc+DPhR8EM4IvxmSmAL1JHvcisc33bFooWeV8bsW3dkuykncfO3vcTRux2B75CGxyh3Yg6diSMzryc0HrSJVo3IKrv+GjdFRTQqNw81PtPwCtAj7T1RVlvYCBhQE4PoAT4BlABHFTEde5GGLGdWq4GHo7qmzrduGf0+b6HAhtt34x694ne39Rn+QfRe71tvutdLMfIph4PgN8RTl/3JcwMs4iaAU4F+7lqzmejvve8LXyuCP2up0THxL6EWQguKdb3OopvDvCr6HjQB3iU0JUoNRPF6YQfMqdRexaLkwgD/IYQ+tvOIHMWi6zbJlDHJuUyzXlf69u23liT/ofI0x+8H6GfTgVhdOaRScfUAnWcQPg1HV8mRGX7EgZzVRFGxW+VdLx5qvPwqJ5rCadJUssPirXe0Qf9eWAV4QvsbeD4WHnR1TnL32AC0Uj96PGR0ee6AngM6Jd0jHl8r6cQTvWtIiR++7VEvYvhGNmc4wEhMbgz+kx9CpyVtu828bnK5bPR0Hte6J8rQh/km6PPxmLgBqBLMb/XhL7Wk4GVwDLgIWBgVPYl4PUo7jeAL8W2M+BKYEW0XEntWSvq3Dah/+Gcc5nmvK8NbVvXYtHGIiIiIiJCcfRBFhERERHJGyXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFmkmcxsgpm9k3QcIiKFSMdIaYs0D7K0KWY2Eejv7gfG77fSa28FfAR82d2nxtb3ADp7uDSoiEhidIwUyY+OSQcgkjQz6whUexN/Lbp76opeIiJFR8dIaY/UxULaJDObABwNfMvMPFrGRWVDzOx+M1sZLY+b2cj4tmb2jpn92MxmA+uA7mb2TTN7MdpmhZk9aWbbxV72o+h2SvR6k+P7i+2/g5ldYGbzzGydmb1tZgfFyreKth9vZk+ZWaWZzTSz/WLPKTWzG8xsYbSPeWb2u7z/IUWkKOkYKdI8SpClrboaeBB4GhgULS+ZWTfgOWAtsDewG7AIeDoqS9kaOBI4FNgxen534DpgZ2AcsBqYZGadom12jm6/Gb3e9+qI7XTgF8A5wBeAR4FHzGxM2vMuA26IXn8KcH90KhLgZ8B3gSOAkcDhwPsN/1lERAAdI0WaRV0spE1y93IzqwLWufvi1HozOwow4JjU6UAzOxFYAhxI+MIA6AT80N0/je32b/HXMLNjgDWEg/5/gaVR0fL4a2ZxNnC1u98XPb7QzPaK1h8Ve9617j4peq3zgB8BY6LXGg7MAl6M6jEXeKn+v4qISKBjpEjzqAVZis1OhJaPMjMrN7NyQitHX2BE7Hnz0w78mNkIM7vPzGab2RrgU8JnZFhjX9zMegGDgf+lFf0X+Hzauumx+wuj24HR7UTCF8EsM7vJzL5lZvq8ikhz6Rgp0ghqQZZi0wF4i3DaLd2K2P2KLOWTgAXAidHtRmAmoSUlV9kGs6Sv2/BZgbubGUQ/Wt39DQsjwr8JfA24C5hmZvu5+6YmxCMiAjpGijSKEmRpy9YDJWnr3gC+Dyxz91WN3ZGZbQZsB5zq7s9F6/6P2p+R9dFt+mt+xt3XmNlCYA/g2VjRHoQvkkZz9zLgIeAhC9M1vQJ8jnBaUUSkITpGijSREmRpyz4G9jezbYHlhNOE9xL6sT1mZhcS+qUNBQ4C/uTuH9Sxr5XAMuB4M5sHDAGuIrSQpCwBqoBvmNnHwFp3X51lX1cBF5vZB8DrhD51exJObTaKmZ1FGDjzFqEV5UhCX7/5jd2HiLR7H6NjpEiTqL+OtGW3Ae8CUwmDQ77i7pXAXsAcQsvCe4RTb30JB/isolNyhwNfBN4BbgIuIExvlHrORsLI6eMI/eEeq2N3NxC+AK6M9vVdYLy7v5VD3coIo7xfI7T4jAH2j+onItIYOkaKNJGupCciIiIiEqMWZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIzP8D281N+Dou0q8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "true_Q_value = history1[-1, 0, 0]\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharey=True)\n",
    "axes[0].set_ylabel(\"Q-Value$(s_0, a_0)$\", fontsize=14)\n",
    "axes[0].set_title(\"Q-Value Iteration\", fontsize=14)\n",
    "axes[1].set_title(\"Q-Learning\", fontsize=14)\n",
    "for ax, width, history in zip(axes, (50, 10000), (history1, history2)):\n",
    "    ax.plot([0, width], [true_Q_value, true_Q_value], \"k--\")\n",
    "    ax.plot(np.arange(width), history[:, 0, 0], \"b-\", linewidth=2)\n",
    "    ax.set_xlabel(\"Iterations\", fontsize=14)\n",
    "    ax.axis([0, width, 0, 24])\n",
    "\n",
    "save_fig(\"q_value_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the DQN. Given a state, it will estimate, for each possible action, the sum of discounted future rewards it can expect after it plays that action (but before it sees its outcome):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "input_shape = [4] # == env.observation_space.shape\n",
    "n_outputs = 2 # == env.action_space.n\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(32, activation=\"elu\", input_shape=input_shape),\n",
    "    keras.layers.Dense(32, activation=\"elu\"),\n",
    "    keras.layers.Dense(n_outputs)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select an action using this DQN, we just pick the action with the largest predicted Q-value. However, to ensure that the agent explores the environment, we choose a random action with probability `epsilon`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epsilon_greedy_policy(state, epsilon=0):\n",
    "    if np.random.rand() < epsilon:\n",
    "        return np.random.randint(2)\n",
    "    else:\n",
    "        Q_values = model.predict(state[np.newaxis])\n",
    "        return np.argmax(Q_values[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also need a replay memory. It will contain the agent's experiences, in the form of tuples: `(obs, action, reward, next_obs, done)`. We can use the `deque` class for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's create a function to sample experiences from the replay memory. It will return 5 NumPy arrays: `[obs, actions, rewards, next_obs, dones]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_experiences(batch_size):\n",
    "    indices = np.random.randint(len(replay_memory), size=batch_size)\n",
    "    batch = [replay_memory[index] for index in indices]\n",
    "    states, actions, rewards, next_states, dones = [\n",
    "        np.array([experience[field_index] for experience in batch])\n",
    "        for field_index in range(5)]\n",
    "    return states, actions, rewards, next_states, dones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a function that will use the DQN to play one step, and record its experience in the replay memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, state, epsilon):\n",
    "    action = epsilon_greedy_policy(state, epsilon)\n",
    "    next_state, reward, done, info = env.step(action)\n",
    "    replay_memory.append((state, action, reward, next_state, done))\n",
    "    return next_state, reward, done, info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's create a function that will sample some experiences from the replay memory and perform a training step:\n",
    "\n",
    "**Note**: the first 3 releases of the 2nd edition were missing the `reshape()` operation which converts `target_Q_values` to a column vector (this is required by the `loss_fn()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=1e-3)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    max_next_Q_values = np.max(next_Q_values, axis=1)\n",
    "    target_Q_values = (rewards +\n",
    "                       (1 - dones) * discount_rate * max_next_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's train the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = [] \n",
    "best_score = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 19, eps: 0.0108"
     ]
    }
   ],
   "source": [
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step) # Not shown in the book\n",
    "    if step > best_score: # Not shown\n",
    "        best_weights = model.get_weights() # Not shown\n",
    "        best_score = step # Not shown\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\") # Not shown\n",
    "    if episode > 50:\n",
    "        training_step(batch_size)\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure dqn_rewards_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXm4I2WZ9u+nkpycvXcauoE+7A0oqDSIMoC4jPsyog58Ku6M46COfjMOfgOKIjjuyiWDosjiAAqKqKAwgOyL0DSbDc3WG72f0326z56tnu+PqrfyplKVVJKqpJLz/K7rXCepqlS9Wareu56VmBmCIAiCIAidhNHqAQiCIAiCIISNCBxBEARBEDoOETiCIAiCIHQcInAEQRAEQeg4ROAIgiAIgtBxiMARBEEQBKHjEIEjCIIgCELHIQJHEARBEISOQwSOIAiCIAgdR7LVA4iShQsX8tDQUKuHIQiCIAhCSDz66KMjzLyo2nYdLXCGhoawcuXKVg9DEARBEISQIKINQbYTF5UgCIIgCB2HCBxBEARBEDoOETiCIAiCIHQcInAEQRAEQeg4miZwiChNRJcR0QYiGieix4jordr6NxDRGiKaIqI7iWiZ67W/IKIxItpGRF9s1rgFQRAEQWg/mmnBSQJ4CcDJAOYAOBfAdUQ0REQLAdxgL5sPYCWAX2uvPQ/AIQCWATgFwJeI6C3NG7ogCIIgCO1E09LEmXkSllBR3ERE6wAcA2ABgNXMfD0AENF5AEaIaDkzrwFwBoCPMfMogFEi+hmAjwK4pVnjFwRBEAShfWhZHRwiWgzgUACrAfwzgCfUOmaeJKIXARxJRNsBLNHX24/f47PfMwGcCQD7779/NIMXBCHWzOQKuOSuF/GZUw5COplo9XB80cfJDPzXn9dgbCaHkw5ZhPe8cmmrhxcqL+2awu3PbAcAPLV5DwCAQDjp0IV4eN0uLN97AB9+zVALR9gafn7vWiwaSOPBF3ciWzCRTibwxTcdikUD6VYPre1picAhohSAqwFcycxriKgfwLBrsz0ABgD0a8/d68pg5ksBXAoAK1as4DDHLQhCe3DpPWvxozuex0B3Ep888cBWD8cXfZzHLJuHKx5YDwBYvXms4wTO6T97CJtGpwEA/ekk5vamMDadw29XbQIApJPGrBM4U9k8vnHzMwCAnlQCc3pS2DY2g9cctADvOnpJi0fX/jQ9i4qIDAC/BJAFcJa9eALAoGvTQQDj9jq41qt1giAIZUznCgCATN5s8Ugqo48zb1r3Y3N6Uihw592b7ZnOOY9PO3Y/3Pcfr8fNnzsRL1s6iGOH5iGTNzE+k6uwh85j/ciU8/jKjx+Ha888HgDAHfj9t4KmChwiIgCXAVgM4FRmVr/m1QCO1rbrA3AQrLicUQBb9fX249VNGbQgCG1HO84PuYIlxtJJA6bZhm+gBgyDAAD7ze/FTZ89ER863kqa3T6WaeWwms6GnZMAgJs++3c47oD5sD8WmO34A44hzbbgXALgcADvZOZpbfnvALyMiE4lom4AXwHwpB1gDABXATiHiOYR0XIAnwJwRRPHLQhCG0LU6hEEJ1+wJrWupNGRFhwd9/ey10A3AGDH2EwLRtM61tkCZ2hhHwDAsD8YM96Gx7ahmXVwlgH4JwCvALCNiCbsvw8y8zCAUwFcAGAUwKsBnKa9/KsAXgSwAcDdAL7DzJJBJQiCJ4z2Ewh5e1brShodfwdvuBTO4kEroHb7+OwSOOtHJrGwP43+tBUOS2LBCZVmpolvAOB7P8XMtwNY7rMuA+Dj9p8gCEIgyP+SEwv00SkLTjqZQCbX2bfwiTKBY1lwtu2ZXS6qLbtnsO+8Hue5En6ib8JBWjUIgiDEABVk3JU0UOj0GByX7uxLJzGQTmL7LHNR5Qom0sniNOy4qEThhIIIHEEQOo82mx+YuSTIuCNjcLS3RB7BUYsG0xgen10WHOZSd10xyLhFA+owROAIgtCxxD3IWB+fstrMiiwqjy+mryuJqWy+BaNpHSYzDG0WJrHghIoIHEEQhBbDrMfgdKYFR39HbhcVAHSnDMx0eOyRmwKzpwVH6uCEgwgcQRA6jnacHnJaFlXHx+B4KJzuVAIz+UILRtM6TJeLqmjBadWIOgsROIIgdCwx91CVoGdRzUYXVXcqMessOMxcYs2SQn/hIgJHEISOox1N/Hk9Bqf9hl8T3i6qBGZys82Cw2LBiRAROIIgdCxxDzLWyRc0F1UbCrRa8LTgJI1ZJ3AKZmlGmcTghIsIHEEQhBahFyLMd3gWlT5p+8bgzDKBU+6ikiyqMBGBIwhCx9Fu8wOj2GxzdlhwypfNxiwqkxkJQ7fgiIsqTETgCILQscS9VYNOwbTu5pOGAebOdlP4BhnnCx39vt2UZ1Gp5bPnM4gSETiCIAgxIFdgJA3DuaPv5FRxvyBjZiCTnz1WHNPkkjgx6UUVLiJwBEHoONplfiAnqNQKMk4mqChwOmyW09+NV6uG7lQCADq+0aiOO4vKSRPvYHHbTETgCILQsbRVFpXJSBpUIno6lYRnkLE1Hc2mYn8mQ2JwIkQEjiAIHUc7ioO8aSKVMJCgWeqiSloWnNmUSWVyqYtKYnDCRQSOIAhCi9Dn+XzByqjpVBeVTiUX1WzKpHJ3EyeyLHizKdA6SpoqcIjoLCJaSUQZIrpCW/5BIprQ/qaIiInoGHv9eUSUc21zYDPHLgiCECW5AiOVMIpuio624Pi7qKZnkQVHZc7pGETiogqJZltwtgD4BoBf6AuZ+Wpm7ld/AD4DYC2AVdpmv9a3Yea1zRu2IAjtBLdNmHGRgukKMu6wWU43Sni5qHpSs9NF5Y5HMkhcVGGRbObBmPkGACCiFQD2rbDpRwBcxWKnEwShAbxcIXGEwciZ1mRnzAIXlVeQcXpWCpzy3yiJBSc0YheDQ0TLAJwE4CrXqncS0S4iWk1E/9yCoQmCIERGvmAiZRSDjM0ODkXxjsGxs6hmVQyOl4tKYnDCInYCB8AZAO5l5nXasusAHA5gEYBPAfgKEZ3u9WIiOtOO81k5PDwc/WgFQYgd7TY/MFsuqWSCirVQ2u1N1IBfoT8AyMyiNPGCqw4OoGJwOve7byZxFThX6guY+Wlm3sLMBWZ+AMCPALzP68XMfCkzr2DmFYsWLWrCcAVBiCtxd1CpaYyZ7UrGmouqw/wUelyUX6sGYJa5qEw/gdOiAXUYsRI4RHQCgCUAflNlU0b8r12CIAgVUSKmwIy8aSKp1cHp5Lt4T4GTtLOosrNH4LjTxAGrFk4nf/fNpNlp4kki6gaQAJAgom4i0gOdPwLgt8w87nrdu4loHlkcB+BzAH7fvJELgiCEj7pTL5hwLDidmkWlNz71clF12QInV+is910J0zMGh9rOxRpXmm3BOQfANICzAXzIfnwOANjC5wNwuadsTgPwAoBxWMHH32Jmr+0EQRDaBhVMyswomHYdHKMzLTjVXFSphC1wOjm62kWB2fm+FZImHh7NThM/D8B5PutmAMz1WecZUCwIguCFIxxaPI5qOC4qk5EvmEikk1qrhlaOLFoMj1vrpD3R52eVBae8X5oEGYdHrGJwBEEQwiTu6bbKC2WyqmRMsA0ZHeei0r8KrzRx5ZrLd9j7rgQzO4JWIXVwwkMEjiAIHUvM9Y1zp27aLqqkYTiTfyffxbsndcCa2JMGId/JpisXpkeQsdTBCQ8ROIIgdBxqeoi7SNAFTs40kUjQrM2iAoBkgmaVBce3F9Xs0XiRIgJHEISOZe3wJF7YMV59wxahRIwVg8NIdXAWlY5XFhUApAwDuVliwVFWGre7ToKMw0MEjiAIHcuvV76EN37/nlYPwxc9BidfsOrgdG4WVRG/HmHJBM2aIGP13bv7ckkMTniIwBEEoeNoF21g2jOZaTLyph1kPBuyqHwsOMmEgfws8c8oAVvmojJKU+qF+hGBIwiC0CL0GJy8003cWtfJLiqvbuIA7CDjzn3fOqavi0oK/YWFCBxBEDqOdrkDdioZMyNXMJE0Zkerhoouqg4WdjrKUCXNNqNDBI4gCEKLUC4qZqvAXSrRuc02dSTIuChgE65Z2OpF1YIBdSAicARB6Dja5Qa4JIvKNJEwDOeOvuPu4rW3UzFNfJa5qMSCEx0icARB6DjaZXpQd+p500SuwOhOGU58SidPcv4xOLMpyNj675UmLoX+wkEEjiAIHYd7gojrhFGwxzWZKQAAelKJWZFF5WPAmWUxOD5ZVFLoLzRE4AiC0HG4J4hMPp4zhhJeU9k8AKCnKzErsqh8XVSzMIvKuw7O7PgMokYEjiAIHYd7goirwFFCTFlwupOJjnVR6Zlt/jE4synI2PrvXcm4BQPqQETgCILQcbgniEyu0JqBVKHgsuB0d+kuqs6d5XyzqGaRi4r9Cv0Rxdal2m6IwBEEoeNwTxBxteCocU5mlQWnc1s16BiVgoxniQWn4JtF1dnffTMRgSMIQsdR7qKKpwVHGSsmM1oMTqemiWv4uahSCUJu1sTgWP/LtJ70ogqNQAKHiE4moldrzz9KRPcR0U+JqD/owYjoLCJaSUQZIrpCWz5ERExEE9rfudr6NBH9gojGiGgbEX0x6DEFQZh9uCeImVw8rQLKDTWVnV1ZVH4uqoRBHe2a0ylmUYkFJyqCWnB+CGBvACCiwwD8FMCTAF4D4Ds1HG8LgG8A+IXP+rnM3G//na8tPw/AIQCWATgFwJeI6C01HFcQhFlE+1hwSsfZnSpmUZkdNtHrb7VikPEsyZFmx4IjvaiiIqjAOQjAU/bjUwHcxsyfAfApAO8MejBmvoGZbwSws6ZRAmcAOJ+ZR5n5GQA/A/DRGvchCMIswT1BZGJqwXGPsztVzKIqdPAs51cHJzUL08QN1ywsFpzwCCpwGEDCfvwGALfYj7cBWBDieDYQ0SYiupyIFgIAEc0DsATAE9p2TwA40msHRHSm7QZbOTw8HOLQBEFoF9olTdztjumZJVlUvpWMExJkLHVwwiOowHkEwLlE9GEAJwL4s718CJbIaZQRAMfCckEdA2AAwNX2OhXjs0fbfo+9TRnMfCkzr2DmFYsWLQphaIIgtBvuCWImpmniZS6q2ZJFVSnIuIOFnQ5XzKJqxYg6j2TA7f4VwDUA3g3gAmZ+0V7+fgAPNDoIZp4AsNJ+up2IzgKwlYgGAUzYywcBzGiPxxs9riAInUlZHZyYWnDcGqanK+G40zrNgqMLNj8XVcKgWWPBMSvE4BRmSRxS1AQSOMz8NwBHeaz6NwBR3BqpM4GYeZSItgI4GsBt9vKjAayO4LiCIHQA5XVw4mnBccfZdCcTyOWtZR2mb0rej3+rBmPWFPozKxT6myUfQeQEteB4wswz1bcqQkRJ+5gJAAki6gaQh+WW2g3geQDzAFwE4C5mVm6pqwCcQ0QrASyGFdz8sUbGLghC59IuaeK6VaPLdk91YhaVW3AmKrioZkuQsbLQuVs1kAQZh4avwCGidQACfcrMfGDA450D4Kva8w8B+BqAZwFcCGAvAGOwLDWna9t9FcAlADYAmAbwLWa+BYIgCB60T5p48XFPysrj6MQsKre7rVKaeH6WuGfU1+sOuBYLTnhUsuD8WHvcD+CLAB4G8KC97DUAjgPwvaAHY+bzYNW08eLaCq/LAPi4/ScIglCR8l5U8Zw0dStNd8oy3RgdmEXlfivkk96SMqxKxsxcZtnoNPxdVOUWL6E+fAUOMzvCxa46/C1mvlDfhoi+DJ90bUEQhFbhniBmYmvBKY7TbcHpJBeV26JWyYIDWOIumeh0gWP99woyFhdVOARNE38vgOs8ll8P4F3hDUcQBKFx3BNEXPsb6RqmWwkc6jwXVbnA8d5OibvZEGhcjMEpXU5EmCVeusgJKnAmAbzOY/nrAEyFNRhBEIQwcE8Q2ZimiZe6qCyBY3SgBSdoDE4qMXsEjrIylsfgSJBxWATNovoBgIuJaAWAh+xlxwP4CPxjagRBEFpCuQUnpgJHG2dfOuE8NqjTLDilzyuliQOYFbVwKrmoOuirbylB6+B8m4jWA/g8gA/Yi58B8BFm9nJdCYIgtAz3BNEWAqereDlOGJ2VSeO2Rvm5qJQFJ64uxTBR371b6xmGWHDCoqrAsWvX/D2AO0TMCILQDtQag/OHJ7ZgTk8KJx/a3PYu+rzfny5ejg2ijnJR1RpkPBtSxdX3K72ooqNqDA4z5wHcAJ/eT4IgCHFgIpPHxXe+gILJZRNEtooF53PXPoaP/OLhKIfnSamLqtSC09Fp4tWCjGeFBcf6Ly6q6AgaZPwEgIOjHIggCEIjfPfWZ/GdW5/FzU9tLZtQc3ENMvYTOEQYncrhVw9vbMWwQkd/nwaVV+9VzKYgY9MJMi5dLkHG4RFU4JwH4HtE9B4i2o+I5ut/EY5PEAQhENNZq9bNZCZfVgcnaAzOjrEZ3PPccOhj80P3xPTrQcYG4berNuHsG57Cmm1jTRtPVJQKHP/6NrMryNi7VYNUMg6PoFlUN9v/b0Bp+waynyfKXiEIgtBEUkkVoGqWW3ACujzee8kD2DQ6jfX/9fawh+eJPvHrMThdyeK9ZyeEo+jutkoCZzYFGbOPi0p6UYVHUIFzSqSjEARBaJCUbevP5s2aY3AUm0anQx9XJfxcVPN7uzA8ngHQGS0b9K+jUgcG5zucBRacghNkXLpcYnDCI2ia+N1RD0QQBKERUk4GDpdZcOLq8iiUuKiKl+N5fSnncVzbTNSCLtLche10erosZ4ByN3YyxV5UUugvKoJacAAARLQEwP4AuvTlzHxPmIMSBEGolaQ9cebypkcMTm0TRrOaPbKfBaeveInthMk+aAyOqgU0lc1HPqZWI72ooieQwLGFzTUAToIVc6NibxQSgyMIQsvI5AvORGHF4DRWyZi5sislLHxdVLrAyXWWwKn0ufbaFpypDhB11VDi1nCl+pAEGYdGUAvODwEUABwB4BEAbwGwGMDXAXwhmqEJgiAE47BzbnEe5zxcVLXGdDRrftFdN/2uGBzFTEcInOLjShac3vTsseAUKrio3BZIoT6CCpyTAbydmdcQEQMYZub7iSgD4HwAt0U2QkEQhBrIh2DBMZmRQDNcVMXHei+qznZR+W/Xm5o9FpzKLqoWDKgDCVoHpwfAiP14F4C97MdPAzgq7EEJgiDUS67A5b2o8rXG4IQ4oAroE/9AuhhYPLe3s1xUQYOMe9OzR+A4LqqyLCoJMg6LoAJnDYDl9uPHAXyaiJYB+BcAm4MejIjOIqKVRJQhoiu05ccT0W1EtIuIhonoeiLaR1t/HhHliGhC+zsw6HEFQZg9ZF0WHIPqs+A0A71jeK9mwdHr4HSCwClNE/cXOF0JAwmDZoWLyi+LijqsD1krCeqi+hGAve3HXwdwC4DTAWQAfKSG420B8A0Ab4ZlFVLMA3ApgFsB5AH8GMDlsGJ9FL9m5g/VcCxBEGYhOVcdnO5UIrZ1VUwG3v2KJXjTEYudNHegmC4NADMdYM0oLfTnvx0RobcrMSssOOonKb2ooiNoHZyrtceriGgIlkVnIzOP+L3OYz83AAARrQCwr7b8z/p2RPRjAFJ7RxCEmsmbXFL9N500araCNMuCw8xYOrcH7zhqScnykw9ZhH9/82H4zq3PdoQFJ2iaOGBlUk1l2v89V6PYqqF0ubiowiOQi0p3FwEAM08x86paxE2NnARgtWvZO20X1moi+me/FxLRmbYbbOXwcPN6ygiCEA+yhdI6OOlkoo46OGGPypuCyZ4TvmEQ/uWUgzG/r2vWCZy+riQms3mce+PfsHL9rqiH1jLUb9Qdk2QYEmQcFkFjcDYT0bNE9FMiOt0teMKEiI4C8BUA/64tvg7A4QAWAfgUgK8Q0eler2fmS5l5BTOvWLRoUVTDFAQhplguquLz7pSBgsk1tTxo1h20ydaE5kdPKoHpbDzda7UwPlOMqXHXfXHT05XAS7um8MuHNuCiv7wQ8chah18WlfSiCo+gAudQAN8B0Afg2ygVPKeFNRgiOhjAnwF8npnvVcuZ+Wlm3sLMBWZ+AFZM0PvCOq4gCJ2D1aqh1IID+Acae9Ucacb04pdFo9OdMjqiDs7IRBYAsGggHciC8/RWq4P6fc8PY8f4TOTjawWmbxaVxOCERSCBw8wvMPPPmflDzLwfgCMBPADg4wCurvzqYNhZWbcDOJ+Zf1ltSEATilQIgtB2uLuJp1OGs9wLL8MON8FoUmy2WLk3Uye4qHZOWI1D9wogcHq6ii5Fk4E/PrE18vG1ApUp5c4qkxic8Agag2MQ0XFE9B9E9GcAfwXwOlji5uNBD0ZESSLqhtXaIUFE3faypQD+AuBiZv6Jx+veTUTzyOI4AJ8D8PugxxUEYfaQzZvQbTDdjgXHe9Lwcl1xE2w46rAVm0+mEh1iwcmgO2WgP52s2gJDFTxMGoTD9xnEjY8FrkTSVvh9/9KLKjyCponvBjAD4GYAvwLwaWbeUMfxzgHwVe35hwB8DdbV6EAAXyUiZz0z99sPTwPwCwBpAJsAfIuZr6zj+IIgdDgZVwyOqinjb8Epn0yaEeTpl0Wj051KYCLT/jVhRiayWNifRsIgJKpZcFLWtLTf/F685ci98YPbn8N0tlCSOt8J+LmopBdVeAQVOE8BOAbAcQAmAUwQ0WStWVTMfB6A83xWf63C6zwDigVBENzM5AquGBxL4FiWnXK8BE4zegGp41aa8HtSCQyPZyIfS9SMTGQcgVPNRbVr0nq/Ry4ZxN5z0gCAnZMZ7NvVG/k4m4kSMV4uKqB5He07maAxOCfAKsb3eVjWnC8AeImIniSiH0U4PkEQhJrI5s2SSrDddn+jvM9tsZeLqhl30EFjcDrBRTU8bgkcIqrqojr+wAUAgC+/7XDMs1tWjE7moh5i0zFNHwuOHV4qXqrGCWrBATNPA7idiP4Gq0bN2wH8I6yA489HMzxBEITamMkVSiaHdDUXlcfipsTgqEq21dLEO0DgjExk8cr952LrnpmqFpwzTzoQH3ntELpTCWzbMw0A2DWVbcYwm4rpVweHiusNyaVpiEACh4jeD+AU++9QANsB3APgswDujGx0giAINZJxtWpQWVR+LqqCp4sqmrF5HTdRJQan3buJF0zGrknLgpPJmVX7ghGRY3VTFhzltuokfLuJ2wpH4nAaJ6gF5yJYrRN+BOAuZl4T3ZAEQRCC446XsWJwis+r1cHxzKJqYpBxJQtOdyqBmVx7F/obncrCZGBhfxqfOulAFGqoKj2/Twkcy0VVMBnMjGQiaAm3+OIXZE5Uul6on6C9qCKrXCwIgtAIboGSM7nEsF+sg+M9YXhnUTXDRRUgBsduFJovmG07qY/YNXAW9qcx2J2q6bWD3SkkDHIsOF//42o8u30cvzrzNaGPs5mYJvt+/+q56JvGCRyDQ0SLAXwYwEEAzmXmESI6AcAWZl4X1QAFQRAq4TbA5AomkppVpC4LTnjD86XgE4Oh09NliZqZvIn+dhU441b8zML+rppfaxiEge4kLr7zRQx2p/DoxlG8sGMCpskVLV9xJlcw8Zpv/sURfu4sOkMsOKERtNDfMQCeBfBBAJ8AMGivehOAC6IZmiAIQnXcEwFzqbWmu1oMjlcWVRMCIJxCb1UsOADaOg7HseAMpOt6/e4pyz110R3PY8PIFGZyJra3cfuGXZNZ5zMBvLqJqxgcETiNEvSW4LsAfsTMrwSgR3vdCuCE0EclCIIQkGoTQV+XZaie8hEJrZpIiqX6/bdRwbbtnCruCJy++gSOYl5fF8btoofrRiYbHlerUJ/HK/efi+9/4OiyWjdEEmQcFkEFzjEAvCoHbwWwOLzhCIIg1Ea1LuGDPZbAmfSpCOzZi6qJdXAqu6hsC04bCpyXdk3hrT+6F9+4+RkkDXK+h1r51ZnHY15vCptGp51lG3ZOhTXMprPTbjz65bcejve+at+y9XqhP6ExggqcaViF/twsB7AjvOEIgiDUhlcdGx0V2OrX8sC70F/0k0ugGJw2dlG9sGMCz9hdwfNm/VV5jz9wAT50/LKSZevb2IKza9ISOAt8YpIMseCERlCB83tYfaKUjZGJaAjAtwD8NoJxCYIgBKKaGBmwBY6/Bac1QcbqDr1aFhXQni6qapa1Wlg6t8d5vO+8Hmzd074xONVcdkrv/vSeF5s1pI4lqMD5NwDzAQwD6AVwH4AXYLVtOCeaoQmCIFTHq1CfTlfSQFfSwEQ2ZhYcVcm4gsDpbmMXlfoM33fMvrjgH17W0L4W9Fti4Ih9BrF4sLskSLfd2DWZreyys38PP717bRNH1ZkErYMzBuDviOj1AF4FSxitYubboxycIAhCNaqJEYOA/nTS14LTqkJ/xRgc/23a2YKjvpePn3AAjlgyWGXryhw3NB8nHrIQX3vXkfjWLWvaOsh450QW8/u6fF12z28fdx5Lw83GqGrBIaIUEf2ViA5j5r8w83eZ+dsibgRBiAPVYnAMIvSlE5jMBM+iamY38SAuqna04CgLVaUYo6DM6U3hl594NQ5c1I+F/em27rC+czLjVGj24lQt8HiyDWOv4kRVgcPMOQAHoDluaUEQhJqobsEh9HUlawoybsbFzq/Zoo6TRZVtv3YNxfcX7n4X9qcxOpWr2tMqruyczGJhv3/K/NH7zcW3Tz0KAHDdIy/5/m6F6gT96V0J4FNRDkQQBKEeqgWzGgYw0J3ExEzwIOPmxOBUt+B0t7EFp9hrKVwXiyoYqLKR2o2dE1nfDCrFnF4rMP7rNz2Nr/5+dTOG1ZEEFTh9AM4koseJ6DIiukj/C3owIjqLiFYSUYaIrnCtewMRrSGiKSK6k4iWaevSRPQLIhojom1E9MWgxxQEobOppkUSROhLJzHpG2Rc+z7DIFizTbtVQxsLnEqVmuthkS0O2tVNtWsyW9FFBRS7qAPA9rH2zRhrNUErLx0OYJX9+EDXulouBVsAfAPAmwE4eX9EtBDADQA+CeCPAM4H8GsAx9ubnAfgEADLAOwN4E4iepqZb6nh2IIgdCDVsqjIFjgbd3kXh2t1FlUlAdCVMGBQe9bBCZIlVg/KvdNOmVQ7xmbQ3ZVAV8LARCZf0UUFAHN7i01JB7rrK5AoBM+iOiWMgzHzDQBARCtThwzSAAAgAElEQVQA6CUc3wtgNTNfb68/D8AIES1n5jUAzgDwMWYeBTBKRD8D8FEAInAEYZYTKIuqyz+LyjvIOJShVaQYZOy/DRGhJ5VoaxeVEXIMzl4D3QDay7Jx3IV3wCDg+k+/FgCqWnDm9hQFTn9aBE69xKU97ZEAnlBPmHkSwIsAjiSieQCW6Ovtx0c2dYSCIMQSvTGmV8BuwrBdVDVlUYU3Pj/UuKt1xe7palOBEyDGqB6WzutBTyqBZ7dNhLrfqDEZOPWSBwAAC6oInDmaBWcqV2hLC14ciIvA6Qewx7VsD4ABex1c69W6MojoTDvOZ+Xw8HDoAxUEIV7oLqqkh1gwiNCfTmAym/dM//bOoopHqwbACjSeacMJzumWHkKauE7CIBy69wDWbBsLdb/NZEEVF1U6mXAe3/zkVhz+lVvaNmuslcRF4EwAcFeCGgQwbq+Da71aVwYzX8rMK5h5xaJFi0IfqCAI8UKvg9PlkZNMBPSlk2D27ijunUUV6hA9UceoZuHoShjItuHkVnCyqMLf9/LFA3hm61jbNKRcOrenpN1ENQuOF/c+LzfstRIXgbMawNHqCRH1ATgIVlzOKKyu5Udr2x9tv0YQhFmOLlBSyfJLWsIgdNnLve6CvbOomlDoz6wegwMAqYTRlnfv6v2FnUUFAIftPYDRqRxGJtojVdxkxgkHL3CeV0sTB4CbPvt3OGrfOc7zm5/cFsnYOhlfgUNEfyGiufbjM7RGm3VDREki6gaQAJAgom4iSgL4HYCXEdGp9vqvAHjSDjAGgKsAnENE84hoOayaPFc0Oh5BENofM4CLKmVbdrwsId5ZVCEO0Idiq4bKAiCVJOQL7WGp0AlSyLBe5thBuFM+qf9xw2QusdQFCRx+2dI5OGxxMRJj4672bU/RKipZcE6A1VgTAC4HMKfCtkE5B8A0gLMBfMh+fA4zDwM4FcAFAEYBvBrAadrrvgor6HgDgLsBfEdSxAVBAEoFSsrHRZVKWJOLl1DwzsJqXgxONRdVql1dVGY0hf6AomhqhhANA5Otz+GO/3syLv4/rwr8mQx0F4ON28VaFScqycg1AC4kojsBEIAPEJFnVBczXxXkYMx8HqyaNl7rbgew3GddBsDH7T9BEAQHfZLr8nJRESFp5yp7CZxWWXA4oIUjZbSni0rpxigsOEofVKtiHRdMk2EQcNCifhy0qL/6C2wKWoDZSJsWNmwllQTOPwP4EYB3w7qd+S9439YwLBeSIAhC0ymJwUl4u6iS9vKcR2fOVqWJBy2El0oSZnLtJ3AKAer81IsSTe0SZGwy1yX0VLPNpXN7sHn3NGZyBad9h1AdXxcVMz/AzMcy8zxYFpwDmXnA48+d/SQIgtA0SuvgFC9pKh5Hj8HxDjJuUSXjgM0oUwkD+Ta04ATpll4vap/VqlgrrnxgPR7dMBr6OIJicn2fg2rRsWyBFS3SDtWbRyYyOPfGv2FsJtfqoQTOojoAgOSoCYIQO6pNcoZRjM0J6qJqaqG/QDE47WGp0Imq0J++Tw+DnCc/uP05/O6xTaGPIyimyXWly//HW5bjTUcsxmnH7Q+gPeJwrvnrRvzyoQ248bHNrR5KMIHDzBsA7EVEXyei3xDR9UT0NSJaHPH4BEEQKuInRtTiEheVyxJyzV83YqdHV+qmpIkHjcFJUFvG4ERV6A8our2CWtoKJnuWA2gWJnNd6fL7ze/Fz85Ygf3n2xacmMfhMLMjbH4XA4ETqMkFEZ0Aq+/TdgAP2os/COALRPRmZn7Q98WCIAgRoltgdGHCmoskpYKMtW2f3z6O//e7pzz32Qx7SaEGC047CpxCwDo/9aBEU9AgY+bWxuuYXL0lRyUW2nVzdk7GW+BMZgtYOzKJxYNpbN8zg/GZXEkmWLMJ2sXruwCuBfBpZjYBgIgMAD8B8D0Ar41meIIgCJXxc1GppUQoWnDyRaFQaW5sarPNqhYco23r4BBFkybuuKhqsuC0UuDU56JSzO21BM7uqdbHtVSiYP9OP3bCATjzxAMbEnVhEFTgvALAR5W4AQBmNono+wAei2RkgiAIAah2Z54wtCBjbZKrNOE0JcjYvppWc120ax2cet0yQTCMGgUOc+CA5ChwF/qrFRUw38r3EAQ1vp5UouXiBggeZLwHVqCxmwMA7A5vOIIgCLVRbe63sqhUob/ixtm8/wubMY0Us4wqb9euMTgFM5oAY0CPwQm2vWlySbZdszG5sZYV6qUx1zdFt2QMxA0QXOD8CsBlRPRBIjqAiIaI6EMAfgbLdSUIgtAS/O7i/+EVSwFYk6Eq9JfTXD2ZvH+H7mZYcGpxUeUqiLG4wswwIup2qMRCULeTydzSqseWBaf+19f6fluFEzgfkbCtlaAuqi/BqoXzC+01OQCXwGq7IAiC0BL87sy//b6j8JV3HgHSLDi6JSRTSTQwMDaTw2CEAZJOL6ogQcYxn9i8KJiNuWUqUYuLim1x0yr3DjODubFYpFqDqltFsb9aiwdiEzRNPMvMnwcwD1Y8zisBzGfmLzBz/BPzBUHoWPyu+cmE4QRnOnVwzGAuqr+u24Wjzvtf/GXN9vAG6kKNu5oFp8t2UbVL1V5Fo26ZStRSB0d9zq1yUYWRLk9EIGqOZbERgmYGNougFhwAADNPAfDOqxQEQWgB7jvzB85+fVlX8WIdHN1F5T87PrphFwDgobW78Prl0ZT7MgOmUScTBpitySPp0YoirjSaOVQJZSEIMuGrbVolDoLGWlUjQRR7C06UHeTroSaBIwiCEDdKa98AS+b2lG3jVcm4kgWnGfNIIeBkUGwzwUi2URuievsvBUG5e4K4nZQoaFWcdlhd1Q2D4p9FZcZL4MTEUyYIglAfQe5qlUUnFzCLqjlp4kEL/fk3Co0zUcbgKNdXELddqy046rCNfhYJopZmggUhyv5j9SACRxCEtibINT+VLG+2WclFpfYZ5WWaA1pwutTY2yyTymSOLF3YabZZSwxOi11UjQbeJgxqaSZYEJzaTmLBEQRBaJwgd7VerRqyFdLEHctAhNdpNRlUu9v1SnFvB0wzwiBje+YKYr0ruqhaHYPT2GdBFP8sKhXEHxcLTuAYHCLqAvAyAHvBJYyY+U8hj0sQBCEQelwC+5ToS3oV+qtw+68mJYpQ4RQCBp96pbi3A4UGa79UwqjFRWW2OMjY/toajcGxLDjxFjhmzCw4QZttvgnAL2GJGzcMoOHQNyKacC3qAfDfzPxZIhoCsA7ApLb+W8x8fqPHFQShvTFdQcZeqBicrJ5FlasgcJqgJUzTEgDVJr4uD/daOxCliypRQ+sCtU2rQpiKxe8a2087ZFEVQnLHhUVQC87FAG4CcD6sjuKhf8rM3K8eE1GffZzrXZvNZeZ82McWBKF9CeKiIiIkDardghPhjWjQLCM9i6qdMKMs9FdDqwb1XbYqAyloxepqGG1gwWnXOjj7ALiQmTdEORiN9wHYAeDeJh1PEIQ2RZ/kKl3+UwmjJAYnkzeRThqewcZNyaJiDuS28MoAawdMjs5VUSz0F8RFpf63RhwUHLHceBZV3C04cauDE9SQdBOA10Y5EBcfAXAVlztYNxDRJiK6nIgWer2QiM4kopVEtHJ4eDj6kQqC0FKCXvSTCUI2b2LPdA6/fXQTsrbAOefth+OV+88t2ValkEepc0wzWLdtlQHWbh3FC5EW+gveqqHQYgtOMU28sf0kDGpZLZ+gBG0/0iyCWnA+DeBqIjoGwN9g9aFyYOarwhoQEe0P4GQAn9AWjwA4FsDjABbAcpldDeDN7tcz86UALgWAFStWxFvuCoLQMEGtLZYFx8S5N/4Nf3hiC16+dA66kgl88sQD8a6jl+C4C+9wtp2x43OitJoUzGB3ul0eRQrbAeZgAq4eimnitQQZRzKU6scPqQGlYQQLqm4l6rOOiwUnqMB5M4A3AHgbgCmUWoIZQGgCB8AZAO5j5nXOAZgnAKy0n24norMAbCWiQWYeC/HYgiC0GUEFjhWDwxidstrnbd0zjbRdGtjtPlCdxqMUOEFbGRRjcGJ+++6iGc02g3z1TqG/FveiCqPQX+wrGbepi+q7AH4MYICZ+5l5QPsbDHlMZwC4sso26luOx6coCELLKInBqTABpBIGcgVGX5d1Xzc6lUPadv+4L8jKglOp2nGjBA0yVinubeeiMhsPrPVD7ba2Vg2tShMPJ2DdaIMYHCfIuM0EzlwAP2HmyapbNgARvRbAUriyp4jo1UR0GBEZRLQAwEUA7mLmPVGORxCE+BP0op+yu3L3pZPO61QKttt9oCw4UYqKQsAYHOWiardKxswcWbpwohYXVWwqGXd+FlVY7riwCPrz+y2AN0Y5EJuPALiBmcddyw8EcAuAcVgxQBkApzdhPIIgxBzdanPKYV6luiySdgxOf7pYtktZcMh1JVSTYrQWnGB3uk6j0JjfvbuxCv1F22yzHXpRheqiivlvIG6tGoLG4KwFcAERnQTgSZQHGX8/jMEw8z/5LL8WwLVhHEMQhM5CXVTvP/v1WDyQ9t0uaRByBUZPV/Gypyw4ykpy1L5z8OSmomE4UoFjBqv0266VjE2Orh6KU+ivDVo1FMJyUbVRFlW71cH5OCzryWtRni7OAEIROIIgCLWi4jCWzOmuWGukK2kgXzBL2jmoIOPuVAK3f/EkpBIGTv7OXc76SLOoAmYZKQtOlGIrCoIKuHqopdCfmnRb5d1RVqaGLThG66xQQYlbHZxAAoeZD4h6IIIgCPXAdjZStUJqyoKTyxcnCWXBAYCD9xrA9rGZktdEGYNjmsFaGTgCJ+637y6CBlHXg1FDHRy1SesqGVv/G/0s2sNFFa9WDTEZhtAqmBlDZ9+Mn92zttVDEYS6CBqsm0wYyBXMEqtMl+tK7N6LLobCJqgA6O+27kMnM+3VpaZgBqvUXA9OJeNaCv21vJt4Y/tppyDjtnJREdFFldYz8+fCGY7QbNRJf8GfnsGnTjqwxaMRhNoJGuuRShBmcibyWtfFdMolcNzZVJG6qIKNu68rAYOAsen2EjgcYauGYhZV9W3VNa5lrRqcGJzGLThxFzhFC04bCRwAL3c9TwFYbr9+VagjEppKu2VmCIIbq2t19e2ShoF8IY+sZpV5+8v3KdnGPQdFmZodNEaFiDDYk8LYTK76xjGiwIxUZC4q638wF1VrKxmrITZcybiNXFRtZcFh5lPcy4ioG8BlkIaYbU3c7wgEoRqBezolDGQL7Liozn/Py/D3R+5dso37whx5HZyAAmCwO4Wx6TYTOJF2Ew/ebNPJomp5N/HG9mMYxcahcSVuQcZ1f+TMPAPgAgD/Gd5wOpsXhydw3h9Wt8xU6kXc7wgEoRpB660kDULBtGJwDl3cjw8fv6xsG/deoq5kHFQADPYkMTbTbi6q6IKME04MTvVtlbBp+27iRvxbNeRj5qJqNMh4EYD+MAYyG/inXz6KKx5Yj7UjkRaErom43xEIQjU4YMG8RMIy8ecKppOZ5MYtOKLuRRVY4LSjBSfSQn/FY1RDXeNaZa1WLrLZ4KIy29FFRURfdC8CsA+ADwL4U9iD6lSKJ1h8fqRxvyMQhGoUAsayqDTbXIGR9Mtjde0nSgtOrS6qtSMTkY0lCkwzuomOiGBQMKuM2fIsKut/43VwJMi4VoIGGX/W9dwEMAzgcgDfDHVEHUw8vvJS8mLCEdqUO57Zjv3n9wZvWmmb+HMFE10J7+3du4k0Bieg5QmwXVRtlkVlWaii278RMKuo0OIg42LgbWP7aYssqpACqsNCCv01kWL/lBYPREP0jdCufOLKlQCA04/bL1B8g2EQCoXKLir3fqK04HANAmCwu/2yqKIs9AfY32cgF1Wre1GFE4PTDq0aHBdVTCrs1TUMIkoSkcTe1Ij6ecdI34iLSmhL9CaLE5kCelKJCltbJA1C3mRkC1whBqf0ebZgBmroWA9BCxQCwGBPClPZQlv1o4oyiwqwvqsgX42y3LTKReWkiYdQyThOCSpeqPkkGROFU3EURPQGIvqAa9nZACYA7CaiW4hobpQDbGeYGY9u2FV2gYyTpoj7CSMIXkxmC87jDTsnsWxBb9XXqEqwuXwFC47mSE4aBOboJsZCwFYNADBoVzMeb6NMqqDd0uslaOuCQkwsOI1XMo7/DWmhzSw4ZwPYVz0houMAXAjglwC+BOBoSJq4L394YgtOveRB3Pj4ZgBa5L/JeHF4IhYR8XEYgyDUyh4to2jdyCSGFvRVfY2y4ORNE11J79lGNzj0dllWoZmI3FTMwWMVBrpTANBWmVRxicExYxKD07CLqg0sOGp8cYnBqSZwXg7gbu35+wE8wMyfYubvA/gcgHdFNbh255mt4wCALbutBn7q7nDb2DTe9P27cdvT21s2NoVUMhbakT1TxYl+fCaPoYXVBY6hZ1H53GLq1+X5fV0AohMVhYAVmAHLRQWgreJwanHB1YNhBJvwW51FFZqLqg3q4KjxxSWLqtrpNRfADu35CQBu0Z4/AmBp2IPqFKayljlZ3Qmqc33nRBYmA6NT2VYNzSHuUfmC4IV7oh8K4KKyCv0xshVcVHrMiCNwIhIVtcSoKBdVO2VSBa1PVC9W2nT17XRh0woLSFguqnboJm6aDKLGrVVhUU3gbAVwEAAQURrAKwE8qK0fAJCJZmjtz5QdJ6AEjmI6Zy2PMkMjKHE/YQTBiz0uq0qQGJyEUSz05+ui0h4rgaNbi8KkliyjdrXgROuiCljoT9umFRaQsPozGXZMWJwpcLRWu1qpJnD+DODbRPR6AN8CMInS3lNHAXghrMEQ0V1ENENEE/bfs9q6/0NEG4hokohuJKL5YR03KqZtgdPTVZqNr4RPHDIiROAI7Yhb4Mzt7ar6Gl3g1GLBcR8rLCq5ytwogTPeRgIn8jRxokAZbvplthUWa3WJbXTebwcLTsGM1mpXK9XOrq8AmAFwO4CPA/gUM+t+lY8DuC3kMZ3FzP3232EAQERHAvgpgA8DWAxgCsB/h3zc0FEuKvV1K7OdEj5RFhELStxPGEHwwh0X09dVvaSXimHIV0gTL43BSVvHiihzKV/BkuSmHV1UJnOkroqgrQt0UdOKul8cUlxK0Lo/rcSMmQWn4lWBmUcAnEREcwBMMHPBtcn7YaWMR80HAfyRme8BACI6F8AzRDTAzONNOH5dqFRWJ4reXh4rF1XMTxhB8EIXOAYB3anqlpCEbeLPVEoT1y7OCyK34JiBLTh9XUkY1F4uKrOGLLF6SAQsfKfH3bTiehdeq4b4l/Wopf1IMwh0djHzHg9xA2be5bLohME3iWiEiO4notfZy44E8IR23BcBZAEc6n4xEZ1JRCuJaOXw8HDIQ6sNZanJO2mCpcvj4KKK+wkjCF7ooqOvKxnIUqAm22zBRMqnVYPOnN5U2bHCJFfBkuTGMAgDbdZwM+oYHCIEc1HpFpxWxOCEVQeH4m/Bifo7r5WYlONx+A8AB8LKzLoUwB+J6CBYHcv3uLbdAyvIuQRmvpSZVzDzikWLFkU93oooF1XBtouqa3AxBqf1P1ZxUQntiC46erqqVzEGrG7iiiDCoithYKA7GZmoqBTs7MVgTzIyd5mb79y6Blf/dUPdr//+/z6LPdO5yLOoggUZa49bcL1jDinIuA1icKKOu6qVoM02mwIz/1V7eiURnQ7gbbDcYIOuzQcBxNY9BXhYcGwn1XTOukiJi0oQ6mNUy2zqSga7T0satQmchEGY0xOd1aQWFxVg96NqkgXn4jtfBACcduz+dU1YF/3Fyj2JtlVDsDTxEhdVS9PEQ+gmHnOB05YuqhbCsEJXVsOqmgwAIKIDAaQBPNeicQXCHYOjkCBjQaidW/62FTes2gQA2DlZrE6RDHhB1SeYIC6qpEEY7E5F5qKqFOzsRSsabj744s6GXh9tFlUwi0xJHZwWXO7UZT4UgRPzy7VVvVoEThlENJeI3kxE3XYzzw8COAnArQCuBvBOIjqRiPoAfB3ADXEOMAY0C06hNAbHcVHFwYIT9zOmQaazBXz31mcxkysLIRPajE//zyp88TorFG/nRDH0L+gkqguhIFYfx4ITkagIGgukGOxJNiWLSr8mPLN1rKF9RW/BqTGLqiVBxqXX/3pphxicfEEsOH6kAHwDwDCAEQCfBfAeZn6WmVcD+DQsobMDVuzNZ1o10KAoC41vFlUMLDj6CX/dypdaOJJouOrB9fjxnS/gsvvWtXooQkgwM3ZOZp0sp6AX1ESNLqpkgjDYk4zOgmPG04IzkSmKqHreu25ViXKuU3WNqo6HW+uiCitNvC2yqMSC4w0zDzPzscw8wMxzmfl4Zr5NW38NM+/PzH3M/G5m3tXK8VZDP5Gcfk+uOjhxyKLSh/Cl3zzpBEZ3Cupc2zXZ+rYYQjhMZPLI5k0smdsDAEgEjGPRtwvi1koYBvrTKUxmwrf+FUxGwWQka7Dg7De/F9vGZrBtz0zo49HRiwnWI3CmNGtplBaHoDE4+jWuNTE41v+GXVRtYMExJQZndpDJaye5q7pUsQ5O63+s7hNeuc86hV67ANxkprOE22xDdzEqsbpkbjcA6842CPp2QVxUSYMw0J2MpHqwurmpxYLzjqP2ATPwxye2hD4eHd0NVo/FSD/XxiPM+jKMYC4nfZtW6INiq4bG9qNaNQRJjW8VBY5Po01ABE5kZHJFUZN3u6gCWHBe2jWFDTsnIxufokzgRHC32kr605bAmRCB09aMTGS0x0rg1G/BCZpFNdCdxEQmH+qk8tDanU42VFcNAufARf142dJB3Pb09tDG4kWjFhz9XIsy6ysRNAanxYX+nDTxRl1UtgUoznGTYsGZJWS0AGInBscVZFwpTfyrf1iNL9/wVHQDVGNznfCdJgTUZy4WnPZmRAsqVhacpbbACZpFpWuJQDE4BqE/nYTJRatro/zq4Y047dKHcMUD661j1OCiAoDXHLgAj2/ajdHJLF4cjqaIvLK6zOmpL4NMP9eirNtDAevCFFocgxOWi0oJpDi7qQpmvFo1iMCJCN1FVWbByVW34IzP5CI17yrcQWudFoOjiilGEUchNI+R8aIFZ/uYFYOyoN8OMg54QS214ASJwSH02z2gJho8FzfunAIz44e3Pw+gKAJqcVEBwLFD85HNm3jl+bfhDd+7O3R3xfqRSUfULJ3bU5cFpmkWHKN2C04r3Duhuajs33mM9Y0VZBwjC06sCv11Et4WnNIvvpLAyebNpgQhu+9oVO2emVwBBZPRl27vn4j6DDvNMtUO5AomMnnTcRM2gu6i2rx7GoBlYQCsWIwg6EIoiGsoaRjO2MczeewVdLAuNo1O4aTv3ImPnTCEbbY4m7AFdy1p4oAlcHSsTKxwJpSte6bxuu/e5Tzfd14PVm3cXfN+9JuJKLO+rDo41bfTL3Gt6UXlff2vFfWTjb+LqtWjKBKjoXQWJTE4Pi0ZMhVcVNkCN6XSsfuEn7KFwJt/eA+O/OqtkR/fD9Nk5EMQeGofkx1mmWoHPnHlSrwspN/QTi0LbvPoNPq6Eth3Xi8AYMWy+X4vK0GPDUjWEIMDNBYsu9uuunz5/eudZWrir9WCM6+vCwttyxWAUOs7bR6dLnm+77xejE3narZ66C6qV+w3N5SxeRG0LkyrXVTq8A13E6c2cFG1UzdxoX68sqjcJ1dlC06hKXVy/Cw4G3ZORX7sSpx17Sr86altWP9fb29oP0UXlQicZnPPc+E1u53Wsvu27J7Ggv40Dl08gD997kQctndZSzpPSuvgBHRRpS0rUSMuKq/fnnID1SpwAGDZgj4nJmkmZ2Kgu+6hlaBbyQBg4UAXsrYVrjsVrN8XULSWXvPJV+NVy+aFMzgPDCLkA5hwSl1UkQ3H//ghNdtUv98418IpmPFyUYkFJyJ064yKwSkXOP4/1FyBI3NRTWbyGDr7Zlz14PpygZPJl9wVDp19s5OWevdzwxg6+2asG4k+u+tPT20D0LjPXF0A1R345fevw9DZN5dMmM1ix9hMyec5Wwgj7kEP8t28exrz7SJ/RywZrKuScdAsqmIWXv2uFt162NuVwL7zirEt9biX9p3X4zwO04IzPFFaK0q5AGsNNFaC7uj95tYkjGqlXQr9FbuJN96qAYi5iypmFhwROBFRasHxFjiVXFDZvBmZi2rrHssUfcX968uC9CazeTy/vTQ749qHNwIAfv/YZgDAyvXh1Fj81cMbMXT2zRXf52QDQuQDP3kQF/5pDQBLcDIzfnK31URw93TzC/89Z3+u1/x1Y9OPHTVKvN33/AjGZ3IYOvtmZ10lV2xQ9Il8656ZEjdNUPQ7y1rq4ACNuahUvM2Hj1+Gn5+xAn1dSWd/9Vhw0trYw/hsFSPjGRABd/3b6/CbT7/GETi7p2oTOBOZPIgsMRclRMF6S5UU+mtJmrj1P4xu4kDMXVRiwZkdeNXBqclFVTArWngA4PePb8ZZ16yqeWzqgtubTnjWwXH3n3ngxZ34wE8fdNLAGg2WU1z4p2fs8fhPHrun6hciD7uEWLZgOhe7VtwFqWDYVvTDiZpVG0cBAB+67K848dt3lqwLYxJ2p2kv6EvXvI/GLDiNu6g+c8pBeO3BC9HTlXAsOEFigdwsHiz6pMK04IxMZDCvtwtDC/uwYmg+9pljWYo2jdbmrp7I5NHXlQztOuFH4CwqvRdVK9LEQ8qiKrqoGh1RdJhm8KzGZiACJyK8sqjcyrtSjE0ub1aNwXl43S7c8cyOmsc2aouG3q6kRwxOHo9uGPU8liKsVEt17EpCr9a7x0pkbSsOYMUuNBuyFWInChz9Z+T+zjIhTMLuiXx+PRYc7cIbpHZOMkFOFmEYMThqXz2pBMadNPHaJ4PPvO5gvPagBQBKLcU6M7kCPnr5wzU1yxyZyJRYxg5Y2AcAWB8wHu/LNzyJqx5cjy27p7F4sHYBWitBC/21vJt4SC4q9ZON8/WjwFLoryN4bvs4zrpmlSUVLnsAACAASURBVO/krC486aRRl4sqUzBLJmQvZnImMvlCzYJjl+1r7+sqt+BMZvJ4xMcFRQj3h6ssW153oSqNN0yBk8mbYOdx82Nw1DHj7EOvl0oX3TDEpHsfqtFmLehF9YJ2E+9KGkgnjbrTnW98bDO+fcuzAIA+u22I7rqpx0XV05XA595wCIBSS7HOCzsmcNezw2Xn8uX3r/N1kY5MZLGwvyhM5vWmMNidxPqAMXfXPvwSvvL71Vg/MuWIoyixCv1V367VMTiuVoR1Y8S4kvFLu6bwL1evwmQmLy6qTuCL1z2Om57ciqe3eN8hKQtOXzrpK3D8xBFzMcA4r1k5/vN3T5U02svkCzDZSqH9j9886WlG37BzEuf9YXXJsR0LTjrp3F18YMW+WDyYxsZdU1g7MokBj9olbMuDsE6vosAp/xy6U7bAsWNlmBnn3/Q0HnxxZ8V9/vzetfjdY5s812XypnOxi8KCs2rjKH54+3O+69UxY3h9aphK72kmBDE5nS1gbm/Keb6gQQtOsErG1jbL9xnEA1V+d378668fR7ZgoieVcO5sexoUOACc4F2/z1ZdJ9yxQ1/749P4f7/zrpBuWXCKAoeIMLSwD+trbBmzfuckli2IXuAE7a5d4qJqUasGg8KogxNfC/B3bn0WNz+1FWu2jSOkskyhIAKnTtTFzy9NUd1Z9XYlnG3cAsdkbzWeN9kJTFNC577nR3D1XzfinBufAjPj+7c951x4fn7vOvx65Uv4+b1ry/Z11jWP4YoH1peYqlWp+3TCcC4Q33jPy7F4sBvPbhsHABy5dLBsX2EHPav37mVNUU0ylQUnWzBx2X3rcPrPHsJohc7g37j5GXzh1094Cs9s3nTebxhuEzfv/e8H8MPbn/e1qClLVbMuUA++uBO/edRb7IVNRWtkCGJyOlfAvN6iqNHjUIKSrCNNHADe84olWL1lDM9tH6/5mApd1JRacOqbDdQNgJ9QV4kEQWOHrN53U2Up90ML+nDv8yO47pGX8I2bnvbNPtR/85m8iaEmWHCMulxUralk3Kh7Coh3FpX6PQLSbLMjUBdLvyJ+ajLr6/K34ADATU9uKYt50S07auJQZr9M3sSabeO46I7n8bfNY/YxrAumu1CXvi/9/FICJ5MvBt0mDUJvVwKjtqBY0F/uQw8zoLF0v+UXaTUhqCBj/cL6+Kbq1VXveKa8IaGyeAGld763/G0rHn+p9oqtfvgF1U43WeCc/rOH8G/XP9GUY01XKKSoPuu7nt2Be58PXhvnhR3juG7lS9Y+cqUWnHpcIIkag4zVOf6Oo5YgYRButLMIg6K3PdHPnZ5U4xacdDJRtl+drbYFR48d0kWIW6io9/auo5eULH/by/cGAHzpt0/i5/etwy2rt3oez30OH9AEC45hBCv0V1LJuEUuqjAEjtpHHC046oYUAOb01G5djQoROHXwg9uew0pblPgFAmfyJgwC0imjmEXl8cP8/K8ex6mXPFCyTL8bVvtXF9tcwSwrVqXEz1bNfaVQJ4N+YhcFTsEpQmhoGSMAML+3/EeqLmJBLxL3PDeMNdtKLSmbRqfwp6dKL5JeF2l1MVYWnCntglypaJ+K3dnlkX2VyXm7qD79P6vwnovvd57f+Nhm7Bgv/yyD4neX61hwYpwFUS9TFdL51fv+6OWP4MOXPRx4n6dd+hC+9JsnMZMrYMZtwamjul2tAkdtv2ggjb87eCF+//iWmrJw9GKZ+ufTo00G9buorNf97rHNTusKHeWi0i0449pj/TUbdk7ikrtfxOsOW4T95veW7OctL9sH//7mw5znqzePecbouSuF67V6osIgClS4z2yxBYeZA7cTqUTRgtP4vsJGv6k7fJ9ghTebQWwEDhGliegyItpARONE9BgRvdVeN0RETEQT2t+5rRrrVQ+udx77NXHM5AtIJxNIasWoggoDXTSpVHEvoaJQKadb9pRf6NT5PJW1JombntziCJyZnFkS9T7YU7xDnu8RxKkmqlUbRh1XViXO+MXDeMsP7y1Z9o8/fQifuXpVSRsGL4GjRN7u6XKBUymjRX1OXm6srCYO/YKM90zl8K+/fhzXryx37TAz/vjElqoBylM+d9WVLDh3rtmBna5Ksq3kT09trakYYqV6RZmc6TTIrAU1OW/cNYVplwWnnkBG9Ts3KJgZXXdp/cMrl2Lz7mnfAHwv/IJzdRdV0E7obrptC869z4/gH3/6YNn6rR4xOLu0Qn66wLnsvnUomIwL/+HlnsfS+1/9/L51eP9Pyo/nvulYNNCMLCrv6+HweAZ3rtmBu58bxvaxmZIby1bcXJgcjosqzkHGe7S6Ysv3Lg9vaBWxETiw2ka8BOBkAHMAnAvgOiIa0raZy8z99t/5zR+ihV6d06/7diZvIp0ykDQMx40VWODky11UyuKQN7lsglUZHlt3+1twHt0wis9d+xjOuuYxx/pkWXCKdQvmVBM49nGvf3QT3vzDeyq+B7/3qi6sY9qF18ulo5apKqr6ZOtXdC1fMB1r2S6P7Cs/C47OsC0yvKq33vP8CD577WP4/m3+gcTWWL3HN5P17iI/kcnjY1c8gk9dtbLifoPw0q4pPO+KFak1dmrt8AQ+c/WqMktbJaq5qPQyA3eu2VFmCVm9ZU9JAD0AJ+B13cik1ZKgwaadStQEtZroIujvj1yM3q4Ebny8uptq3cgknt8+7pxnbnSBEySbywv9GrTJdk0//tJuDNtd14sxOMXfsW7V3GKfh7mCiZue3Io3HbEYS+Z6W12O2ndOSfAxUF4qQr/RMwhNadLrF4PzD/99Pz52xSP4yC8exrt/fH/Lm20WzHBcVOr3+ODa+gLeo2R0svg7Wy4WnHKYeZKZz2Pm9cxsMvNNANYBOKbVY3OjVxL1u3PN5Eykk0ZJOXF90nd3WM57WG2sx0rgFFOM3ZPz2LQ1uUzbHcB11LPv3Pos/vfp0rgUlVWkzKeD3UWBM89D4FRyQ7jxs0aoz25Uu9hWsuAoYaMLyXEfF9WMNpF7WXCsGJzS1HT3hVpZt/Z4CCT1nryEpM501ltQzLjek2LEnpTcFaTr4cRv34k3/aBUfNbalkJ1vK6lRH8lC85MziwJcv/YFY/g5/etxTNbx5wYq7dfdB9e993SAoEqFXzDzklM5wqOa+e0Y/cLPC4dZS0J0kkcKM166e1K4m0v3we/XbW5zO3q5t+ufwL/9D+P4g9PbMEbDy/vQa6Lk3otOPo1yOqqzXjPxffjPRffD2YuxuBo54p+TqgEhUvvWYtdk1m875h9fY/VnUrgkf98Q8lYlZB6YccEdk9lS85PFR8UNYZBni7DTVos4raxmVIXVUticLjhIn9qPwBw/k1Ph9KIuFFmcgWnwOdu7VrhFsOtJDYCxw0RLQZwKIDV2uINRLSJiC4nooU+rzuTiFYS0crh4fCa/emUWHB8JlvHRZUgzyyqg/fqL9l+WBME3hYcdfdfbsHRJ6Jxd72OCufz7qkcNu6ccjLCdAtOj0cPmZkaJsrtY94CR92x7tTM5TN5E7mCWdLjSllwlFtHr2Tr56LSJ/JdXi6qvFkMMrZFojuGSokYr7on6rXV3Bt+Vj01PrcYUE0OoyoAWmsndTV56ZayqWzeM9ZDUUlEzeQKZVa3n9+7Dm/90b342h+fdiYdt3BXbqi1w5PI5k10pwys++bb8M33ertSqqHuolN1Wk3OfutypJMGfnbPOrw47C1Gp7MFPPHSbqwdnsTweAanvsoSDq87bJGzTUkWVZ1j0V10ScNwROnm3dPYPZVzzh/9XFEd2fvTSTy7bRymybj4zhfw90csxsmHLkIliAj93cWbsltWbwMz443fvxvvveSBkt90OtWcaaWvK4Hd0znM5AqO1dLrxqTVhf6sGJzGT+69tczBnRUySZvF/zy0Ae/97wfwzT8/gw07J/G+Y/bF2gvf1uphlRBLgUNEKQBXA7iSmdcAGAFwLIBlsCw6A/b6Mpj5UmZewcwrFi2qfNLWS1qb/P0CXjP5yhac5a50zO1jGYzP5DCZyWPz7mJwopqA1QUrXzDLLTjaZDzqOsErBdVt3DWFW1Zvc+4udIHjZTqv1I/HHZTrF3Oh7u62aJNlJlfAL+5bh1O+exdWb9mDHWMzzvueyhawY3ymZAL1a3yoW4J0gaNScWc0C44SiTMua8uIsuB4WC/U/quZm6vF4LgFkOoMHVV6ZbVO6pOZfIkwVgJH/5x/cteLJYHYtRwjkzfL0pV32MfYsnu65O5PR03Oz9qTV08qASKqu56IKvRXb2r2wv40DtmrH79dtQlv+N7deHTDrpLf5fB4Bis37HLcpAPdSZyyfC888/W34OdnrHC2G9AspUGtSZVIGFQS76OyMhf0dTnn7I7xGUdIv+agBVizdRwjkxlMZQs44eCFgT5T/VLyld+vxh/sprFrhydLbvTSdYq2WnnjEYsxlS3gfT95AG/6wT342+Y9uObh8iKGtz69zbEqlN0ANoFCSDE4R+83F997/9EA/K+vzWSNHYf507vXYipbwLzeVKyK/AFW3EusICIDwC8BZAGcBQDMPAFABShsJ6KzAGwlokFmDl6LPCS6g7ionBgc8syicteb2D424zmB5PLlLqqyGBxtgrBM/sUUzSB3LGpYusDxmgT8XEOPbtiFUy95EBed/konzVTdUbovduq5bg3I5E3HonPxnS84ncQB4JmtYzjugjvwwVfvD8CaEPxqe+gCR7f4JA0DuUIBE5mC8173TOcssej6LFUgppfAUcuqzUnVsqiUFU6JPTXxhHER9KJaw9Iv3/AUdk/ncNXHjwNQFB/657xp9zSGxzPI5k1P8evuFaVjWXB8RGnedN6/G3X8l3ZZgr+nweaNKtas3swloLT+zqmXPIj95/fini+dglzBxLEX3A7AssT1dSXxjqOWeHbTPmhR8fys10WlkzQI67RifJ+0Y7kO3qsfj9lxOcddcAcA6/w7Ztk83Pb0dqy2a0Ut9Ym9ceN2f//h8S3OY/23Um9cUa289qCFWDyYdsplPLV5D35w23M4csmg894A6/r2g388Gmdd8xge27gbpx23f1PGp7DSxMPZ1yGLLcu/n4W8mbywYwLHLJuHnlQC970wEiijrdnEyoJD1m3EZQAWAziVmf3ktvooWyIX04GCjK0JTFlwmLnkAnHY4lKB4w6wVKh4HMeCY3JZ4bQ9JQInuAVHoSZ5PYsqyF2YElpPbdoDAHh4XTH4bYctcNzBhmq/egO/mVwBfWnrM9XFjc7K9dZd6aKBtK8lyW+SVXfuuzWrzuX3r8e/XLOqTIzsnCwPMlb+bhUvUq15ZDWBA1hNTRVhuaj035fuo/dzoyrWjUyWWAC8XFTqd+XVhZ2ZKxaUy3i4qADguKH52DE248QguVEWHGXh6m4wtqPWIGMv3AUGN9ria1h7D8v3HsTNn/s7nPP2wz33oVf5DcVqR6Up6YpDFvcjmzfxwIsjzrL95vfiiH2sLJc/20HkfsHFbtTv6+vvPhJH7zsH92g1jSYyzY/BSRiEc99xhPP8hlWbkC2Y+MIbD3WWXfaRFbj1X0/CiYcswrFD88oa8DYDDsmCAxR/f6224Jgm49lt4zhq3zn41zdabUP0+SMuxErgALgEwOEA3snMzi0+Eb2aiA4jIoOIFgC4CMBdzLynFYMsseD4pIlPZwtIJ+0sKpPLLCkHLy6NwXnCp9BctmDtX1XezebLrQ56RpJ7AvLLFtKDnJWIKrXgVP9pqElLWaiSWrEHFeTorhisLuh6IOBMruA5QXaVWMqs9XsNpitYcLzfq3IDuN13t67eXvZZKt+2soqtG5nEwf/5Z1y/8iXn9e7JeiZXwIpv3O48r+aiAkonBCVwxqbzDTUy1a0k+v6rWXB2TWZLAlCVu1HfhxJ3XjEOn/vV43hso3+hRC8X1R/OOgErhuZhx3jGsRgBwEV3PI+3X3QvTJMx4bp56G7UgmM05qICrN+fF/qE8+oD5mPZgj7fTCJd1ITRcXsqW8Da4Ukcslc/rv7kq53lByy0rjGf/9XjzrKhBX047oD5GEgncZ1dCmFpwJo1SuAcs2weTj50UUkyxNf++LTz+Kilc+p/MzXyjqOW4PpPvwaD3Uk8sn4USYPw2oMXoK8rga6Egdcv38uxlh93wHysG5l0MsyaRTYfXgPKBX1dMKj1Ame9Hfi/fO8BrBiaj9995rU486QDWzomL2IjcIhoGYB/AvAKANu0ejcfBHAggFsAjAP4G4AMgNNbNdbuADE428cy2Gsg7Vhw3C0dFvYVL5TdKQN3PuvdFTybtwMwbavBZCZflnVUMBkD3aWtDaqNb15fudquFoPjRokANSfrdynP7bCCMKdypc1AlaBwBxR7jXOuNh5lFdkrgAWnzzURKrHmZX3QrS35gukEGY9n8jBNxuotlob+9988qQmR0s94eDxT4mbxTRPXBJj+PY2MW+PKFsyqYsQLZsbQ2Tfjwj89U9ynFsRdLQZndCqL8Uzescg5MTglwjlnb1sucP74xJayZTpKwOrBtUftOxd7z+lGwWQnxgYA/rJmB1ZvGcPIZAbMwH7zi5OvV+B7LYRiwfEoMPjy824tqUau142pNpYwKJiMxzaO4tDFAzjh4GLuRa+HIDxgYS+6Uwm81a5QDJSe9xWPY5/HC/rSOPYA7/f4y08chwt86ulExbFD83GIbRE/cukc9HYlsXAgjYX9XSUC8k1HWO+52u81bNaNTGB/VwHFekkmDCzsTzdd4Hz5hidLqqKrEgiv2n8eAOCV+8/zdMe2mtgIHGbewMzEzN1arZt+Zr6ama9l5gOYuY+Z92HmM5jZ25fRBPS+G3qGyievXOk0ttw2NoOl83rsGByzrMCUHox11NK5nhMHUJ4mPpnNe6Yhq8Ja+n4KJvu6bbwqFQ/21FZhVQkNdeFTLymYjOfsADTmUpeOGrvbguNlCdMLu43N5NCVNDCnJ1U1i2qu672pycSrM7n++cxosUDM1vvTU8LXDk+WvG+FO17HL51+Oltw6gvpNUn0DDq9GBtgiY2jv/a/TjqmF0o4XacVJ3zj9+92HlfKoprJFZzxqs/HKwZHrVu1cRTLz/2zI1CDtO+48sEN2LBzqsy9s5ctFvR4CdUyY7UdVzGkuXO6G8zOicJFBVi/B1UY8uy3LsffH7m46n7u+dIpTsxTGOyczDqJC7d/8WT8+szj8c6jl+B7/7+9846vqj4f//u5Nzshi4QsZgIh7C2ggKBQEVcFJ9a27jr6/apt1a+1P221tVq12oGjVqlaR1u1rdYiLoQyBMEiguwNSSAQsnc+vz/OyL03NyEBQgbP+/W6L7hn3c95cs7nPOeZl47gpWtPY8Zga0xOj6jvnZnV6t9wqhPHR4UyundCUCVt8oDk446VOhYc5fcm24qQHBPeqNhgv6RoRvSK5/WVezjjlx/7ue7aCseVMyjtxBW/S42LaDaj8URTV29498tc/r0ul9q6eh5dsJG7/vYlsREhjbKBOxodRsHpTPj6mH0fZh9+nc/8ZTut6pn1hvT4SMuCU2dcJeCbI9MbTWxDmzHpBqaJ15vgloioMC+xESEU+Tw4mwv8DFQCwP8NuSXZHRf9fikHiitdGThvS07lWcfX72slCbRuJMWEUVlTHzSA2XeMNXWGqDAvMeGhTbqoHAuEr2Lknk+Ixw1W9dunxlf5quNwWbX7IC2urPGbSJx4i8CA2UDFqbkYnPR46wHpuISMMWw7WOqmgBaU+cejfLLxAEUVNcxfujPoMeHo9WrKm3Cjgn+22aHSaqpq69zzcRS5+nrjuqheWraTypp6pj22iEcWbHRr91w4Ip3YiOZzFlIC3DvO92VbGz9orpm/CvDvOZWVfHyTqfNAPp7MpZgmznFTfglej3Dj5MwWKVAZ8ZFMOUpq9tH46/cmct2kfu73HPt+698jhvGZ3YkJD2HOmJ5MyU7m3KGW9cLpEZWZHMOr14/n5etarmS9esMEnrpiJBGhXqLDQxiS3nEq1j5w4WAev3QEs4alAZaiee+sxjFQ03N6sL2gjH1HKnjygy1tPq69hRWUVdc1ypo9Hsb1TWTp1kNMe2xR0DntRPLVviJG/nQhJZW1lFXX0f/H/2beom0ApMVFnhAXa1uiCs4x4PsmGexh66RAZ8RH2nVwDHW2v3pYz/hGE1u/5KYb09UEpImDfw0Zh/AQLwnRYX4WnOaCS4OZr0WER+cM5/3bp7Q41fP99Xmuy6aiuo7y6lo3G2x0n3hrHD5F9XzjU8K8HrpHh1NVW0dZVS3pcf5vxwkBikpUqJeYiBBKq2qZ+eRiFq63jHgPvbvBr7VAoIJTbwzhIR4/t5iD8yAFy5VzuLzatRoUVdQEfVMKtOAUBvS9akqxrKipIy3Oegt2Yn3yi6s4Ul7jvvUHlvd3TNHdY5puYHc0BSfQgvPxxny+/9oXVNXW+Sk4heXVDQG9oR5KKmt4fOEmXli6w6dJacN1+PSibW4BvztmZJMV8Db3w29k+30PtH4MzYjjhsn9mD06gyubyGzxteC0NBi2KdwsqpBjn5SHZ8Rx18yBQdfFR57cNNlxfRM5K6ehkGBzD9EZg1O4ffoAxvq4z07vn8TkAS1XsjLiI7loZIb7/Z5zc/jOxD6AVfn8qStGtmb4J5T+Pboxx6dY4di+iYzP7N5oO1/XWlyQF6ETzRd7LMtrzgm04Fw8yvob7CgoY9Hmtqn1BpbL/vuvfRH05fPcoak8csnwNvvtE0WHSxPvDPj6GncUlHHnX/7LtWc0vEldYvdq6ZkQ6cbgOBacYGmhM4eksu1AKecNT2vU5yXQRQUNVUh9CQ/xEB8Z6vewbcrtBU2nJF9mV4kNVgk4KCLuA7+supYt+aUUVdSQlRzNmD4JvLJiNxXVtTz14RaSu4X7pRLGR4USEeqhssaKwcnqEcN+n2yy+ICutBFhXs4dmsrWAyUs2nSQ99fnMyU7mef/swP+s4P7Lxjst5+Tom8pOF5KKmsJ83qabJC6v6gCY2Bweiwb80rYd6SCfYUV9EqMZM/hBkWntNqKz3EeZoF1XMqr61i6tYB//nc/D88exqb8Ep76cAu5RZVcMa43H2884Mr3a7sq7syhqby+ao9bW8Jhix3L1Fzm1lEVnIAJ6plPt7Nyx2F6JkQy0echcKis2g2M7ZcUw6a8Yn778Va/fQMLKH688QCxESH0CRJjcNtZA0iPj+TOv1i++9QABSfU6+HH51l/s60HSnktSA2TtLjWN9VsCseCE3IcnQ89HuGWqf15dMEmAG6dlkX36HB+9u6Gdim+NqJXPLNHZ5AQFdZsg8tuEaHcPj27yfXHwulZSYzrm0io18ONZ2a6LseOzMhe8YR6hZo60+ZxLBXVdTz83kYyk6Jda/aJYEh6LLdOy+L3n2xja/7RewK2hpq6eu5+80u+NaEPxRU17Cgo46YzMwnzevCIkBQTxu7D5fzfuYM6XM2bYKiCcww4Co6jvLy1Zh8b9jcux5MeH+lmUdX6dO12eOna01i/v5jkbuE8cOGQoNV3q2rr+XhjPu+vzycy1EtFTR0bcxtf1OEhHqLCwv1iWx5buImwEE/wXkRHuTZ9g4yvPK03n2464Kd8OOQXVboum3e/zHUfpo9dOsK1NL25Zh9P22ZNwFUYQjxCeKjXDUINnAQCLTFRYV4GpcUy76oxzJ63lP1HKth6oKGirBOL4uyXEmv5qutNQ3p6WnxE0JRaaIgLGt07gbfW7OPxhZvYnF/KrGGproIzMKUbm/JLKKmsdd8AiwIsOPsKK7jq+c8AuHh0BvMWbWOx/aZ1Wr9EEqJC3Ric9fusIOYh6XFkp8T4tTQwxvCVHeTcVCo1NA56DqS0yulibnjk/Y1uo9SnF23j5eW73O0Ky6rdDMHMpGi/sTTFgvV5nJXTo8nJbvbonvz1870s336IHkHiVxz6dI/ighHp1NXXMzQjjhCPsDGvhCnZyVw/qR/nDU876liOhojgkeOLwXF4+brTWLvnCLedZaXI1tbXB3X7tjUx4SE8cVn7WU5CvR7u80nV7uhEhHq5Y0Y285fuZEdBGcaYNnOzfLn3CHnFlTx39ZgTWhtIRPjROTms2H6Yr/NKKCqv4cmPNnPN6f3o3f34gpnX7jnCW2v28daafUwflEJcZCh3zsg+aan/Jxp1UR0DzsPSt5ZNoKvq0jE9iQoLcZUgJ8jY14IzJTuZm6c2BPslRIVy2Vj/njA1dYZr51uFu7JtE3QwC0RYiIf0+EjXpXKkvJoPNuRzw+R+jbaFoxcQ8n0IPDx7GIkBLhJHGckvrnQrKVfX1vP+eqvfVVJMuBts+OLSHX779rdjKSpqrFT6z3YcZm9hBdHhIdw2rb+7XWCwYlRogz6ekRDFviMVfhYPxzXkZIU4QYb19cb9m2XER/LEZSM4o39j87Wj4GQmR5MQFcpmO77k0rENvY8mDbCyVD7f1VBPI9BS5tsM7+01+/yyyEb1jicxOoxXP9vNW2v28oclOxjbJ4G4yFByUmP9zueNVXvcwOaCUqvI3hMLN7HncDnvfrmfRXbmna8FJ9BKAg2NF7cXlPHsp9spqqhx3YGB6epOwHNmM27TQJysoUfmDGf6oMYBts7fI9Dl6Euo18NvrxzFvKvGcMvU/tw4JYsnLhtJdHgI950/mFF2tsbxEuLxEHYcLiqHyQOSXeUG4MYpWVzmc50oHZdbpvbnpjOzKKms5WBpFfMWbWVXEKv48eLcyyN6xZ/wY4Plkly3t4irX/iMF5fuZPbTy3hjVWMraHMs2XKQH/xlLfe8+SW5RRV+dYI+/Dqf84andVrlBtSCc0w4hf6yesSwwX7L9bWciMCv7JLaThaVY8HxNvO2ICI8eskIv2wY387TYV4hPS4iqCUlPMRLRnwkJZW1FFfWuIXxJg9Ipnt0OKt3FbL7cDm7D5dTVFHDrdP683VuMdsOBr+xhEjR6AAAFchJREFUA2uFpHSL4Csa3uh/N3cUd7zxX/JLqoKmbSd3C3cflpU19X6WpP49Yvhk00EqauqYMiCZJVusINPo8BB+eM5AvthTyNKtjTvmJvsEqWbER7Lgq1zXcub1CBXVtUSEetxg6SRbKas3xn2DSo+PZPbonnxjSCrfev4zN2sHrBosYKXBOts/cMFgpg1siHO4fFwv3lqzl7fW7OPsQSl8nVvsZwVx3GLdIkI4IyuJ5dsPuX/D757el4hQr+siufMvaxGBR21fdk5qN/62ei95RZW8s3Y/767LJSe1G9kp3fjn2v1c9uxy/rvnCMWVtcxfthOA7b+Y5afg9E2KcqtIAwzoEcPybYf4w+LtfjVcrprQh892HCYtNoJ8u4z/ayt3M2tYGiIwuk+DQhEfFUpaXCReD27VWLCU3KraOjd+KDulG89/Zyx97/mX39/Nyc6rqKljUv8ktyp1e+DxnBgLjtK5Gd8vERG46eXVfLH7CCt3HGb+NScuq231rkIeX7iJhKhQenRrm+aT5wxJZcmWAgrLq5k7vjfvrt3P3W+uY82uI/TuHsX1k/vx5xW7mTE4hV4+LuTFm625d9ehMt5Zm8vm/BJq6urZfbicZdsOkZUcTb2xwi+ceJ/Oiio4x4Bjxg/1CrOGpfpV3w0P8fCbK0e53wMtOC2pf/HonOE89dEWcosq/NxLecWV9Oke3YSC43GDMNfsKuS+v39FqFcY2SueCZndudbOtrj5ldUsWJ9HTmo3PvrB1EYPI4dAs22geyEqLIQesRHsOVweNC06ItTrBjKLWGmpjgIxoIdliaqsqeeGKZks3nKQJVsKiLGrGd8+PZu1e1ZxWkCtjXE+D92MhEhq6gwfbbQsRlZqfhWRoV63zLVTWdPQEKybbRdYjAkP4e+3nhH0/LvHhLmp0oH1PnonRnHZ2F48u3g7V20t4KaXV/tZ1M4Zksq/1uVSXVvP0IxYFtiB0P93bg432am5G3xcPxMzu5NpW7RyUi2r2F1vfum6tGaPynA7uzvK2AofC9Hq3YV+vcjS4/zjMEb2imfLgVJ+/t7XfkGoPbqF+2XzbdhfzKzfLOGl5bvoHh3GZJ96KivvnU5YiIcH393gKjjZKTG8+/1JLfLDXzupH//+Ko9pA3tw1fg+R92+LQnxeFTBURiaEcfNZ2a5GUF5RZUs+CqPmUNTj7Jny/juiyspqayle3RYm7nApmQns/iuae73+y8YzA0vrea9r3IpqaxlxfZDLNlSwK/e38Ta+79BWIiHrQdK+fYLK/2Oc+Vpvdl1qIxl26x5ZfboniREhfHeulzGnCDLaXuhd/ox4CgpxsC8q8bwo3MasipW3judc4Y03CTOG/2nmw/47dscl43rxdJ7ziIpJtwtNAfWTTggoAKyE1gYHupxK5Le8+Y68oorOTsnpVHxpcgwr9u0sDUEpvhGhnpJiQ0nz8dFFYjjUprQrztjA5STsBCPW2a9Z4L1duFYTcb1TeSrn57jZhw5+CobGXa69a5D5W4vnbV7jpAQHeYGZDuukbp647qRzh+eftRzTYgK42cXDiEhKtRVOhwiQr3cPj2b9LgI/rBke6MMgwlZ3cmIj+TnFw/z23dYz4ZSAHfPzCE1NoJeiZF8e2Jfd3lOmqWALPbJjOibFO3W/bl9+gCum9TPz4314YZ8PwvOt0/v61quwAqYdnD2E7EaLvoyOD3WVf6yU7oR4vVw/aR+nNY30f27OArS6N7xLLzjzBYHGeakxrLugXOOOwvqRHCiYnCUzs8dM7LdIPuNeSV875XVfi1kjpVKn9YkJ7O6b3iIl5euPY11D5zDVeN7u5bxipo6Fm7IY/Hmg37FQB0GpXXjximZpMSG88EdU7h1Wn/mju/NK9eP7xSBxM2hFpxjwMlAcmIrfOvYdAuok+H1eDAGfvKP9da+rbhgZg1Lc10RYMXjjOmTwEvLd5HcLZxVP57OvW+v49XPdhPm9boP+rziSi4amc6TlzcOPhyY0q1RKnJLCMyQiAjz0CcxmiPlNU32UOoeE0ZCVChXT+zjVw4+LS6CzQ+d6353qtXmBlimAgPzfBUGxwoEcP7wNJ5dvJ3dh8u5dExPN9jY6dpsjJUmuzGvuNmHbKhXiAm34qauntiXq32Uj0n9k9w088gwL1Oyk3l91R6/c8otqmRiZiJXT7CsFL6T5fh+DQrFzVOz/GKvHJyOx770TYpmUFosb3y+hzmje7K9oIw//seKaeqVGMnKnYfplxRNRnwkS+85C4DP75vBq5/t5t6313FWTg9++s4G+veIYeuBUq6e0IcHvzk06PnfPDWL+97+ige/OQSgUfDoIDfuqv0b/R0rA1K6kZnUfHxRZnJ0mzU+VToOoV4Pr904gfv/8RV/st3MT3ywmTOykkiLi2BiVneWbj3Eaf0SG81FtXX1LN9+iLS4CEI8HjbkFlNRXYfHYzVaBfjDt8e6BRZPNvedN5g1u48wND2WxVsOcturX7jrLhqZzprdhW7iRE5qLKf1S+Sze6e3y1jbElVwjgFn7nNcIVMGJDEsI46C0qpGCkxg75qmyvgH45ujMvwUnAtGpLtuG6ec/iD7rfpIeTXJMeEkRIVSWF7D5WN7BbXS3HRmlusqaQlOzR6nAq9DmNfDzKGp/Py9rzEGYiNC/HpigRVTs+YnMxAR16py8agM1yXjcFZODx5dsMlPCYAGC8wd07P5n7P7+51Pr8QorjmjLy8u3cmcMT15dvF2wLLyOH2RnKDWueN7c++sQdQHaa0+MbM7y7cfYsfDs5qVwys+PX7AsjI5Cs7iH00Lmr3gKJyzR2e0uDT/jMEprNp5mPKqOqrr6unbPYrhPePZ8fAsRMRN7d56sJTzhqXz/JLt1NebRo3u5o7vzVw71sU5t+LK2maL8V08qicXjmh6rE7V0kvG9Ay63hdfi11H4s2bTz/qNh//YGrbD0TpMPjeO04GEcAvZw/jnrfWMbZPAk9eMZLU2Ai+zi0hPiqUt9bs49cfbg56PK9HCPN6GNe3/e6ByDCv5UIWeHzhZn73yVb3JezqCX349WUjeXbxdh5ZsNHt1dUVUQXnGHAetM7zUkT4x61nBK1VcsW4XkwekMSmvBKu+9PnQd/Sm2JEzzj6JUWzo6CMd78/icFpsa4CNSHTUnT625aM7QVleDzCp3dNo6K6LmhJ+aYI7N3ksOXn57pvsoEKjojQKzGKM/p3p6K6zg5utrKOJg9I8tsOLNfO+p+eE7TAYE5qLBsfnNnInRYTHsKmh2YS5vUEVdbuv2AIP/jGQGLCQ/jOxD78afkuJmZ2J8zr4bWVuxnTJ4HND53rZq4Fs569cv146o8hVdRx8cwd37vJ1EwRYeODM1vlEpl31WjqjeHXH2zhmU+3uaX1nfGJCH++fjxVtfWs2V3IM59uY+3eIqYP6tHkMZ19W9JzqDlFLCLUy8YHZx61CKTvdaMoHZ2RdpbT/GvGkZkUQ0FZFbPnLeOet9YBVt+lmU8uIS0uwq1LFUhqbARv3DSBlTsOc9ebX3LvrEHtUjbAF+devnNGNpeP60VaXASVtfVuo+XvnZnJd0/v2y6tNU4Wcjzdizs6Y8eONZ9//vkJP+47a/fz/de+4Pzhafxu7ugW71dQWtXqoLPff7KVpz7awur7prsul+LKGsK8HiJCvRSWVTPqwQ/47ul9eeDCIa0+l/LqWgQ56kWeX1zJ+F98xLVn9ON/pw9wH5ZlVbXUG8PjCzczf9lOPv3RVFLjIk56aqExhgMlVa5id6i0iu6tUCaPhfziSnp0C2+TIMI6uz1Cc+fgtHmoqK6nX3K0X4d4RVFazsGSKr/eVVc+t4Ll2w9xxbhezBqWxq1/XtMo3u5/zurPRaMymPXUEq6d1I+7Z+YAVjHMwBdC5cQiIquNMWOPup0qOK3ni92FXDxvGXfNHMgtU/sffYfjoLaunr2FFe6bfDD2FpbTo1vECS0mFYw9h8tJjYsIapGoqasnr6jSLx1RURSlM3K4rJqNucWM6BVPdHgIeUWV7C0s58WlO1m85SBv3DiR/j1iCAvxsPtQOSlx4Z26XkxnQxUc2k7BAasJma/LSFEURenaFJZVc6isusN30e7qtFTBUZv2MdJcB3BFURSl65EQHebWpVI6PloQQlEURVGULocqOIqiKIqidDk6jYIjIoki8raIlInILhGZ295jUhRFURSlY9KZYnB+D1QDKcBI4F8istYYs759h6UoiqIoSkejU1hwRCQamAP8xBhTaoz5D/BP4Or2HZmiKIqiKB2RTqHgANlAnTHGtzb2WqBRZTsRuVFEPheRzw8ePBi4WlEURVGUU4DOouDEAEUBy4qARk00jDHPGWPGGmPGJicnn5TBKYqiKIrSsegsCk4pEBuwLBYoaYexKIqiKIrSweksQcabgRARGWCM2WIvGwE0G2C8evXqAhHZ1UZjSgIK2ujYXRGVV+tRmbUOlVfrUHm1HpVZ62grefVpyUadplWDiLwOGOB6rCyq94DT2yuLSkQ+b0mpaMVC5dV6VGatQ+XVOlRerUdl1jraW16dxUUFcAsQCRwAXgNu1hRxRVEURVGC0VlcVBhjDgPfbO9xKIqiKIrS8elMFpyOxnPtPYBOhsqr9ajMWofKq3WovFqPyqx1tKu8Ok0MjqIoiqIoSktRC46iKIqiKF0OVXAURVEURelyqIKjKIqiKEqXQxWcViIiiSLytoiUicguEZnb3mNqT0TkNrv3V5WIzA9Yd7aIbBSRchH5RET6+KwLF5EXRKRYRPJE5M6TPvh2wD7vP9rXTomIfCEi5/qsV5kFICKviEiufd6bReR6n3Uqr2YQkQEiUikir/gsm2tff2Ui8ncRSfRZd0rObyKyyJZTqf3Z5LNO5dUEInKFiHxtn/82EZlsL+8Y96UxRj+t+GDV4HkDqz/WJKyeWEPae1ztKI/ZWOn7TwPzfZYn2bK5FIgAfgWs8Fn/MLAESAAGAXnAzPY+n5Mgr2jgAaAv1gvG+VgtR/qqzJqU2RAg3P5/jn3eY1ReLZLdQlsGr/jIsgSYYs9hrwKv+2x/Ss5vwCLg+iauPZVXcJnNAHYBE+y5LMP+dJj7st2F1Jk+9sOpGsj2WfYy8Mv2Hlt7f4CHAhScG4FlAbKrAHLs7/uAb/isf9B34jiVPsCXwByVWYtkNRDIBS5TeR1VVlcAf8FSqB0F5xfAqz7bZNlzWrdTeX5rRsFReTUts2XAdUGWd5j7Ul1UrSMbqDPGbPZZthZLy1f8GYIlGwCMMWXANmCIiCQA6b7rOUXlKCIpWNfVelRmTSIi80SkHNiIpeC8h8qrSUQkFvgZ8IOAVYEy24b9kEbnt4dFpEBElorIVHuZyisIIuIFxgLJIrJVRPaKyO9EJJIOdF+qgtM6YrBMb74UYWnzij/NySrG53vgulMGEQkF/gz8yRizEZVZkxhjbsE618nAW0AVKq/meBD4ozFmT8Dyo8nsVJ3f7gYysVwszwHviEgWKq+mSAFCgUuw7smRwCjgPjrQfakKTusoBWIDlsVi+WgVf5qTVanP98B1pwQi4sEyZ1cDt9mLVWbNYIypM8b8B+gJ3IzKKygiMhKYDvw6yOqjyeyUnN+MMZ8ZY0qMMVXGmD8BS4FZqLyaosL+97fGmFxjTAHwBC2TGZyk+1IVnNaxGQgRkQE+y0ZguRcUf9ZjyQYAEYnG8l+vN8YUYrkZRvhsf8rIUUQE+CPWW9AcY0yNvUpl1jJCsOWCyisYU7GC1neLSB7wQ2COiKyhscwygXCsuU3ntwYMIKi8gmLfX3ux5BRIx7kv2ztQqbN9gNexIuejgTM4haLmm5BHCFak/MNYFokIe1myLZs59rJH8I+k/yXwKVYkfY590Z8SGS7AM8AKICZgucqssax6YAXLxgBe4BygDLhI5dWkzKKAVJ/PY8DfbHkNAYqx3ArRwCv4ZwWdcvMbEG9fV87cdZV9jQ1UeTUrt58Bq+x7NAErM+rBjnRftruQOtsHSAT+bt8Au4G57T2mdpbHA1havO/nAXvddKyg0AqsLIW+PvuFAy/Yk0c+cGd7n8tJklcfW0aVWOZa53OVyiyovJLtyfCIfd7rgBt81qu8ji7DB7CzqOzvc+25qwz4B5Dos+6Um9/sa2wVlpvkCNbLxwyV11HlFgrMs2WWB/wGiLDXdYj7UpttKoqiKIrS5dAYHEVRFEVRuhyq4CiKoiiK0uVQBUdRFEVRlC6HKjiKoiiKonQ5VMFRFEVRFKXLoQqOoiiKoihdDlVwFEXp0IiIEZFL2vD4Y+3f6NtWv6EoyslHFRxFUdoMEZlvKw+BnxWtOEwa8E5bjVFRlK5JSHsPQFGULs+HwNUBy6pburMxJu/EDkdRlFMBteAoitLWVBlj8gI+h8F1P90mIv8SkXIR2SUi3/LdOdBFJSL/z96uSkTyROQln3XhIvKkiOSLSKWIrBCRSQHHmykiG+31S4DswAGLyOki8qk9pn0i8rSIBHZIVhSlA6MKjqIo7c1PgX8CI4HngJdEZGywDUVkDlZ37FuAAcD5wEqfTR4FLgeuBUZh9a5aICJp9v69sHoHfWD/3m/tfXx/Yxiw0B7TCGC2ve0Lx3+qiqKcLLQXlaIobYaIzAe+hdVc1JffG2PuFhEDPG+MucFnnw+BPGPMt+zvBrjUGPM3EbkTuAkYaoypCfitaKAQuN4Y85K9zAtsBl4zxtwnIr8ALgEGGnvyE5H7sLog9zPG7LQtQjXGmOt8jj0S+AJIMcYcODHSURSlLdEYHEVR2prFwI0By474/H95wLrlwHlNHOuvwP8CO0TkfWAB8E9jTBWQhdXheKmzsTGmTkSWA4PtRYOAFcb/zS7w98cA/UXkcp9lYv+bBaiCoyidAFVwFEVpa8qNMVtPxIGMMXtEZCBwNjAdeBy4X0TG06CEBDNLO8skyLpAPMDzwK+DrNvXuhEritJeaAyOoijtzYQg379uamNjTKUx5l/GmDuAccAQ4AxgK1Z2lhtUbLuoJgIb7EUbgPEi4qvoBP7+GmCIMWZrkE/FMZyfoijtgFpwFEVpa8JFJDVgWZ0x5qD9/9kisgpYhBUfczYwPtiBROS7WPPWZ0ApVkBxDbDFGFMmIk8DvxSRAmAHcAeQAsyzD/EM8APgSRGZBwwDvhfwM48AK0TkGeBZoATIAS4wxtzU+tNXFKU9UAVHUZS2ZjqQG7BsH9DT/v8DwBzgN8BB4BpjzKomjnUEuBt4DCveZgMw2xizw15/t/3vi0A8VmDwTGNMLoAxZreIzAaewApWXg3cA7zi/IAx5ksRmQI8BHwKeIHtwNutPXFFUdoPzaJSFKXd8M2Qau+xKIrStdAYHEVRFEVRuhyq4CiKoiiK0uVQF5WiKIqiKF0OteAoiqIoitLlUAVHURRFUZQuhyo4iqIoiqJ0OVTBURRFURSly6EKjqIoiqIoXY7/D2HIMj8VWBbGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\", fontsize=14)\n",
    "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
    "save_fig(\"dqn_rewards_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad at all!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Double DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(32, activation=\"elu\", input_shape=[4]),\n",
    "    keras.layers.Dense(32, activation=\"elu\"),\n",
    "    keras.layers.Dense(n_outputs)\n",
    "])\n",
    "\n",
    "target = keras.models.clone_model(model)\n",
    "target.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=1e-3)\n",
    "loss_fn = keras.losses.Huber()\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
    "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
    "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
    "    target_Q_values = (rewards + \n",
    "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 49, eps: 0.0100"
     ]
    }
   ],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = []\n",
    "best_score = 0\n",
    "\n",
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step)\n",
    "    if step > best_score:\n",
    "        best_weights = model.get_weights()\n",
    "        best_score = step\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
    "    if episode > 50:\n",
    "        training_step(batch_size)\n",
    "    if episode % 50 == 0:\n",
    "        target.set_weights(model.get_weights())\n",
    "    # Alternatively, you can do soft updates at each step:\n",
    "    #if episode > 50:\n",
    "        #target_weights = target.get_weights()\n",
    "        #online_weights = model.get_weights()\n",
    "        #for index in range(len(target_weights)):\n",
    "        #    target_weights[index] = 0.99 * target_weights[index] + 0.01 * online_weights[index]\n",
    "        #target.set_weights(target_weights)\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure dqn_rewards_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXmcI3Wd//96V5I+Z3puZuQcDhFERWUQFcULFRfddb0VTxSvRdn1t+u1qLig6319vUAQhOUQBS9QEBQQBGGGYwZmGGDugbm65+o7naTevz+qPpVPVT6VVFWq0p3u9/Px6EcnVZWqT5JKfV71PomZIQiCIAiCMJ2wJnsAgiAIgiAIaSMCRxAEQRCEaYcIHEEQBEEQph0icARBEARBmHaIwBEEQRAEYdohAkcQBEEQhGmHCBxBEARBEKYdInAEQRAEQZh2iMARBEEQBGHakZ/sAWTJwoULeenSpZM9DEEQBEEQUuL+++8fYOZFjbab1gJn6dKlWLFixWQPQxAEQRCElCCizVG2ExeVIAiCIAjTDhE4giAIgiBMO0TgCIIgCIIw7RCBIwiCIAjCtKNlAoeIOonoEiLaTERDRPQgEb1OW/8qIlpLRKNEdBsRHRZ47c+JaJCIdhDRp1o1bkEQBEEQ2o9WWnDyALYCeBmAOQC+AOBaIlpKRAsBXO8umw9gBYBfaq89D8DTARwG4BUAPk1Ep7Vu6IIgCIIgtBMtSxNn5hE4QkVxAxFtBHACgAUAVjPzrwCAiM4DMEBExzDzWgDvBfABZt4LYC8R/QzA+wHc1KrxC4IgCILQPkxaDA4RLQZwNIDVAI4DsFKtc8XQegDHEdE8AAfq693Hx4Xs98NEtIKIVvT392c1fEEQBCFjNg2M4Bd3b0K5YuN7tz6O4WI5dNtb1+zE3esGWjg6YaozKYX+iKgA4EoAv2DmtUQ0C0BQjewHMBvALO15cF0NzHwRgIsAYNmyZZzmuAVBEITWccOqbfjWnx9HT0cO37v1CewbLeG8fzbe2+JDlztFXTd97fRWDlGYwrTcgkNEFoArAEwAONtdPAygL7BpH4Ahdx0C69U6QRAEYZpSsZ3/46UKAGCkjgVHEIK0VOAQEQG4BMBiAG9m5pK7ajWA47XtegEcCScuZy+A7fp69/HqlgxaEARBmBRsdozwYooXktBqC85PABwL4A3MPKYt/w2AZxHRm4moC8AXAaxyA4wB4HIA5xLRPCI6BsBZAC5r4bgFQRCEFqOEDYvCERLQyjo4hwH4CIDnAthBRMPu3xnM3A/gzQC+AmAvgJMAvEN7+ZfgBB1vBnAHgG8ys2RQCYIgTGPYVTZiyRGS0Mo08c0AqM76WwEcE7KuCOBM908QBEGYAdiewJnkgQhtibRqEARBEKYkyjXF4qMSEiACRxAEQYgFM6PSArNK8BCic4Q4iMARBEGYgTRjFfnRbetw5Of/mHnathqjCBshCSJwBEEQZhjXLt+Kwz+XXKBcfd9WAMDe0Yk0h1WD0jW2KBwhASJwBEEQZhhX3bcFAHDrozsTvT5nOfki5Uq2wsN2fVRliTIWEiACRxAEYYZx7NOcTjd/WLk90evzSuDYdmpjMqF0zUQ52+MI0xMROIIgCDOM4aLT+qB/aDzR6/M5JXCytayw66SacHs2cEglHMmyEkyIwBEEQZhhDI45XXKSCpS85UwdWbuolG4pKQtOyOHEhSWYEIEjCIIwwxgadwRO0lTvgmvBKVWydlFxpOO0ImVdaD9E4AiCIMwwhsad7Kmklg8VZFxqkQVnooHAEQuOYEIEjiAIwgxjsEkLTj7nTB1ZB/8qC06xrGJwzFQyFlpCeyICRxAEYQawdc8oblnjpIVXLTjJBEreapWLCu5x6ruqShlncwntiQgcQRCEGcBbf3oPzrp8BUoVG6MTThZVUsuHsuAUM0/fdrOoys54w4KaJQZHMCECRxAEYQbQP1wEADy1d8xbljyLyrHgNIqNaRZlmFEWnDCLk8TgCCZE4AiCIExz1u4YRE9HDgCwec8oAKC7kEseg6METoticJRrKiyoWbdESU0cQZGf7AEIgiAI2bF7uIjTvnen93zL7hEAwPzeDgwn7EVVaFGQsZIqyhUWZsHRY3BsBtwsdmGG01ILDhGdTUQriKhIRJdpy88gomHtb5SImIhOcNefR0SlwDZHtHLsgiAI7ciIW7VYscW14PR1FxJbcHKeBafSYMvmUBYcJaRCLTja+8i6fYTQPrTaRbUNwAUAfq4vZOYrmXmW+gPwcQAbADygbfZLfRtm3tC6YQuCILQnwfYGu4acWJyugpVCFlWLKhl7LqqQGBxtHBJwLCha6qJi5usBgIiWATi4zqbvA3A5izNVEAShKYJX0X5X4HTkrCbq4LQmyJgDMThRsqhE4AiKKRdkTESHATgFwOWBVW8goj1EtJqIPjYJQxMEQWg7gtO9J3DyVhOVjFuTJh7sJh6lDo4IHEEx5QQOgPcCuJOZN2rLrgVwLIBFAM4C8EUieqfpxUT0YTfOZ0V/f3/2oxUEQWgjBtx08c68BWbATiAIOBAbkxXVLCqVJh4lBkcEjuAwVQXOL/QFzLyGmbcxc4WZ7wbwfQBvMb2YmS9i5mXMvGzRokUtGK4gCMLUJejp3zvqtGnoyLsdwRMIgmDwb1bUZFFFiMFJItiE6cmUEjhEdDKAAwH8usGmDEASAQVBaFu+8NtH8K2bH8v8OGHTfYeb6p3EpVNtodCaGByVrRUW1KwHS/9+5Ta85Sd3ZzouoT1odZp4noi6AOQA5Iioi4j0QOf3AbiOmYcCr/sXIppHDi8A8EkAv2vdyAVBENLlin9sxg9vWzdpx69acOKLlFZZcJJUMr7gxkexYvNeKfgntNyCcy6AMQCfBfBu9/G5AOAKn7ch4J5yeQeAdQCG4AQff52ZTdsJgiAIGmHzvBI4SSw4ap+Zt2qImkVlWC7BxkKr08TPA3BeyLpxAHND1hkDigVBEIRGmCf6jpzTuiFJDI4SD62KwVFjDK2DY3gPom+EKRWDIwiCIKRLmAWns9BMDI7zmqzTxINupigxOApbXFQzHhE4giAI05hGQcZJLDjBCsNZERxaWAyOSaSJwBFE4AiCIExjGsbgJGi30LI0cYMFxxQ8bIrNkRgcQQSOIAjCNCbMktHZRBaVF4PTYguOfuxGy0TfCCJwBEEQpjFhAqeZLKpgC4WsMI3d5FIrmWJwROHMeETgCIIgTGNCXVRNxeDUz2rKEtMxJQZHMCECRxAEYRoT6qJKIYsq6zgXowXHEG9jjMERgTPjEYEjCIIwiWRdcTdMgzRVB4fV/4zHrhlrcpbTncckekxCS/SNIAJHEAShhTD7M4Gy7n7dOAYnvptJjT/rOBfWktwLOSVwarczxeBIFpUgAkcQBKFFjE1U8Lzzb8FfHt3lLWt1qrXC60XVRJp45hYcbfcFN2bIaMExvAeJwRFE4AiCILSI4WIZ+0ZL2Dgw4i1rdbE8RVPdxG3//6zQxVk9gWNs1dD6+GdhiiECRxAEoUWoCXu8VPGWZd+RO8yC47h8krjIlMjI2kqi7z5vhbuoJItKMCECRxAEoUWoKXdUEzhZ93MyCYK8RchZ7ZVF5QmciHVwJItKEIEjCILQItSEPVIse8uyrgZsisGxLPIEQzILjvrfuhicfMwYnKyz04SpjwgcQRCEFqHm3GFd4EyCBSdH5KVdJ8miapUFR997vk4WlclaMwk1CIUphggcQRCEFqGEwWix6qL67HWrMDpRDntJasfUyaVkwclc4OhBxla4BcfktpI0cUEEjiAIQotQc/OIJmhWPrkfv1rxZGbHNFk3iKBZcJJkUakg4+bG1vA42tjVeE2uJ9M4JMhYiCRwiOhlRHSS9vz9RHQXEV1IRLOiHoyIziaiFURUJKLLtOVLiYiJaFj7+4K2vpOIfk5Eg0S0g4g+FfWYgiAIUwWTiwoAejvzGR7TMNEzkLdSqIOTuQWn+rjgFSYMH0+jZcLMIqoF53sAlgAAET0DwIUAVgF4EYBvxjjeNgAXAPh5yPq5zDzL/TtfW34egKcDOAzAKwB8mohOi3FcQRCESUdV5h0JCJw53YXMjhkWYpPLNWHBmYQg40KdVg02s1fXx/RaYWYSVeAcCeBh9/GbAdzCzB8HcBaAN0Q9GDNfz8y/BbA71iiB9wI4n5n3MvOjAH4G4P0x9yEIgjCpqEl3RIvBAYDOfHbRAmEipJkYHK9VQ+Z1cGpdVOYYnGorB4XE4AhRf1UMIOc+fhWAm9zHOwAsSHE8m4noSSK6lIgWAgARzQNwIICV2nYrARxn2gERfdh1g63o7+9PcWiCIAjNoSbskUBQcZZTcdg830wWlRIPLXVRuRYak6aymT0XVvW1InBmOlEFznIAXyCi9wB4KYA/ucuXwhE5zTIA4EQ4LqgTAMwGcKW7TsX47Ne23+9uUwMzX8TMy5h52aJFi1IYmiAIQjooPTAasOBk2bTSNNEzmrPgVCsZA6u37c9M6PgK/dVxqVWY0V1w7sGPO7AvdDthZhFV4Pw7gOcC+CGArzDzenf5WwHc3ewgmHmYmVcwc5mZdwI4G8BriKgPwLC7WZ/2kj4AQ80eVxAEobU4k26wuF+Wrp5GFpyNAyOxrR365qf/4C7ctnZX+MZN4K9kHJ4mzuxYeJ74yuvw3/90rLtdJkMS2ohIofvM/AiA5xhW/SeAimF5s6hTk5h5LxFtB3A8gFvc5ccDWJ3BcQVBEDIjbNLNcjIOj8FxBMPl92zG8w+dhzc+76DE+xwcLyUfYB18hf4a9KKyyBE59WJ1hJlFU7mJzDweZ3siyrvHzAHIEVEXgDIct9Q+AE8AmAfgBwBuZ2bllrocwLlEtALAYjjBzR9oZuyCIAitJmzOzdKdYrZ4sCcEAGDLntFY+wzW1slq/L5mm7l6dXAYlvt+ROAIilCBQ0QbETH2jZmPiHi8cwF8SXv+bgBfBvAYgK8COADAIBxLzTu17b4E4CcANgMYA/B1Zr4JgiAIbUTYpJtlQGzYrvOawDlobnesfQbjklsRg6OCjE3HYgYsct4PUfL0d2F6Uc+C80Pt8SwAnwJwH4B73GUvAvACAN+OejBmPg9OTRsTV9d5XRHAme6fIAhCWxIUG6971hL86ZEdmXa+DhNVliZwghlIjQgKsiSBytGOU30cxUUF6BWPMxmS0EaEChxm9oSLW3X468z8VX0bIvocQtK1BUEQBD9BsXHCYfPwp0d2ZByDE2GbmAMIbt6aLCqVJh7ionItN0roiAVHiCrb3wTgWsPyXwH45/SGIwiCMH0Jzs2FOpN2WhhjcALP44qB4D5bEoNTx4LjFzgSgyM4RBU4IwBeblj+cgDxotMEQRBmKByQFvXiSlI7Zp2J/tOnPcM5fkwx0DqBY6iDE9JsU7mmROAIiqhZVN8F8CMiWgbgH+6yFwJ4H8JjagRBEASXW9bsxJX3bvYtU+0FsvSmmJpTqrn/jc89CN+46bEEFhxHUKjXZRWD4+tFlQuvg+NYcJzHuTqWHmFmEbUOzjeIaBOAcwC8zV38KID3MbPJdSUIgiBo/Hn1Dtz+mL99TEc+fNJOi3r7zlvJMo5sZuQ1gZOk3UPU4yjyVniaeMVmL3tKYnAERUOB49aueQ2Av4iYEQRBSIZpuvWsEpPkorIS1oyxbUYhZ6FYdoSNyUqUBqZCf2EWKc9FJXVwBJeGMTjMXAZwPUJ6PwmCIAiNMU24VbdLlsetPiZ/w23kEtaMsbkaE+O8PhuFw4YsKtPnqKeJSwyOoIgaZLwSwFFZDkQQBGE6Y5pvC3UCZ9PC6OZxbSNWUy6q6vTRihicRpWMlYtKibaMNJfQRkQVOOcB+DYRvZGIDiGi+fpfhuMTBEGYFpgsCh0tSROvPtZFCZC8rYHN7IkzoDVZVAUr3NrFXBU2ykqVpWgU2oOoWVQ3uv+vh98tSu7zXJqDEgRBmG6YJuZ8i2Nw8jkCSlVrUtVFFW+ftS6q1llwTMeqMKMQ6EWVpWgU2oOoAucVmY5CEARhmmOOwVEuqtYcV+8/BQDKoJM0yFiRnYsqWgyOqdBfVoHPQvsQNU38jqwHIgiCMJ0xWRRUmnjLXFS5gIsqcZAxey6jJK+PjF4Hp06PKVtrtplUtAnTj6gWHAAAER0I4FAAHfpyZv5bmoMSBEGYbpiCXltRydjXkTtgwcklDjIOuKgyEhP62OvFC9mSRSUYiCRwXGFzFYBT4GhqFXujkBgcQRCEOkxWmjjXseAQEYjiiQFlbdLdXZWMfGymSsYmMWYzewKomkUlAmemEzWL6nsAKgCeCaf31EsBvBVONePTshmaIAjC9MFc6C97a4M+0VfTxKvkiGJZcNSm+RbE4Oi9u6pp4rXb+SsZZx/XJLQHUV1ULwNwOjOvJSIG0M/MfyeiIoDzAdyS2QgFQRCmAcYYnBZkUZkykXQsi2K5mJQYKrSg0J8pxd3YHV1LE1cxOJJFJUS14HQDGHAf7wFwgPt4DYDnpD0oQRCE6YZJw7SmkrFuwVGzf3V9jiiWwFL707OosrKWsCEDzDRUm9kTNlbCwGlh+hFV4KwFcIz7+CEAHyWiwwD8G4Cnoh6MiM4mohVEVCSiy7TlLySiW4hoDxH1E9GviOhp2vrziKhERMPa3xFRjysIgjDZmCwP+Ra4qHzF8gwWnJxFsVxMane+GJzMWjVUHxfcjDOTtamiVzKWbuKCS1SB830AS9zH/wOn+eYGAB8H8PkYx9sG4AIAPw8snwfgIgBLARwGYAjApYFtfsnMs7S/DTGOKwiC0HK+d+vjOPuqBwCEFPqzrNhBvnGpGDKRdHJWMguOLwYnsyDj2gwwk+vJVMlYsqiEqHVwrtQeP0BES+FYdLYw80DY6wz7uR4AiGgZgIO15X/StyOiHwKQ2juCILQ137v1CQDAD9/ln5i7ChbGSzZyFjkuokmqgwM4AidWDI67bUcu+zo4+m49y4ypkrGWJi5ZVIIikgVHdxcBADOPMvMDccRNTE4BsDqw7A2uC2s1EX0s7IVE9GHXDbaiv78/o+EJgiDEQxcxPR3OvWXOIlhEmVbdtQ0uKj07Ke7x2d026zo4QUtNvk68khODE8yiEoEz04nqonqKiB4joguJ6J1BwZMmRPQcAF8E8F/a4msBHAtgEYCzAHyRiN5pej0zX8TMy5h52aJFi7IapiAIQiz0MJXuglM6zCLHpZJlxo+vDo5lsuDEs3YowZTzxeBkIXD8z+ul1LOvkrHE4AgOUQXO0QC+CaAXwDfgFzzvSGswRHQUgD8BOIeZ71TLmXkNM29j5goz3w0nJugtaR1XEAQha/SJubsjh7xFICInBqbVdXACWVRxrB1eFpWVbQxO8DOplyauu6gARziKi0qIJHCYeR0zX8zM72bmQwAcB+BuAGcCuLL+q6PhZmXdCuB8Zr6i0ZDgVFMWBEFoC/Tptqcj53OpZOuiqj4Oq4MTRwwoMVTIZ2vBCe6ymnFm2pZ9FqWsRaPQHkSNwbGI6AVE9Bki+hOAewG8HI64OTPqwYgoT0RdcFo75Iioy112EIC/AvgRM//U8Lp/IaJ55PACAJ8E8LuoxxUEQUiTLbtH8fd18UIQdTdUdyHnWVMsN4vKthm/WrEVE+V01Y6xDo5G3CDjapq4XgcnAxcVghacOr2otDRxwGlBITE4QtRKxvsAjAO4EcA1AD7KzJsTHO9cAF/Snr8bwJfh3NwcAeBLROStZ+ZZ7sN3wEkt7wTwJICvM/MvEhxfEAShaS6+awNuemQH7vvvUyO/Rrc8HH/IXK9QnmURmBm/X7kN//XrVdg5OI6zX/n01MZqqoPz2dcd4y2L36qhtpJxFq0aamNwwqs+21qaOOA8Fn0jRBU4DwM4AcALAIwAGCaikbhZVMx8HoDzQlZ/uc7rjAHFgiAIk0GxZGMipl9Jtzy8bdkhOOoA5/7Ncq0N/UNFAMDe0VJ6A4VfWFkWYdPXTvett2K6c0y9qLIo9FcTg9PARRWMwZFKxkLUGJyT4RTjOweONec/AGwlolVE9P0MxycIgjDlqDDHnkD1zck3GRNshieYCoZaNc2gCwUyhC7GtuCoXlSaosgiyDiouRoFGesuqriiTZieRLXggJnHANxKRI/AqVFzOoC3wwk4Pieb4QmCIEw9KnZ8gaO7iix9MnYzflTsTUc+bYFTfUyG1AzLihfkbKpknIWYCO6zXqE/Zn/auhWzv5YwPYkkcIjorQBe4f4dDWAngL8B+ASA2zIbnSAIwhQkicDxW1KqqIyfkqsyOgyZTs3AIcetHj+eQKm6qLKNwQnuMkfkBmTXbhtME3c+09SHJLQZUS04P4DTOuH7AG5n5rXZDUkQBGFqU7E5ttVCD1PxW3CcybjUAheVZTDhJA4ytjJu1RDYJVnh6d9ODI7fKiZZVELUXlSZVS4WBEFoNyo2x7Za+Cw4ms4g10VVcuNYJsdFFSdNXLmoso3BqXFRkVMY0TRU5moFY8ARcllWhxbag8i/JCJaTET/SUQ/IaKF7rKTiejw7IYnCIIw9agwgzl5iwWqcacwiuUWBBkbBE5cC46K1ym0OAbH8lxUhiDjmiyqeO9JmJ5ELfR3AoDHAJwB4IMA+txVrwbwlWyGJgiCMDVRk2cS1w4Af8ZPwEVlEiHN4A+2NbioYhb6a1kdnMBzy3LEmLkODvvr4EgMjoDoFpxvAfg+Mz8PQFFbfjOAk1MflSAIU4IzL1uOb94sIXdBlLCJM7H76tEEXFQVLcg4bctD2HEVuZitGrwsqoxjcMwWnFrhwq41zV/JWHpRCdEFzgkATJWDtwNYnN5wBEGYSvx17S786Lb1kz2MKYea0ONlH5nr0eTceJHsBE4DF1VMC84nrnoQgD8GpxXdxJ0YnNrPXB3aqrHgiMCZ6UQVOGNwCv0FOQbArvSGIwiCMHW4dvlWXHhHrcBL4qLiEEuKihdRdXDSFgv6cU2F/uLUjGFmbBgYqVneCoFjWWQs4Kee66FLTnXo1IcktBlRBc7v4PSJ6nSfMxEtBfB1ANdlMC5BEIRJ59PXrcL//qnWRacsHkljcHSdYVmqknH8fcY9brMWnKLWCPSkwxd4j8staNUAuDE4geXq86JAmrhYcISoAuc/AcwH0A+gB8BdANbBadtwbjZDEwRBmJo0G2QcrNnCzCi54iHtgN1GdXAcC1K0fRVLzoZfeP0zsWh2p7e8FTE4AIxp4mozqWQsBIlaB2cQwEuI6JUAng9HGD3AzLdmOThBEISpiCdwIloJmNlX6E+XGZ6LylUZaU/MjXaXs6Ifs1iuAAA685bPzdYKFxVQFYM6SgjVVjIWgTPTaShwiKgAx2LzXmb+K4C/Zj4qQRAmHSmUFk5cC47NdXpRWf408bQtOJyii2rcteB0FXI+l5DNjkiyTGlaCTELnNr6NhVP4OhZVPH6awnTk4YuKmYuATgctWUJBEGYxmRR22S6EF/gsO8CWtNsk9lz/6RtefCniTcXZDzuWnC6CrVTR9qtEYwxOIb6NuwKGX8WlQh0IXoMzi8AnJXlQARBmFqUUroFtm3G3x7vn1YTjpp8o8bW1vSuCmRR2cwYK1W8bXXW7hjEjv3jscd4x+P9GJ0ohzb5VMSz4LgCJ5+rWZd69pdhmSlNvGJwUTlZVNPnfBOSEVXg9AL4MBE9RESXENEP9L+oByOis4loBREVieiywLpXEdFaIholotuI6DBtXScR/ZyIBoloBxF9KuoxBaGd2Dk4ju37xyZ7GADgpS03y+X3bMJ7f34f/vjwjlT2NxVQ1q2o2UPM4QX3nOq8CBU4p33vTrzwf/8Sa3y7Bsfxvp/fh1d/52++/TXbqkFlUXWaLDgZBkcrnB5T5u2CQcbSqkGI2k38WAAPuI+PCKyLcxZtA3ABgNcC6FYL3d5W1wP4EIA/ADgfwC8BvNDd5DwATwdwGIAlAG4jojXMfFOMYwvClOekrzoT2aavnT7JI4EX9NosW/Y4gm2qCLc0UC6dqO4kmzkQC+Ovulth9qwjaUzMSiw9tW/Ml9pNJhdVjErGngWnUGvByTJ2SGFR7eejvoNgmrgYcISoWVSvSONgzHw9ABDRMgAHa6veBGA1M//KXX8egAEiOoaZ1wJ4L4APMPNeAHuJ6GcA3g9ABI4gZERaFpzpSNmLwYm2vc0cbsGxCKWK7YmHNISCvo+B4Wp3nTALTtRjekHGrXBRmYKMTYX+jDE4YsERYnQTz5jjAKxUT5h5BMB6AMcR0TwAB+rr3cfHtXSEgjDDKKVUCpanYX5CJaaLyubwVg3KnaI+7zSCjMMmd2Ml4xgp1UqEmVxUaRf7M72F+i6q6jIyFAQUZh5TReDMArA/sGw/gNnuOgTWq3U1ENGH3TifFf39/akPVBBmCmlbcEzukXYlbpCxUwfHHAtD5K8QnIoFxxVLwWwnc5p4dOuLGqey4Nz1mVfg3NOP9R0zLcwxOIYgY0Ml47xYcARMHYEzDKAvsKwPwJC7DoH1al0NzHwRMy9j5mWLFi1KfaCCMFNIK4tqOuK5qCLH4AR6QgVcVLrASaPQn7KmHDa/17fcJDHzloWKzZGy3KoxOM7UcfC8Hhw01wmn3DdaamLEtUStg+NVMtY+1O5CzotDEmYuU0XgrAZwvHpCRL0AjoQTl7MXTtfy47Xtj3dfIwhCRuiT7nRK8U4D24vBieqi8qeJ++vgkCccgHRjcA7o6/QtD62Dw8C7fnYv/uOXD9Xdb9VFVY3BmdfbAQDYOzrR1JiDhGVRBT8er5KxNpv1dOYxOiECZ6YTKnCI6K9ENNd9/F6t0WZiiChPRF0AcgByRNRFRHkAvwHwLCJ6s7v+iwBWuQHGAHA5gHOJaB4RHQOnJs9lzY5HEIRwdAuOFP3zU222GW172/ZHIvlbNSB1C46ycuj9ooD6Lqp7NuzGbx58Cvdt3BO6Xy9NPF+dOha4AmfPSLoCxxxkXCu2TZWMZ3XmMFIspzoeof2oZ8E5GU5jTQC4FMCcFI53LoAxAJ8F8G738bnM3A/gzQC+AmAvgJMAvEN73ZfgBB1vBnAHgG9KirggZIseg9OMu2o6GH9qJtWYnb+DQcZBC04xbQtOJUTgGLa1LPIFCG8cGDZs5VAsVUDkFzhA3uPhAAAgAElEQVTzshI4huB0UwE/Ngicno68CByhbpr4WgBfJaLb4Pwu3kZEg6YNmfnyKAdj5vPg1LQxrbsVwDEh64oAznT/BEFoAbqoKZUZ6Ii/j5/esR6/vv/JFEc1OdgM5PTmkhxX4PjTxHVLikWEcU1MplGBV43rgNldvuWmQO8ckS9jrlgnuHy8bKMzb/n2M7e7ACB9gWP6aE3dxNVzXeD0duQwWqqk3h9LaC/qCZyPAfg+gH+BU8zvazAX9WM4LiRBEKYRPgtOwhTg3z20DcPT4E66YrOvUm78IOPwQn85i3yfdZpBxtFcVP6FqieWifFSBZ2BGjj5nIW5PYUMBI6hF5Whm7gSc/rb6OnMg9npndXTEbWerTDdCHVRMfPdzHwiM8+DY8E5gplnG/6C2U+CILQZF9+5Aaue3OdbplcyTuqiSmOyngrUVM9VlYzd/zes2oabV4e3ogi2atAJio40XFReDM6soIvKHGSsM14n+6hYso2NNuf3dGBPykHGYVlUNYX+vCBjzYLT6Yia6SCuheREzaI6HIAUlRGEacoFNz6Kf/7h333LfBaccrJJt52LreniLGipUc+VGDn7qgfxkSvuD99XIItKJygw0rHgOPuY3eW3XkSy4NR1UVWMbRrm93Zgb+pBxiExODVis7pO0dvhjPGrNz4qsTgzmKitGjYT0WIi+jcAz4TjlloD4MfMvDPLAQrCTISZJ70wnm7BSdqXShcG7RYJoY+9osWo2DZ71oWoMTgV7TVBggIjTQtOIee/hzWFowSPX8+CM16qGNs0zOvtwNY9owlGGo6xkrFVu9xUyVi5pX770DYcuqAXn3r10amOTWgPIllwiOhkAOsAvAtO5tM4gDMAPEFEL8pueIIwM5kKnp1SCllUbWzA8YmXXz/wZLUZpvam4jTbDCOoY/Vtk9YfUiIpZxFectRC7ViNXVT1LDjFsm1s07B0QQ829I9g/1h6xf7CLDjB5aZmm72dmghr55NQaIqoLqpvAbgawNHM/B5mfg+AowFcA+DbWQ1OEGYqk1VFeOueUaze5nRFSSMGp53L5euWlPNvWINv3fwYAP97imptqbedLjAKOfK1PEj6+akChHmL8H8fOgnveeFhodsGjDwolsMtOBNlGx3BFwB4/XMOxETFxk2PbE80XhMqs6u3I4fjD5kLoEGhP6qNwQGAWV0SZDxTifrNPxfA+5nZu8oxs01E3wHwYCYjE4QZBDNj+aa93vPJKqz30m/cBgDY9LXTfanDiYOM2/juuRLorbR9cNxZrn03UeNl6vVp0lsMdBVyAQtRpN2HHk+5n9QhTF5PXRjM7+3wOoYb9xvIJlM85+A5OHBOF+58YgBvP/HQZIMOMDTuWIN+/bEX49inObksRIaAb0Orhl4tc8oUWC3MDKJacPbDCTQOcjiAfYblgiDE4Nf3P4m3XXiP97zcQgtOmBtEd1VMxAwytm3G4zuH2jqLqiY1XsXdcHwLS73t9BYDXYWcX0AlFIjBGBwlYkytGvQ06vm9HXUtOHaIwCEiHDK/B7sGi4nGa2Jo3AkO1gOlc1ati8qYJt5RdVFJJtXMJarAuQbAJUR0BhEdTkRLiejdAH4Gx3UlCEITbNo94nteSrkzcz3CJt9SEy6qn925Aa/57t+wbf+4t6zdmonXNHV0FU4lgQspqouqq2ClInD0GBwd01fwXNf9AzjuoHoxOBU2CxwAWNzXhR2D48Z1SRh0LTizuwreMosI/UNFjGl9puqliQPA6IQInJlKVIHzaQC/BvBzOMHG6wFcDOBXcNouCIKQIuWEhfWSEDb3NtOqYeWT7W/YDYoS9ZX4LDgRBUhdC06gC7a+bdIYHGUBzEdwUT1jyWzvcWc+VzeLyrbZaAUCgMV9ndg5OJ5aY9bB8TKIgNmdursJ2LZ/HB++YoW3TB3OCgkyHpGmmzOWSAKHmSeY+RwA8+DE4zwPwHxm/g9mTrf4gSAIdWM20ibMStCMBaeNQ288gjE4ngUngQCpJ1h1g0itiyrS7g3Hcy04bn8JFYdiikfRLTKdBaspC06xbGNwLJ7FxEm7r41rGhovYVZH3meZ2ehaOu98YqA6JoOLSg+Ensl1cOq5G2cCUS04AABmHmXmh5l5FTOnW/RAEGYwwYmnlUHGYWJEt+BMxBRcpn22m+gJihJT7Zt0YnD8Qca2YbKPizqesuBYdSw4ALDi3FNx56dfgc58rm6rhoptjuMBHIEDADuHorupdg2O44jP/xHXrtgKALh1zU4c8fk/Yt2uIQyOlWsKFW7odwTOQq1Cs8lFRUS49VOn4ND5PRgpzsxJfvW2/XjGuTfh9yu3TfZQJo1YAkcQhNbQyiDjMDeLLrJKde7qAeDwz92I//7Nw6mOa7KpjcGpXZ52DE53IefbNq0YnKqLyixOFs7qxCHze9BZsDDeMMjYvE4JnB37owucdf1O5/LrHngKALzGrI/vHMbQeAl93QXf9ur9zOmuCh9TmjgAHHXAbBwwu3PGxuB86XerAQAPbN7bYMvpiwgcQZiCtDLIOGwS1TNmGrmomIEr791Sd5t2CzIOfgfKjZIkCDjo7tLRXT7dhVzdFhFRqVpwnEu8EjaNvoLOvFXfglPXReVYVXbGCDQOWppUP6u5PQUMjddacG75j1PwrIP6sHe0WlBQGdpyhhOstzM/I2NwyhUbq57cP9nDmHRE4AjCFKSVQcYccqgKM7ryziVisgoPTiY1FhxDmngahf70ebmzYPm2TerWKwfiUtQxQrSJR1ch1zBNvJGLatdQ9FRxlQ2lRJPqSN6ZtzBULPkyqADgiEWz8MpnHIB9oxOo2IzxUgUfutwJODYNq7czNyNjcDYOjHiFOtVn9e6L78Uta2ZWZyUROIIwBWlpmnjILFqxGZ1uY8W4MUGM1o0/K4Ii0zZYcCqBANmwDKIoWVSFHCFvWalkUVVsG3mLNMuNclXVVzjNWHC6CjnM6S7EsuCo1g5qn6phZ8UGBsfK6DNUIZ7X2wGbgTf8v7twx+PVHtBhNX5Gi2V87vqHccOqmROL8uiOIQBOsPW+sRIe3T6Iu9YN4DcPPjnJI2stkWtYE1EHgGcBOAABYcTMf0x5XIIwowhem1vZ4iDURcWMTteCUy+ryzSpT4cg47DvIFjJWN+sYjPyudqJtp5FLucJHAs5K5kLrPZ4fiFSL01cx7Hg1AsyZqMrSLG4rzNWDM6gW8xPuah2uwLHZsbQeK0FB3CKEQLAmu2DXgf3045bgqMOmFWzbW9HDtv2j+Pq+7bg6vu24PXPOTDy2NqZx3YMIm8Rnn/YXOwdLeG+jXsAAPdt3DslGvm2ikgCh4heDeAKOOImCAOobS8bEyIaDizqhtOt/BNEtBTARgB6NbSvM/P5zR5XEKYCwctNK4OMwybRis3oUAKnjuCKam1qt7YNwffMAC79+0aMaXViyjYH0ukZhmbbDSw4zn9H4Fj+Vg3aaWDb7MsUqkelwp5o0I8RJQZnomI7QsZwrEZjWNzXhZ1DRVx4x3osmdOFf3nuQXWPV23O6d+nbbMxBgeoChydH5/xfOO4ejpnZh+qdbuGcdiCHhwwuwu/X7kNK7c6dakGhovYtHsUhy/sneQRtoao3/6PANwA4HwAO4H07c/M7MlvIup1j/OrwGZzmXnmOVSFGUeplRacsBgcu1pPpBLY6La1u1C2Ga9+5uLI8Tltpm+MPY++/Ic1gWXs265k2+h27/c4YqyOslKMlyp1LTgVZlgR+yqVbUZeS3eK7qJyxj5RttHdUavUKtzIgtOFO5940ptQX/+cA0NdWgAw6AqcsZL/sr5+YARlm3HwvJ6a16gqxd2FnCc2w0TXrIDA2T9WwpzuWqvQdGPTgCNi5vZU3+vblh2Ma1c8iUe3D84YgRM1BudpAL7KzJuZeZyZi/pfBuN6C4BdAO7MYN+CMOWZChYcm6sWnKCV5gOXLcdZbnBnVIHTbhac4PsyueIqNvvEi+7K44DrKowD53YDcHp/5Yj8MTgJ+l6pbfOJXFSWOxZzoHHFDhcTQDWTSvGPDbvrHk8JnOFiBb976Kma173g8Hk1rzl2SR9e+vSFuO5jL8YHTl6Kr/7rs0P3f+LS+TjqgFk4wp3QH90+WHc8jXjkqf1TPo7Fthmbdo9g6YJezHXF3ClHL8IX33AcACcAeaYQVeDcAODFWQ4kwPsAXM61V5TNRPQkEV1KRAtNLySiDxPRCiJa0d/fb9pEEKY8k50mbtvsTZKWoYOzzoRB4Ji2bre+m7UWnBCBo71//bG+dT0LzkHzur3HuUCQMScUODUxON7/aBacsI7iNofXwQGqxSHfddKh6Mxb+OvaXcbt9o1O4Lr7n/RcVCu37sM51zzkrb93w27M7+3AkYtq42q6O3K44oMn4ZkH9uFLbzgO7zopvHv5Cw6fj1s/9TL8+mMvRm9HDt+95fGmWkm85ad34z9+uRL9MTLF0qJYruCqe7fUPQ9ue2wX/vZEP4plG4ct7PWsXQfN7cKszjwWzurE5t0icIJ8FMA7iOi7RPRBInqv/pfmgIjoUAAvA/ALbfEAgBMBHAbgBACzAVxpej0zX8TMy5h52aJFi9IcmiC0jJb2ojIcqsIMm514i3zOqjtBmwKQTXNIu1lwgu95whB8W7GDLqoQ91IdK9eBc7u8x0EXlf6yODVxVBaVwsumamDBURY703t19lvfRfXPxx+EOd0FfPSUI/HcQ+Zi+aY9xu0+c90q/H+/Won7t1SL0C3o7cA1H34hAGBgeALHLJmdWjDs/N4O/Nsrj8K9G/dgZ8yO5xsHRvDAlr2wbfaE3x8moTrwtcu34vO/eRifu36V0RLFzPjApcvx/kuXAwAOX9Dr1RVa0ueI6MMX9mDTwMxpQhA1Bue1AF4F4J8AjMJ/c8IALk9xTO8FcBczb/QOwDwMQHVX20lEZwPYTkR9zNyczVEQpiCT3YtKTdw5IuQtqonB0Ykeg9NeAid4pzw0Xhv+V+Ggi0qz4GgvrycQF/ZW3To1QcYJ2zaUbfb6UAHR6+AU3NeECexGQcbPPngOVn7pNQAc68mPb1+PkWLZsyRs3TOKYtnGk3vHAAD7tIJ9xzxtti9mplDPVJQA1d4hbk2nV3zrdgDAzf9+irfs7vUDOPMlh6c2tih0dzifzbUrnsQjTw3ij+e81Lde/ywBp4lqV8HChXdswD89ewkAYOmCXtz++MzxbEQ9g74F4IcAZjPzLGaerf31pTym98JvvTGhfukzI9dNmHG0sheV0UXlBs9aFiFnUV2XWdQJo91qBQa/A6PACVpwKiEWnIi9qGotOPGLCgKOQFZVjIH6zTZ1lFsr7FiNgox1XnjEAlRs9jXGfOk3bsOp37nDsxTpHLag11fLpl5wchKURSuOJVEXrDc9sgMAcNQBs2JbgZph/1gJmwZGfOLv0R2DWgaaw0bN9fSZ047BotmdWLZ0PjZ97XQ8fbHTMf6ZB/ahf6iI797yeM3rpyNRBc5cAD9l5kydd0T0YgAHIZA9RUQnEdEziMgiogUAfgDgdmaWWtTCtCA4n0x2kHHFdlxUVQtOnRicsmmdWTS1E0Gr1aBhQqgE0sR1y0dUCw7g1GuZ3Zn3YnCUtUvfRxwLTjDN26oG4dRFiaIwC2JY+riJkw6fj4WzOrzgYd2CN6a1TzhgtmNZWdLXFRhzugKnkXgz8ej2Ie/xJXdtwHMPmYsTDp0Xq5hhM2zdM4of37YOZ1x8byA2y+kxtX+05MUDbXKDh2/4xEvwsZcfadzfu046FM86qA/f/8sTuOTODdm/gUkmqsC5DsCpWQ7E5X0ArmfmocDyIwDcBGAIwCMAigDe2YLxCEJLCF50W5ombjiU56KKEoMTMV6o3VxUQavVkKHkfzBN3JdFhWgWHAC4/wuvxvJzT/WsI6oAXpK2EM62wRgc538j0ZC3GrioOHotnnzOwmnPWoLbH3NcInpg7tod1Uu8SlleNLvT50JL2UPlvfc4QvGRbdV76MHxMt743AOxuK8TA8PFzG9Cnto3hpd+4zZc+LcNGBwr+b6TjpyFP6/ZgX/9yd/xiasfAABs2j0Ki4CjXWuNic58Dtd9zMkXmgk9uqLG4GwA8BUiOgXAKgC+Wxlm/k4ag2Hmj4QsvxrA1WkcQxCmIkFrwVSw4FTYcZ+kFYPT7llUJsoVfwyO/lnYMSw4XW5LDFUF+aSv3oq1578uspsrSNDSErXZZj7XwEXVIMg4yPyeDoyXKxgaL+Hv6weM27zymANw78Y9OO7AvoC7LhsXVZxg7aJW1DFnEV5//IG4efUO2OwEQi+Z01Xn1c0xoAnCiiakr/3Ii3DJXRtw9X1bAQBdbubb+l3DOHhej9H9p9OZz6GvK9/SaumTRVSBcyYc68mLUZsuzgBSETiCMFMJTigtDTIOy6KyGTlyLuz1xmNyUU3HLCoTFWbfZ6O/xi9OoolA1YtJZev4OovHDDLOGwRCI21Sz0XF7LSliGrBAZxtmZ1A3YHhCd+6C974LLz8GYvwtDndeNkzFuGYJX2+Gi1pu6jUuON+jgDQ05HDC49YgIWzOrF4tiNqdg6OZypwgmUG1FgOnNuFf33ewbh59U7kXPcxM2PF5j046fAFkfada+B2ni5EEjjM3NpwcUGYYQQvNqVWpokb6+BUrQB5i/wTd3Cs07UOTgTLlG2zz3VQKptjcH502/pIxwz2gfL1uYqVJh6MwYmWJp6vk0WlxhLHgqO2HRiewGueuRjve/FSnHHxvQCANz//YK9a8jFL+mr2nbYFR+07zsSufhtXfPAkHDLfSbVWXdNbFYcDVOtSAY4Ife1xi3H5mS/AVfduwRO7hrBlzyh2DhZx4uHzI+0vZ1Gs86ldkW7igjAFCF50K5OdJq7q4JATg6OPL1jYT02Gjea9dovBCbPgHLagx7dNWB2cJO/3P1/zDCxd0IN5bon9pJ3Fy5VAq4a4MTiG808dP05sjG7tefZBc3DyUQtx1YdOwt2ffaWxFYQ+vDhCKgq5Jiw4xx3YhwNcy42q1ryzhcX+9PMs53aJP+XoRegsWLAZeMCtJ3Ti0trKzyYsolixSO1K1GabP6i3npk/mc5wBGFmUmvBmdwgY3XHWLXgVEVN0MqgXFSNJqR2c1GFTYTfeuvx2LpnFD+5fT1KFTtSHZyozOkp4NRjF+Oq+7a4+0geg6N3NY8qFfJe77HwLLg4Lipfw0/38YuPMhahB+C32sQ5ThTUvuOch7YmKhRKmOnxOa1AWUr1zzRHzm9zcMwJSlcirBFBq+x0JWoMTrDZRwHAMe7rH0h1RIIwA6mNwWmdi8o0mTlBxtU6OPodfbBPkbrwNpqQ2u16GhY8Pa+ngBOXHow/PbIDa3cM+T6bsDo4cegq5DBeqoCZE/eiKts2OgvVy3vVRRXNgmN6754FIY6LKmbQcC4weadJro51Koyy4T17cUotPqHVjYVewDFnESpaoHvUz8yyxILjwcyvCC4joi4Al0AaYgpC09S4qFp48TE2kfSCjGvTxPUy/rZWB6bR/NVuFpyJkIlQ9Wt6wdL5uGXNTmzbP+at0y1dSb/CLtftUKpwYhdVaLPNBq9TVh+j6OVaa0YjLJ8waPw6fX5O24KjxhI3lokoWIwxvqsrCcHfpfrd+cRWjlz3Va34qYfE4DSAmccBfAXAf6c3nOnN+v5hnPf71TNCOQvxqLHgTAUXFVddVJUQgVO2q1lEOSLc/tguXHznBqNoarfraVg/pk6347YK6PzH+mrH7E/9cqVXIZaNodaN8RpeliuxigXqOM02ay/vUWNwTC5Sdd2Kk90Ut3CfP8g48mFijSVO/L4pLT7fIoET3L2ynAY/U6cWk/M8qgUn2LV+utLsKbQIQG27V8HIR664H5fdvQkbZlC7ep3hYhkX3LAG4y32XbcDwTTiVl58wu7Wbdu5gOa0GJyRYhlf/N1qb7uybXtBx5ZFeP+ly3HBjY+GZFG11wW1VLHRVai9RKqaNUcvdi59+u95omLjVyuc+iRJ3646ZrFk+76bOJ9frQUnWhaVEkWmtHY9yDUqceva+Fo1ZBVk3EQ2GuC8J6Lsb0KC37cS3Pr3qmJpPAtOxO8mZ1Hb/R6TEDXI+FPBRQCeBuAMAH9Me1DTleoJNf1PLBPLN+3BxXdtxGnPWoJlS6OlM84UgiKjld3EjS4qL8jYuYiqi+vP79qIu9ZVC7aVNRdVo4tru90xOgIn59WkUajCat2FHHIW1fT0md/bASC5oOt0BdR4qeLbR9zYEd1dob6ahi4qLwYn3EUVx3UUN+1b33dWQcZRaxI525pbUzQqfpkGwe/bc1H5RKOFSkWz4MQQOK2stTVZRA0y/kTguQ2gH8ClAP431RFNY9L9ubYfKvW53Sa6VjCZMThmFxWqLqqchVG3rHvw7rdc0WNwpleQ8UTZRofBT6I6bhMRZnXmsW/UX8BudpeT4p3cguNm6ZT9AieKYNo7MoFb1uysteCo/xHr4BizqGK6QQC/m2nSg4y9OjjRX1MOETi5FmQhBb+DiYrtpYhXx6GqHEeLg1Mo19Z0Rwr9tRB1Ys6A88qIuiCIwKllMisZm0z2KsjYCjTbLAQm/HLF9u72dYFjOsfbrQ7ORMU2lr3XJ5hZnXlfkDFQzYBLbMFxjzlesmO1ewCAf//lQ7jjcaf30/MOnVsz5sZZVOEZQtUg44bD8NDjgKIIFn8vqrQrGTv/4xb6MwocosxrVQWtuMWSXfMZ5iwnAUDdjDT6fquvkxicUIgoT0QSexMTdepN/9PKjPpBzYT6C/XYsX8cjzy137dsMiw4yzftwb7RiTq9qJyLpn63WghkafhdVPWP1253jKUKGy04OrO78jVibt2uYazbNZSKBccXgxPhnNixv1pdt6CJi7guKlOZgmRBxtXHUVxO+r7TdlHlrfAaP2GEtbxohQUn+HspuhYcHXXzEWZpCsPJokplmFOaur9eInoVEb0tsOyzAIYB7COim4horvnVAjPj/s17au5c2+w6nxrqjmQm3DnU40e3rcNHrrjftyxoscn64lmq2HjrT+/BmZctN1pWVJdsZcFRE14wNkN3UelDNrZ/aLOvfaJcqbFYBZndVWsE//Ytj+PU7/wtucDxWXC0GJwIH6CeudWpB0hHteDUSxNPEmQcM0082yBj53+cIGNlxQwSrO6dBcFrQrFk13yGlitwKhWzEAsj14IYoqlAIwvOZwEcrJ4Q0QsAfBXAFQA+DeB4SJp4KL9fuQ1v/sk9+O1DTwGo+r8rNmN9//CMm+jV+51p7zvISLGMwXF/YGqrLTgjRafy6aPbh4wxCRXbrYNj+Vs1qNcpynbVRaVbGMwCp72+91KFPRfVa565GMcfPKdmm1md4V7+tIOMo5wT+iF165Oa+hrNgcrKUS/IOK6lQBHFIpNlJWMlVOKU6ZhMC44xBidXa8EBnBuWuD3CZsJ1uFEMzrPhiBzFWwHczcxnAQARbQVwAYD/ymZ47c2j24cAANv2OWZjci8zOwbH8KFfrMCPzzgBpz1ryaSNr9WoC8JMd1FNVGwUA5k5+l2lRY5wGJ0oo1RhzOkupD6GYVeoFHLmYMNgHRz1nalgY4XuompUdbfN9A1KFRuFHGHt+aehkLNgu/25dGZ1Vb+b3/7byXjjj/7uPU/6dr008bLtq9kSRTDpW+gWHK/QX+Qg45RcVDEtMr4YnJQtOElcVLbNRqHVkiyqoMApV2rElhKEJvFTD8uKVw+oXWlkwZkLYJf2/GQAN2nPlwM4KO1BTRdGJ5xJpMftXaJ+r7uHJ2AzsDeQfTHdEQuOQ6ni1I7RPwf9YtaRdywmp3zjdhz/5T9nMoaRoiNUCjmrTiXjah0cNdbhgAWnVLGrAscXL1J7zHaz4BTLNgo5C11uOnghZ3lF+BS6BUc1yFQkbtWQr1pwKnFdVNr2HbnqWL1WDQ2icKKkiSevg9N4+2CGUJokCTKeSjE4E2W7Rlyq76JYjmnBkUrGAIDtAI4EACLqBPA8APdo62cDaF1L1TZD3e32BLrmjrmF7sIqpU5XqkHGM+t9B1GTh17wUL8b7HBbIwwMZ/fTqlpwLGNsTDXI2Jn0lIhRol3frlSudVEVTUGqbXY9LYVkUenoMTgqOFiRNGusy3NR2b59RHGthFpwah6YoYCg1akksOD4GkMaKivXI7M6OHEK/XE9C06LY3DKtTE46vlEuTYAuR4q+2q60+iM+xOAbxDRKwF8HcAI/L2nngNgXVqDIaLbiWiciIbdv8e0de8ios1ENEJEvyWiKV8pbswVON0dfk+gEj5hzfxaxbm/fRhLP3tjy44nFhwH9b2PaQJHv5h15HMti8Ep5MMnM1WmPp8jLQYn2GjT7KIyifd2s+CUKuY6ODq6BacrYN0xWUGiUE0Tr/jio6JZcGr3A1Stx5HaJViEktFFVV0flbgWHN84MquDE0PghATvtiLNuiYGp1zrhvJZcGIVYIwXi9SuNDrlvghgHMCtAM4EcBYz636VMwHckvKYzmbmWe7fMwCAiI4DcCGA9wBYDGAUwI9TPm7qqLvdapEt55ESPhOTLHD+7x9bWno8qYPjoASB34KjZb/krczr4HgCx7KMwkN9V5ZFyGt3e0YLjq0sONXlwY7jQBvWwXFdVPXQLTidgbYOwYDsqFTTxP1ZVHEnpA6fwFEuqsYULMKFd2zA2Vc94FueqA6OnvYdU7CkXQcnSZPMCodkUVktyKKyTRYc/4efS2zBmRlBxnVPVWYeYOZTAMwDMI+ZfxPY5K0A/ierwWmcAeAPzPw3Zh4G8AUAbyKi2S04dmJGVPVX90RSp99Uc1G1auKpSJo4gFoX1QcvW44ndg1761UMTpYMaS4q09fv1bZxXRYTZRsfvGw5lm/ai+ceMhcff/mRANxCf+VaC47JOtlunkk9iyoM3YITFEPBeKWo6BYcXeCYrCpB9N+yHi8UtZIxUJ00b1i13UQez0sAACAASURBVLc8iYtKn3SDk3MjohatizuW2F3ZDcG7LcmiMsTgBEVMUoEzUyoZRzrjmHk/M9fckjHznoBFJw3+l4gGiOjvRPRyd9lxAFZqx10PYALA0cEXE9GHiWgFEa3o7+9PeWjxUJYa9UNQv9exKeKiUph+qN+/9Ql8/Mr7DVs3f5yp5vudKNs44+J/YNWT+1pyvLJnwXH+/2XtLt96Jwanem5kIUDDXFSLZncCqJ6blptFNVaqeOM8bEEPXnXsAc52NmO87BfywPRwUcW14OTcJoyKoDsvKpZF6MhZGC9XfFabsYnG+9N/Wp1GC07jSTDsPdtJgox9hfsiv8w5Trr6xnOXxW1aaoodaomLKjA/TFTqxOAYigDWQ3c7T2ei9qJqFZ8BsAaOeHkHgD8Q0XPhdCzfH9h2P5wgZx/MfBGAiwBg2bJlk/oNKnO+slyo33o1BmdqnGDlCiMQH4nv3vp46sfJohfVFf/YjE0DI/jC65+Z6PXXrtiKP6/egb+v242Htu7Dcw7Ovm7lhCHIWCdowSmW7ZoA1mZRAievuajetuxgvP3EQ/Dmn9zjBQ47dXACxcWIfF2nTe9jWgicio2OfP1J40VHLMQHTl6KpQt6ATguP+V6TuqiAhx3V7Fk+6rNxrUI+VxU7v8oc2DYRKncpnGzdbzHk+2iShKDY7NRaLXCglPjoipVaqxn6vlEzCwqS+rgtB5mvld7+gsieieAf4JTObkvsHkfgKFWjS0JNRYc9zIzVnIuVFPFReVYC9KdQM3HSd+C84XfPuL8TyhwPv3rVd7jVglOU5CxTkfe8omG8VIldYEz7FoXGNXA1HNOPRrFkj8+LKeJGcVTe8d86cTBbttAmMBJa/StIUqQ8ZyeAr70huO85/kcQRlakrqoAMf1NVIs+6x3w+ON9+erZKwJHO8rbELgKIEaq5u4L8g4nmDJKosqzvWnbNfGvQCtqYNjKvQXvNlQz4vlSoJWDW32g0xAypUGUofh/CRXw6maDAAgoiMAdAJI38yQIsEYHMVUCTJWtKqxYzWLamq87yD3btiNi+/ckPlxSgEXVZCOQBn4MCHUDMq6MFG2q8GjbryNPkblogKq8SY7h8Y9N0bFZrMFx5gmnt15dvV9W7wmk2lRiuCiCqJPMs1YcGZ35bF2xxAuuPFR53lnPpJg0n9afgtOdBdVmABI0qqhGYGTfquG+JWMbdvsWstZlPl1MyhASpXaflPq5iN2kLFYcFqL29PqJAB3ACgDeDuAUwD8O5xx3kNELwXwAJzA5uuZuT0sOBV/DI7nopoyFpzWnOjVLKqWHC42f16zE39esxMfeukRmR5HnQ/1LDj6dxImhJpBTb7FcjWQ1aKqybsaZFydGI5ZMhtHLpqFt514iE8IFQ3nsckaluVpduEd63HcgXPwsqMXpbbPiYqNQoMg4yC6IBqeaEbgFHD/5r3e877ughcYHhVfkLGXJt74dWGxgYlaNdAUsuAoF1UMoV22bfTka6fJfI5qqpGnjalbeU0lY81FNac7nvCcCWniU0bgACjAaftwDIAKgLUA3sjMjwEAEX0UwJUAFsBJW//AJI0zMuouNjSLaorM9K0qvFfNopoa73uymDCkiesEY3DCtmuGYc2Cow5FPgtOdTJTHcR7OvP4+lueAwDYvHsEgCPWoo4vy2w9vR5PGjBzpG7iQfRJPIpLKYy+QBPPvu5CpP3p6fnGIOMIVpEwy4SaEGPFeujdxOPG4KRswVGCKV6auFloOYXy0v9d6phuPEOzqGIGGbcihmgqMGUEDjP3AzixzvqrAFzVuhE1h7EMf6AOzpTJomqZi8o93gz4YdVDfe/Fuhac6rmRucDRXA82V+8IAefirszgPVockJ5yG9WFlqWLqmzbqZ5XSuA1ShOvR3MuKn/bhznd0VxUulWh0xBkHEUzhN14tdxFlbIFB4hfgbhi12YuJdlPEkz7D34mamzFUsw0cUvSxIUm0O+kghaLah2cqXGC1ZsY0rzrbpc6OFmPT7kmw4RBZ87ymaezcFGpY+vF5MwuqmoMjt5yRLliSm4WVRRLR5aGu3LKFhy1r0LMXGX95zKcME0c8KefA0BfVzQLzrjPglProorybsI+x+p5ksxFZRIK9UjbRaX2GatVg21+v5NRB8c5rrnQXzGuBWeGxOCIwMkI/U6qHHRRRbDgbN0z6rkBsqZcZxxp/oinah2cIFlb1lTl3zDhUsgFY3DSt+Co9xjuoqpacBTdmsBRk1XZzaLq6Wyc5ZXlHWOpYqdqiVQWrLguKr0bVNCCs+lrp0feS9CC09ddaGjBqdjsi33SKytbMVxUYV+T+lkkteDEFSxpu6jUPuPEntSz4GQdw2ISIKHdxKWSsREROBmhB156MTiBION6aeJf+v1qfO76h7MboEY9wZHmZK8+h6ke3JZlbJQT22G24By+sBebvnZ6TRGuTASOaz2cqPhdVDnPglONt1Dj7PEJHOfSoc7l3o7G3u4sLeJlm1ONJfMsODFdVPp7HGkiyLiv2/95zurMY2i8VPc1weuJLs7iuKh0dAtuolYNTdXBibV5JPIxLS9Oob8wC062N0ImwR7monLWRf/AHHd08rG1CyJwMkJ3UdVYcEqNLThD4yUMNRGkGId6d76lFN1oWVpw0hRNWWa3VWz2JsGgcBn2iu/5L8Ljhr5OzaIHwKvHFlWDQr06OBZ5IqZHEzGq/sZw0Zl0eyfZguO4qFK04HguqpgCR3scJkyPWNjbcD9BC87sLicGp57LOHg8XzfxGC4qHf08VL+xOC4qq5ksqgwsOFZMy0s9gZO1BcT0ewlacHzNTGN8XK0QaFOBKRNkPN0wW3D8Z2A9gTNRtlsWhFyvx02a1owsu4mXbBudVjrF8LIs+Bd0PekTlkpxzlmW7yKcRQyOfrevgp0tIu+Wp6QFGY+5lgjdgqMu+ioupCeCBSdTF5Vtp3rBVp9PZxNBxvr39vQDZgEA1vzPayNN9MEsqlmdedjsWMx6O82fdTBd3+9ei+6iCu5Tr3kEtHeQcdwCdxWuZ8HJVuCYzuegmy+pBcciarvecEkQgZMRvhickAnTVD9EMVHhllU6ric40pw0qhac9N9XucIIue7HJsvPXReM4yXb++zPOOlQryJuPkco+rbLLgYHAMbL1VYiFmqDjJUFp9sQZKxqs0Sz4KQwcAPKKpZmDI4SubEtOOwXsADwo3c9H684xhGvUYQgUOvym+UKnuFiuY7A8Z8n+Zweg+P/H5ViqeIVeKwkCTKeQpWMgfgtCiqG4nrA5GVRhcXgOI+j7ztnxasH1K6IiyojTFlUwRO2vgWn0rI6OfXGkaaLKsssqlQDTDP83HX313ip4om+g+f1eCnJqnu3IotKxiWtcZ8KevdXMq7GWxwyvwcAvH5LaoxEU8OCo87fbLKomrHgOJ/rAX2dkYVNGMpltX8sPA5nz0h432N1/O6YLT/0mzA7gQXH12xzkuvgAAnSxJmN43Dq4GRswYkQg5O0W3vOstwbg+ktckTgZIR+YSiHuGbquULSLlxWj3o/+DQne/WDzaLuTj03W+x9ZSlwtPc+VqoY05GDd2nfuOkx3L1+IPVxKEvAmOaiUhdzdf5aRDjrpUfg6rNeiFMCVYLzFnlxQ70dkxeDk0Vsl7pBiVsHxx+D41rGEhx/To8/Buegud0AgCf3joa+5ubVO0PFx4uPXICrz3ohnr64pj9xXXyu9gSVjPNW+HndiMxcVDF+3hWbkTMEt7TEghMhBidplpr6nU/3QGMROBnht+CYBU49V8hE2W6Zi6pukHEWMTgZTHRpXmyyFTh+C44prsF0Yb9/096aZc0wUbE914MSOETVi6QeZJyzCC86ckHNPjrzOQyOqyDjCBacjD5WVeYgTeE8MOxYQxb0dsR6nR47pT7DuHEvAHDi0vm48D0neM8PdwOTNw6EC5wbH96GU56+0LjOCvkOG6FfxxIFGTeRJp5NkHE8oe10Ew+Jwcn4BtRc6M8/ZetWmzgCUhkmp3uquAicjDDVwYnloqrYDYNdf/fQUzj7qgeaGGXjcQTX3bBqGz56xf2JjqOETVo/Kn0/WbgnskDf91ip+h3r8RKmC9X8WfEm2rBjn/qdO3Dz6h0oaQKnWKrAoupEnLPIF2QcRmfe8lwmkQROCsL2n75/Jy6/Z5NvmfoM0/zedg2OAwAW93U1va+khojXHrfEezyvp4C+rjw2DYTXxtq5v4hnLOlLdrAQiiUbn7z6QZx/w5rpEWRM8YKDyyFZVK2w4JjGWWvB0R/HF57TvZqxCJyMMGVRBS0X9dw/pbLd0D1038Y9+Muju5oYpX98pufBSWP5xj3485odidKyKym7EvSxBe/ef/fQUzjr8hWJ9ptlhWldtBY1C05YNsRxB/a5Y2o8ef/t8X68/cJ7Qi+8e0YmsG7XMD76f/eDuRoYPFaq1KTz6kHGYXTmLewfdQROTwQXVRrX0jXbB/HF3632LVNB62m6qHYMjiNnUWwLjhqB7rJLYsEJQkRYurAXmwLFP3cNjuP0H9yJLbtHMVGx0dORwx/Ofgkued+ypo8JONex36/chkvu2ojz/rAGQLzYmGYqGWdRByduk0nbZuO4c7kWVDKOFINT/ZCSfC9iwRGMPL5zCGdf9UDoXaMy7XZqjRPjuKiKFcdFVb/uhY1iudJ0oFipjmUpONmPlSqwOVkRMyVCTD/cJOgCsGzbuHn1DnzjprUAgHOueQi3rNmZaL+tsOD0dOQwrsXghMUqvOPEQwDUz7hTfOLqB3Hvxj2hgagqG0qdLrPcwNWxiYDAIU3g1LPgFHIYHFcxONlbcMImpnICC86Nq7bjWzc/Frp+52ARB8zuTJzJo1u00jJELF3Qi40BC841y7di9bZBXHzXBgDOefXsg+fgVccuTnyc33z8xXjnCw4FUJuZBfgbaDaimWabWbio4tavmUwLjrlVQ5008RiFcNR+pnpV+WYRgZOQT137EG5YtR1rtg0a16sJqbczHypwwi7IerVbdQKWKjb++zcPY8f+ce0YjtjYPTKBz/x6lbGU++bdIzjv96vr/hiDvbL0cQXHOOa63ga1IoT7x0r49K9XYmC4GHoM5zgpW3DK+jgZH7nifvz49vXGY8bab8oCZ92uYZz3+9Ww7er3OrsrjzEtiyqfM5vyVfZLMUYtnLA0/GDrANXvaKxk+yainEWYqDSOt9BrxLSiVUOYRdP7rcQQzv921QP44W3rQtfvHBzHAUncU554rAocShRmXMtB87qxc3Dcd0OjRKui2WwtAHjeofPwLlfgjBj6acXteZTkdUm2j4IV00UVXujPyaL63q2Ppzm8mmMHqV/oL77AmepV5ZtFBE5CVHBX2GSiJqSejpy3TfCEtdl8Epe1arfq4n3XEwO48t4tOPe3D4OZ8Z1bHvfM1RffuRG/XLEVF9+5oWZfZ1/1IC67exMe3e4XYvpFMhjro08UF9y4xieqVErxoGYl+PpNa3Htiifxx4e3mz4K7X3Z7vsO/1Fdu3wr7tu4p+5+TOMOrzUUP8U67eDu9196Hy67exO2D457Y+7rKrh1cJQFR4vB0cROZ8FC3iJMVKK/jzAxFJwMVTG50YlyIJ0X0Sw4msBpRauGMCtWVTin973tHBzHkr7O2K9Tb3GWZsFJyxCxoLcDpQpjcKwqVEcDltQo9YiioKog7xwcr1mXtA5OXL2SlQUnVpBxWKE/d2zfu/WJzFKtTUKsfquG+N/LdK+FIwInIXqzQROqBkZvR7gFB3CCdu/f7M+Q8buI/MGexbKNtTuG8IO/PIFHnhp0j+Fc1J7aO1azf7Wv4LXikrs2eo+D70E//uM7h/GxK6tBxUow6ALnz6t3AGhcY0O34Fxz3xZs6B+u2ebT163C2y68p+5+/rFhN25/bJff0qRNbs1WAU67Do6aJCqVWgtOtaCc+UKVtwgdeSuSBUddaIOFAfeNTuDHt6/z2iooVG2V0aCLSo/BqXOF0FOoo8TgNG3B0QTOA1v2euddNcg4vboeOweLiQKM1fF1wdfMRP21Nz0b73vRYQCARbMdwdWvWUqVhUV9BnHr3IShxOv2/bUCJ85ESkSwSNVNmnwLThzXku3eaBpdVNrvdc/IBP7fX55IvSBn0LIONCr0F78+0XS34Egl4wR895bHscIVJWGTYbFswyLnTsjLojJcfM+55iEA/k7D+oV8IhCjUarYNXfCSvyYLka2IXNpx/5xXHDjo97zGhdV4KR/cMs+77FnwXFdVMzspdQGXSBBPHdb2cZnr38YnXkLj13wurqvMfGOi/4BALj1Uy+r7jtQX8b0OEjYZJh2qwa1P73uzeyugq8dRy4k3TNvWejMW5FicNSog6LuM9etws2rd+IDJy/1LZ/tWhlGi+WaC2VJq4MTRme+OpnOipRF1XCTuujWuDf9+G4Azu9Gt9xUbPZNPo1gZt/Eu3FgBPdv3ov9YyXMjxlgDGhBxilZcN7huooAYOEsR+AMDBdxlNv6QVlw9o1O1By3GdR3+9M71tesi980kxK56TLrRRVRBHt1f0LSxBU/vWM9fnbnRuRyhI+//Kh0BgrzDXEwJixpEHdeLDithYg6iegSItpMRENE9CARvc5dt5SImIiGtb8vTNZY9TRVk48acC7Gnfmc744h6p3DhC8GxnmNSagolDVl2/5aC446f0cnKhgvVXDDqm01AcK1LqrayVQtU4JBHVMXECMT9e9g1NjVRTnKpF0PfxaVufKvflfFzPjdQ09plW/DBE42Qca6xWa2VnofCMbg+N1VnflcLLdZ0C336PYhAPAynhR93Y4FZ2Si4rtw5i3L+wyjuKi6C7lId49pWnB0fK7KmCpKPwdHJ8p4xbdux3/+aiWAaKItjNld6VhwdHSBs2twHHc83u+5HVUV4+4IlrQozO0pGK1BS/q6EtWziROYrMgki4oocqyWlxYfUuhPsWPQsajtGQ6vJJ2ESK0atLHF+V7UtlkUXZ1KTBmBA8eatBXAywDMAfAFANcS0VJtm7nMPMv9O7/1Q3To0n74QR+4oli23fgJq5o9FFXg6BYc97G6Ky/bXDOBqWJr2/eFW3Du37wXn7z6QZx91YP4ayC1PErw81bX/aUEgzrmsBZsbApy1lGTT9h2+jh2DY3jkaf2192f30WlWXAmzALnljU7cc41D+GHf13njsc8Yf593QCGxsNL4kdlff8wrrp3i/d8dKLss+AA1c8vLIuqkLMcF1WMWKKgBUfFUD21zy+Awybhns6cJ9zrZ1E5l4+ejmgCp9mbxTBrablOUHwjdIFz97rdvnVJBI56j3osTFqeloVuLaQbVm7Ha773N7zv5/d5v8O9yoKTQpAx4Fzjlp97qm9ZR97CPz7/qtj7ylmUqO1CZllUUS04djQLznb3d5Uks7Qe5hicYKG/ZBacaiXj6S1wpoyLiplHAJynLbqBiDYCOAFAsspyGaEHV4ZZLYolG515y5eWqE/gszrzvom+XLG9Ym/6Ham6YKuJumJzzQSmgg7H3Loq+o9P7embWkpscKILtjkwWTbURKyOrY45pL2HRi6qqgXH/JnpYuTUb9+BwfGyz3UXJMyCM+oTOLWWnXW7nNifsLv9G1Ztx/EHz8VZpxwReuwonH/DGtz+WL82lqqLqi9owdHrWQQuWlFdVOrL1j/His2eMKgVONV2AD6B05HzVTIOQ7kxujtyke4en9o3hu37x/C0Od0NtzURFoekf49x70iLpQrgWrL6A1mASdw97H4JabmodOb1OALnJjf2CAA273YqG+8ZiV6PKCrNWLB0ckSJPoOsWjVEtYaq8yosTVyxdodjId1Up8p0Ekw3xMHWIUl7fXlBxtM8BmcqWXB8ENFiAEcD0Kt6bSaiJ4noUiIy1iQnog8T0QoiWtHf32/apGl8FpyQSd1zUeXImEWlfOgK/eJqtuBUgwmDd/N63ZMay4Ph/B0a94+5XGFv0neOUXsB2DAwjP1jpaqLyj3OUAwLTsVgwdk3OoH+Iee96xPz4HjjuyG9Ro8uynSrWlHbp4q1UGOvNxkGJ7skBNPmxyZsT6go95DJReW7K8tZ6CxYsVxU4+Xqd7TyyWr81LaAwOkqWF5ws34N11ONGxX6A1wXVcSL64v+96+RtjNhsuDoqfdA/J5kugAeGPJ/X81YcGZ16AInnYnaJCKVa0pZcNJyUSlee1y1nk7Sd5HLUSKxMtkWnHrNRXVLivoNb94dXmU6CSbxccBsf2ZfYguOVDKePIioAOBKAL9g5rUABgCcCOAwOBad2e76Gpj5ImZexszLFi1aZNqkaTo1gRNmtSiW61twjlnib3q3c7CIofESRoplPLWveiegLupqYixX7FoLjiZq9gbiLEwn8J4R/4X8T4/swKnfuQM3PeKkeZssOOdc8xAuuGFN1UU1VuuiimrB0bf75DUP4f2X3gfAHBBczzXjs+BoE5vuotL3qcSfEk/1esmooM1m2DdawpuedxBu+MRLvLGo967cDUMGF1XO56IidOTiBRkrS8ervn2HF4wLOEG+elxFR85Cl2uFCVpwFI1aNajts7jbDmISeePlik+oxrXgjGvnV1CQJrHgvP45B9a8Ns1PJixLSv220nJRKX70rufju28/vql96F3q///2zjw+qups/N9nlkx2kpAQICxhRxYBZRNUXHBDrRbclbauaLVvW1tb+9ZWWtvXpX1t9VeXWrXue+tW+2K1ilUUJIqCIItA2ANECFnIZJk5vz/ukntn7iSZkJAQzvfzmU9m7r1z59wn957znGc7SX2vg+rgtNZqYdeo8lRw4o/fvi/s6osPFK929u7hzuxzJQfoQn9xdDkFR0R8wJNAPXADgFKqWilVopRqVErtNLefKiLtu/BKK0ltjYvKjsERzyyqEXEKTpix8//F6Fvf5IrHmpYYaIix4ES8YnAcFpzYgdnr/o3NtrJq5KwpM6w4ieIYtlXUxllwrNTjtKA/YcC1RVMdnKZty7dWsHJ7JXtq6j3TLJs7p2vF9gRZVHtq6u2ZmKVMWIpOcw93xf4D76gq9jfQIz1opxvX1jfabbACRq01j4J+7zo4RhaVP7kYHPPY3VXxVqhcxyrVwYCPVFOZcXaUGa214AQdLqoOmG3H4iWDcEPUpdwm76JyWHBigkTbUlPmjjljWfrzmS7rT3vKZvHPTuaMMb0994kYVrn2JOD30TMjZJ+/Lfh8XUfBSSpN3OyvvZT82AnH5EF5AHy6aW+LE73W4tU/FWa5FRyndTCZOCfrntQuqoOIGP+tR4BCYI5SKtEoY/1XOr5X9SDUqiBjw0VlWXCUUq6baUShW8Ep80jxhiZrim3Biaq4WIR9LgWnZQtOmUfxLmgKGk00SBhVVI33dgyOOWD36ZHaaheVE6u9S0v3eNascXYWSrnrnDjrujQkiMG56aXl/Pofq8y2GsdbZv3mBsMDVXAaIlGq6xrJTU+xLSK1DRGq6xpJC/rtAfDxjzYBsfUsmh7LoF9a7aJqqoOT+FinZSHF77MHRGff6HRztCaLKj0l0GkWHGdmGrTBReVQmnZX19EjrUkBzAoFvb7SLEG/j4KskP0sQfsqOD3Sg0wb6r1ieHrQ327uMCeWotfWisx+6TpBxr4kFJzmLDixcYQXTOxPwCf85f0NjL71Td5d3f5rBAL0aqb4ZFsK/bVjbcwuSZdScIAHgCOAs5VSdsCAiEwRkREi4hORnsC9wEKlVPNpNh2Ey4KTwMJQWx8hFDCzqKIqzpIytNAdg/P5lgq8sCrYWrEk9Y1RV6cM7niVilr3LNRrsEs0eFtWhESDhFMJa7LgGL9dmJ3a6jo4XizduMfTRXXcXe/y8PsbOOOe9znjnvddMydn/I/z3LUxnc9jH5Yaba61aoY0GBlNzTzdsXJ0cuVjSym++Q2Kb36DF0u2eH/flHFOetCO2aqtj1IVbiQzNeCK4wJ3ob/YGJzWuqgswg2ROGWgKMcI7E13KDjBBC4q50KR2amJB3lLwQn6xdX+jsJLBuGGyAFZcJxWw/LqOorzM+zPB1IVOMVhkWvvcfqyKQN4+qop9mdLgU5rZ/eURVrQOG9br8Pvkzat6dVRq4knU+gPvBWt2NjLowbkMKaoB4vMTLxlCfrzZPCyaMf2G06SU3CMv7oOzkFCRAYC84DxQJmj3s2lwGBgAVAFfAHUARd3VltTWxGDYy3WZ1lwYlOS8zOaNPHUoI9313hr/FYgbdjs3GvqGuNu/EhU2Sm/scpLMuZSa1BsSDCYOt1xsWnivVtjwWlm8FlauidOMbF4/CNjqYnVZVWua3cqOM7tXumakahyBWDvrqprswXn347Z2ZOLN8XtX761gkm/fRuAnPQU/GY1YsuCkxUKxAWD+pvLogr6W6XgWP12XWM0ziI4sGc64FZegn6x72Xnb1oDpU/cqeSxWFlUfp8wsGdG3P7YjA+LtprFPS049RHPrMPmcFZvrYsJMh5kygkOrGie04LT3gqOiDDdYcUZXGDIvr2WaYjlQM/r87VNWWmL1afFcyYRZOy1TpxFbGjCwJ4ZnOoIyE45QIV/V1WYHfvCDI+ZCDdHckHGxv3pVS25O9FlFByl1CallCilUh21bjKVUk8rpZ5VSg1SSmUopfoopb6llCpr+awdg9PP7RxMr3q8xF7YsqwyTFFumhmDE40zBTpnNEcW5cQFB1vEponX1DdSWx9/U1pl3J3niURVs5V8Y7GUoZYCz7JTA7Y1pLqukdSgjx5pwVbXwYnF7xO+MONwvAg6Bv8dHlYkZ9sBz9W015RVuRSi8uq6Ztcu2lVVxwUPftRiKXOvQdeqswOQY7o80oJ+ausbqQ43GBacgHvQaK4OTijQsotKqabYrLqGCFsr3Cmr/XINC05GjAXHClp1jiWWEpSREmg+yNh2bxkxFsNiMgPf+dEMnrlqCg9/a6IrqL6t9UISWnCSLPTnY1V6uAAAIABJREFUzMayrKF1jREqw40uRS2UQEFrDc4qzx3hNnKSk2YErLfXMg2xWMp4W68i4PO1LYuqIwr9+aTVyxNEmrHgZMRNUIRzxhfZn2PjuZKlpNSolH/M4J4tHmtZ8JKxkvntGJw2NO4QossoOIcSzs7L6Yt9+8udPPZhKTsrw0Siir45acaMIaLsWcO54/vyxBWTXecbU9Qj4W/FpolHlbf7JD3FT3ZqgH2OIONklBtocje1NAsuzE6lKtyAUoqqukYyQ0EyQwFq6hpdMTK19REufmgxn242HtZEM/eJA3OJRBUfrf/ac79zBvWZw/TrtFJUuVLP4xWcrXv3UxVutC0S63fXcOa9HzR7nR+X7mHtrqpmj6mPRKlvjHL+gx/y4fpylFL29YLhogJTwTEtOJmhQJx1I9Fq4gG/eBb6u/XVL7jn7XWudljiDTdEWFvmbvcVxw7i6uMGcd0JQ+xtKQGfraQ4O3FrQEtkgbGwFACrs3x+3jHcdu4Ye3+/3HSmDc1n5qhC1/mrW1ECwAvPLKq4IOOWe2ynomS5cLfsMTziAx0WnANRTJyy6+jwJGvC1TMz+aUlWoOlOPXMTH7xUTCuvy3WmI4KMq6ui7TqPrHut4CHpvXdE4fyq2+M5pmrp/C3644BDDfwPReNBw68zMTS0j2kBf2MH5ADwK++MZpXr5/ueWxvM4khGQuOdUk6yFgTh9OC42W1sOqNFOWkmXVwlO2eGdsvh+OHu9PXBxXEm/ctGmLSxAG+9pgdhAJ+cjNSXBacRDV6EvHYh6U8sHC9bfL/8anDef2GY+OO65UdIqoMM221qTRkhAJEFdz8txV2uvnS0j18tOFru9hdIouJlYGwcod3SJXTouxUgl79bDuj+xqJdE4Lzl6PFO/y6noqww0MNmMsXv50m7cQYljawsrmDZEoG8trWFq6l5teXE7F/gbX7M0qzpaW4qe2wYzBCQUozA7x41OH28c5O9GRvZuSA4PWWlQxsVSPf7SJP7y91v7sjLXaVVXHH95e5wqYHdk7m5+fOYqjBuTa24wgYysGp+nclpWnJQXHMnNbA1FeRgpzpw70PNZpVWut2/Td1bv42d+X25+96uDEBRm3Igan3qXgGIqjVcPEy9XWFkIuBadjNJzHLp/E41dMtidcbVkctDVkpQb51TdGu+J+ksHfxiyqjpDbzFGFlFfXMf3Od1ixtfkQzg++MvqtUX3jk3VTg36+Pa2YaUPyOXpgnr39nPFFTC7Oi6uplCwrt1Uyqm82J40s5KbTRjB36kDG9c/xPNb6vydV6O8wqWSsFZw24IzB2Vhew40vfOZaVuC8B43VsPvlptkxOJYFx0vLPn10b74zrZgXrz0mbl+siwqg1KOgVCjgIyct6BrcE7m9AL59zECuOnaQHXxqceeC1fZaPHOO7sfYfvHWJStV8YrHlrKtopbMUIBic+b7fMkWrn3qU8BQcABKy2tQKj7Q2qK4ZwYpfh/rdxnXNXtCkT0rAreV7LXPt7u+e+MphpLgDPb2uu7y6jqqwo12EOnmPa2rOvqLV1fy4fryhPvrG6P2uVICvrgMtR5OC059xA4yFhFuOGmYfZwzSDctxc8b/3Us150whOy0gJEm3sKM01nQsMRcLPI3DmuKF8GAU8GJr4PTkoJjp9G2omN1uhNbcmVaXP7YUp79eIt973tVMq5tcM/GvZToynADP3huGX9dtJFHP9josoZZE4eN5ca9Nyi/fRQcp7LRUQacE0b0YsbwAvv/1FEKDsC3pxXTPy+95QM98HWhOjizxvbhptNGsLOyLmHco8Ury7Yzrn9O0vdEz8wUyqvr2F/fyA+f/4yN5TVU1xnvW1p+Bgx385dllYzsnUWPtCDXnzi0WfeTVRsnGWXFshi3pwXn3TW7mP/ayi61QnmXWarhUMIZmBmJKv7+6TZWba+MO65vTpqdRWV1vM4b9YkrJrNyeyUFWSHmf2O0ZwxKXWOUd1bv5M2VO203x+od8W6TUMBHekqIrXubqtX+/l9rSEkQvzG0MIu5Uwc2+5B7mWYBepkd6cemdWPG8AImFufFHWftL/26xvNBCvqFhogiOy1In5xUu+z8b7851uWysWb/l08v5q+LSl3nKMxORcQdcLw3Ro7ZqQHKq+uoDDeQm55CTnowbtkCi2lDenLG2D4s27yXIQWZ/O7NNTyzZDPThhhBnbEPb11jlFJzcAz6hZ2mgjP/7FFs2Vtrr9idluIn7AgyjiW2Mx/dtwej+xrKpfU/tFZu/6PDcvPBunI+XF/ORZOaVp226t9MGOA947MI+sXOCHQrOKYFp4XVDpsqvTZ7GOD+/7RUL8l5bjCup39eOvWRiKtwJlhZVM3H4Dz6wUZe+Ww7r3xmKMczRhS4vg/GPZqdGnDVCToQnBVnOzoGxypVUZjVNhdSR9NWC05HBBkDXH/iUJ5avMl+br0or65j1Y5Kfnr6yKTPn58Z4v++KOOcPy1i3a5qokqRFvTz8rJt7Ktt4NHvTGr2+zv2hakKNzKyT+vKvFmKrVfdq0S0Zx2c+sYod/zfal4o2UJ1XSP9ctO46rgDW+amvdAWnDZgmZ+dtWxiZ6XnH93Prg8SiSo7yNhpwTl+eIErJiI3PcgFE/u5ztMQUXbhv+FmoKaXqT4l4KNvTpo9cFfsr+etVTu5+rhBntdgdYaJ6u9MGJCTsLM/MsaqU5SbRkFWyJ7p5KQHqWuM2PEyG8tr4gYeq8ozGCXx+zrWJwoFfK7Cd5Zsx/WLH7DzMlII+nwut0dsx5WfFaLM7DRy0oN2kT0vLpkygLlTB3L3BeO5/sShnD2uL0tL99ixRbEBshX7G/jtP7809tVF2GWuLHzyEYX84qxR9uCWFvSzv77RiMHxyEwKNqMlWPdbXWOU376xiqcdC3he9sgS7l+43hX3Y32nbwtrPgV9PjvexqnLttaCE0nCgvPM1VNsd2JrLDjrHEuH7KwM8+WOSu57d32c0uVc3wu808Q37HbfD/sdClad7aLaz6D8DESEX541ilvPHtViG5vDqdR0dAyONQHoSAvOgeD3SZvcTW1JLW8txT0zPC3hFiWm9dlynyeDNTmz7uFte2t51VSuP1xfTnVdIw8sXM/63dWe319dZkyWj4gpBpuI3mZtnET1zbyw+t6731rriptsC2+t2smjizbaz/UbK3Yc0PnaE63gtAGr0N8QR9aI03IiAr873yhvbmVRWRac5mYlIsJd57nLojs77xS/0LeHdycWCvgpykmjKtxIZbjBjsI/blgBvzxrFGeO7cNYRzCz1Rla6Y7ZMYPu366dZi/+efvssZwxpjfTh/bkL9+aSP9ct6nacnNdZsZfVOxv4NNNFdQ1RplcnEdVuNF2AVhBvqnBpuq3WakBiswsn1DAl7Bj658XP2DnZaQQ8IuteBT3THcFHIMxo1q7s8p+n59EMObk4lx2VtaxZU8tL5RsoWTT3oTHllWG2WoqmLEFuVKDfj7dXEEkqsj0KCDXmoJ69ZEoexK4He99Z53rc3HPDHw+YfaEIm47Z7Tnd3w+R5q44760FJsWXVRWlklM22cfVcTPZx3h2jZtSD73X3oUAH98e23CkgAWzjW0Nuyu4Yx73gcMl5QzBm7HvjAPv7/R/vzn99bz7y938sG6JreiNWBYbNrTNLCFHS4qK/7mimMHcfl074lBMlhi6WiD/T4zo7FXF1ZwDkYhyGQozs+g9OsmN/WG3dW8WLKFF0u2sOnrGpZs3ENq0OfqM1vLrLF9OGZwT04ZVUh6ip+STXupbYhw8eT+hBuiPPFRKXcuWM1jMdZogBdLtvDMki34hFZbcL4xvoiRvbOSumcHmO7GFdv22QH2beWNFe6wgWWbK3hg4XrPOj4HG63gtAHLrB/0C7PGusumhwI+HrzsaPtzrAWnNQ/6XXOOpCgnDZ+4AyLLKsMJgyBDpgUHjHLht7zyBUG/ML5/DlccO4j7Lj2K17/XFDAcu6ZJ35hYHOegdfHkATxw2dE8fdVUThlVSHaaWxmyFJwrjx3E7847EoDXPjeCeM8zLVKWK8yywjRGovagmhkK2L/fXCaAM2jWIjXoNzMjTCuPIxCvV1aIiycPoCAzZHdm+ZkhxvSN77SKctJI8fuYMsidljnBDMq9683V/OSl5Vz7pHth+2G9MpkyKI9+uWlEoop7/72OvIwUV6YdwMTipuBeL2NNc9dtKTgLvihjxdb4AmJzpw5kd1UdhdkhOx28ON/owO6+cDxzjyl2Hf/rc0bbwdbWMh/O+ApLqW7JRXXyEYWkBHxcNsUdWHz3BeM9V2K3gpdXl1Xx4ifuAomRqOLlZVtt96LTDfu3T7e6jr3nogn2PffAwvUui1DJpr1c+XgJlz2yhA27q6mpa2TtTvdMeaPDolPXEKGuMcL2ilpXkb/24P5Lj6IoJ63ZYontgZU5GbsQY1ehrWtRdSTFPdPZU1PPmyvLuO/drzjpf9/jppeWc9NLy/np35bz/NItHD+soEUl34tJxXk8e81U/vKtifzlWxPt7d+eVgwY9yw0xSi+ubKMbRW1LPiijJteWs7bX+5kVN/sVi/2mpeRwoIfHM+QgtbXzMlJT+H5a6YC3jGdraUxEuW9NU2LWlvlIO5csJrXY+IlOwOt4LQB62FVCu6/9GhuOm2Eve/j/57JaaOblB5rLar31u5yfbc5LpjUn0U3n0R+ZoiV25uC0sr2hRkWU/jJGtBCQZ9tBbn5bysoqwxz8sjChJUve2YYVoxZY3tz3LD8pGqTxHbY1u863z/78RbGFvXg6IHGwG5lUlnFsGrqI3btlYxQgDFWNpRjZh/rCsv2UHDAcO9YqcdON9af5x7N7bPHuiw2BVkp/OCU4a504OGFmcw9ZiBrf3uGXU/Iwhr0/rHcMLs6s9mmDs7jrRtn8Py8Y/jz3Cal1iuW6toZQ/jH944lOzXA+P65cfubi9PoZyofP3lpeVwA9W+/OYbbzh3DivmnseS/Z3Lm2D5m2xLXz/jWMcW88+MTAMPCl50a4BdnNblkLNfrvBnN+9H75qSx9jdneGaZeOHssMv2hVm84WvbRfrn/6znh89/zn89t4xoVLG6rJJx/XqQEvCxJCaT7bTRvVl080ktKmBPL9nMmp3x8WobHS7MF0q2smp7JVEFg/LbFkSbiNPH9DHaeQD1dFrD92cawerNlfHvTMb265GUJeQHM4d1uFtvvDkRmvfkJ/zuzTWufYs37EEp+FUCy2cyTB+az9ypA5k6OI+RvbPJzwxRFW7E7xPW7Kxi5fZ9zHvyE655osQVWzegjQHdyWCFFLRFwYlGFW+t2sknm/a6+uwTR/ay31tuuc5EKzhtwHKtWL5LZx2b2Mqvfp8PpYxsHEjOrzxrbB/edWjHDRFlKwwFWSFK7zjTTjlP8fvtWW1ZZZhzxvflgcuOijvnxZONYFTL/XT/pUfz5JVTXIGfV7Rg6oy9Rqf1Z1ivLFKDxhpHd8wZS//cdHxiBByn+H2cOKLpAbj+xKEAZKcFOGVUIRMG5DDZEaz82g3HutwriWbCOelBdjkCa4N+Y8bYx4xBKXRYq/IzQ2SGArx304n2jPfBy47m2hlD4k+MMSjHKj0WTovS6L49WPubM+ifl8aco/p5Hj+mqAfL55+WtF//hOEFzJ5QFLd96c9ncmmM9eRns45g4+2zWm2uPvPIPiyff5orfqNnpnFvnTSysJlvJk8o4LMHrteXb+fivyxm9v2LWPBFGX965yv65abx/rpyHv5gA1/uqOSIPtkUmoN2bLYftDxZWFq6x7YEHeUIuP77MsO6mBkKUNsQ4dqnDKtccTuliB9sLpw0gNI7zoyzGnYVbj17tEuBbokfzBzOhtvP7MAWwZTBPZk7dSDFPdO57dwxcQrVpEF5dv9xoNx27hieu8bICj2ijzF5uG7GEJTCrsW1crtRqf3y6cWA8T/taAqyQqSn+F0Kv0XF/vpmM77+/J8NXP1ECRc+tBjArn9l9X29s1P5dPPedl1dvS3oLKo2IDG+9eOH5TO2qAfl1XVxCkzsrKo2CUvJuROK7HWUAM4e19ceHK2IeSsQrWJ/PQWZIXLTg+zd38CFE/t7WgVunz2W22ePjdt+0aT+3L9wPWt+c3qLHWXA76P0jjN5cvEm7nl7nSt7oyArxGe/PBVoyjYryk1jy55a+uel2RYngHkzhjDPoVj8/bppxMa7OdfXSQ36uerYQTz8wUbXMaeP6c197xpm3/556ayYf5rr953B4M4A4+w0QzFqqfqt5SK6YvogHl1k/HaPtGCcSTgl4GPhj09MavZ59ri+LZpyRYT/vWAc3585jBm/W2hvTxQE3tFZO21FRFj/P7P4/nOf8drn2+mVFWL7vrCtYDz6nXH8ddFG7lywhkhUcWS/HLbvC7NlTy1FuWlxmW/9ctNcwchOzh3fl9c+387bmTvJDAU4akAun252u/demHcM763dzZ0LVgOHroKjaRu3nTuGSFTh9wmXTB7AvCdL2FBew4bdNVxuupPam0nFeazfVc1/nTyM6rpGV/8O8L2ThvFLR3JCRyJiLLFSWl7D51sqEDGqYoeCPq56vIRVOyp5Yd4x9qT6q11VlJbvJyc9yD3/XkufHqn2chJzpw60a2BtvH0W+2ob8PuErA52z7aEVnDagHXzWeOiiPDq9dM9S8lfNKk/xw3LZ01ZFVc+XtJsBk8s4/r1YFB+BhvLa/jH945lVJ9sW4GaOthQdIb2MgbvDeU1+HzCez85kdr6SNIZFTedNoLvzxyW1Czw0skDOP/ofrY1yCLWLZZhKikjemfZssvLiA/0FZG4dXv6mwqRNUP471lHcNPpIxhxywL7mG9O6Md9767n6uMGecrXGaznXKrg3PF9+f2/1nrG9jixguVmje3Njn21/N8XZbzzoxnkpMdfQ7KxBvdcOJ67LxjX4nFWZ/Tlr09n3lOf8J+1u+PkfiggIhw1IIcFX5TxyLcn0btHKufet4j6iBGQPqIwizPvfZ9e2amcP7EfRw3M4T9rdzNjeAGrtle6AhdfvWE6+2obuPqJEr7Y1hRI3CsrxIWTBvDKZ9t5Z/Uupg/tyfDeWWSk+Fl404n2OmHpKX6unTGYktI9rNpRSa7HPanp3ljPq98n3H+p4WZuiEQPaB2y5rj+xKFcc/xgUgI+bj17FFcfP5iyfWHmPPAhN502wrNf7EhGFGbyzxVlvLtmUdy+HmlBfvD8Mu6aM47y6jp++Pxnrsngny6ZQFFOelxMpoh49o2dgRxoilhXZuLEiaqkpKTdz/v659v53rPLOOvIPvzpkng3UCLKq+vomZGSlHZ+37tfcc+/1/HJLTNtbbgy3GBXod1bU8+E297iO9OKmf+NA/cZdwQ/+/sKnv14M2/fOIOhvTKpqWtEpKneSnMopdhdXUevLLfCVnzzG0wuzuMFszhi2b4whdkhT9kqpRj0s38CUHrHma7te/c3tNip/PzlFTy9ZDPL559KVijg2Z6DSX1jlNqGSIuKWVelMRJl7/4G2/UXbogQbojYnWJVuMFYRsJUtsur6+iRFqQxolCouPsm3BChPhKloTFKZmqA2nrjXF/tqqK2PsrA/HQyUgLsqzX+19b9+MktM+mZGaIhEqWytqHNSxFoNAfKzspwp6T5b/q6hln3vE//vHSUwo5ZG5yfwV3nHcmFDy22a+UU5aRx78UTuOzhJQT9wue3ntpp1mIR+UQpNbHF47SCkzzLNu/lm/d/yE9OH8F3Txja7ud30hiJsnVv8xkeW/fup1dWaocHM7aV/fWN7Kmpp19u+wXOle0L0yMtGLcqdyKKb34DcCs4raW+McrOynCbK7lquhbRqGLTnv3tVrVYozmU2VkZtidLn27ayyUPL+FHpwzneycPo7S8xnYNj+6bTU56ChX766lvjHZqWQKt4NBxCg7AF9v2uVxGmq7NzsowUaXaLXBQo9FouiPLt1YwondWlw1ah9YrODoGp400twK4puvRVau8ajQaTVfiSI+K8YcqXdOnodFoNBqNRnMAaAVHo9FoNBpNt+OQUXBEJE9EXhaRGhHZJCKXdHabNBqNRqPRdE0OpRic+4B6oBAYD7whIp8rpVZ2brM0Go1Go9F0NQ4JC46IZABzgF8opaqVUh8ArwFzO7dlGo1Go9FouiKHhIIDDAciSqm1jm2fA3GV7UTkGhEpEZGS3bt3x+7WaDQajUZzGHCoKDiZQOzKX/uArNgDlVIPKaUmKqUmFhQUHJTGaTQajUaj6VocKgpONZAdsy0bqOqEtmg0Go1Go+niHCpBxmuBgIgMU0qtM7eNA5oNMP7kk0/KRWRTB7UpHyjvoHN3R7S8kkfLLDm0vJJDyyt5tMySo6PkNbA1Bx0ySzWIyHOAAq7CyKL6JzCts7KoRKSkNaWiNQZaXsmjZZYcWl7JoeWVPFpmydHZ8jpUXFQA3wXSgF3As8B1OkVco9FoNBqNF4eKiwql1B7g3M5uh0aj0Wg0mq7PoWTB6Wo81NkNOMTQ8koeLbPk0PJKDi2v5NEyS45OldchE4Oj0Wg0Go1G01q0BUej0Wg0Gk23Qys4Go1Go9Fouh1awdFoNBqNRtPt0ApOkohInoi8LCI1IrJJRC7p7DZ1JiJyg7n2V52IPBaz72QRWS0i+0XkXREZ6NgXEpFHRaRSRMpE5MaD3vhOwLzuR8x7p0pElonIGY79WmYxiMhTIrLDvO61InKVY5+WVzOIyDARCYvIU45tl5j3X42IvCIieY59h2X/JiILTTlVm681jn1aXgkQkYtE5Evz+teLyHHm9q7xXCql9CuJF0YNnucx1sc6FmNNrNGd3a5OlMdsjPT9B4DHHNvzTdmcD6QCvwMWO/bfDrwP5AJHAGXA6Z19PQdBXhnAfKAYY4JxFsaSI8VaZgllNhoIme9Hmtd9tJZXq2T3L1MGTzlkWQUcb/ZhzwDPOY4/LPs3YCFwVYJ7T8vLW2anAJuAqWZfVmS+usxz2elCOpRe5uBUDwx3bHsSuKOz29bZL+A3MQrONcCHMbKrBUaan7cBpzr23+bsOA6nF7AcmKNl1ipZjQB2ABdoebUoq4uAFzAUakvB+R/gGccxQ8w+Letw7t+aUXC0vBLL7EPgSo/tXea51C6q5BgORJRSax3bPsfQ8jVuRmPIBgClVA2wHhgtIrlAX+d+DlM5ikghxn21Ei2zhIjI/SKyH1iNoeD8Ey2vhIhINvBr4Ecxu2Jlth5zkEb3b7eLSLmILBKRE8xtWl4eiIgfmAgUiMhXIrJVRP4kIml0oedSKzjJkYlhenOyD0Ob17hpTlaZjs+x+w4bRCQIPA08rpRajZZZQpRS38W41uOAvwN1aHk1x23AI0qpLTHbW5LZ4dq//RQYjOFieQh4XUSGoOWViEIgCJyH8UyOByYAt9CFnkut4CRHNZAdsy0bw0ercdOcrKodn2P3HRaIiA/DnF0P3GBu1jJrBqVURCn1AdAPuA4tL09EZDwwE/iDx+6WZHZY9m9KqSVKqSqlVJ1S6nFgETALLa9E1Jp//59SaodSqhy4m9bJDA7Sc6kVnORYCwREZJhj2zgM94LGzUoM2QAgIhkY/uuVSqm9GG6GcY7jDxs5iogAj2DMguYopRrMXVpmrSOAKRe0vLw4ASNofbOIlAE/BuaIyKfEy2wwEMLo23T/1oQCBC0vT8znayuGnGLpOs9lZwcqHWov4DmMyPkMYDqHUdR8AnkEMCLlb8ewSKSa2wpM2cwxt92JO5L+DuA9jEj6keZNf1hkuAAPAouBzJjtWmbxsuqFESybCfiB04Aa4Bwtr4QySwd6O16/B14y5TUaqMRwK2QAT+HOCjrs+jcgx7yvrL7rUvMeG6Hl1azcfg0sNZ/RXIzMqNu60nPZ6UI61F5AHvCK+QBsBi7p7DZ1sjzmY2jxztd8c99MjKDQWowshWLH90LAo2bnsRO4sbOv5SDJa6ApozCGudZ6Xapl5imvArMzrDCvewVwtWO/llfLMpyPmUVlfr7E7LtqgFeBPMe+w65/M++xpRhukgqMyccpWl4tyi0I3G/KrAy4F0g193WJ51IvtqnRaDQajabboWNwNBqNRqPRdDu0gqPRaDQajabboRUcjUaj0Wg03Q6t4Gg0Go1Go+l2aAVHo9FoNBpNt0MrOBqNRqPRaLodWsHRaDRdGhFRInJeB55/ovkbxR31GxqN5uCjFRyNRtNhiMhjpvIQ+1qcxGn6AK93VBs1Gk33JNDZDdBoNN2et4G5MdvqW/tlpVRZ+zZHo9EcDmgLjkaj6WjqlFJlMa89YLufbhCRN0Rkv4hsEpHLnF+OdVGJyC/N4+pEpExEnnDsC4nIH0Vkp4iERWSxiBwbc77TRWS1uf99YHhsg0Vkmoi8Z7Zpm4g8ICKxKyRrNJoujFZwNBpNZ/Mr4DVgPPAQ8ISITPQ6UETmYKyO/V1gGHAW8LHjkLuAC4ErgAkYa1ctEJE+5vf7Y6wd9Jb5e//P/I7zN8YC/zLbNA6YbR776IFfqkajOVjotag0Gk2HISKPAZdhLC7q5D6l1E9FRAEPK6WudnznbaBMKXWZ+VkB5yulXhKRG4F5wBilVEPMb2UAe4GrlFJPmNv8wFrgWaXULSLyP8B5wAhldn4icgvGKsiDlFKlpkWoQSl1pePc44FlQKFSalf7SEej0XQkOgZHo9F0NP8BronZVuF4/1HMvo+AMxOc60Xg+8BGEXkTWAC8ppSqA4ZgrHC8yDpYKRURkY+AUeamI4DFyj2zi/39o4GhInKhY5uYf4cAWsHRaA4BtIKj0Wg6mv1Kqa/a40RKqS0iMgI4GZgJ/C9wq4hMoUkJ8TJLW9vEY18sPuBh4A8e+7Yl12KNRtNZ6BgcjUbT2Uz1+PxlooOVUmGl1BtKqR8Ck4DRwHTgK4zsLDuo2HRRHQOsMjetAqaIiFPRif39T4HRSqmvPF61bbg+jUai00OKAAABPUlEQVTTCWgLjkaj6WhCItI7ZltEKbXbfD9bRJYCCzHiY04GpnidSES+g9FvLQGqMQKKG4B1SqkaEXkAuENEyoGNwA+BQuB+8xQPAj8C/igi9wNjgWtjfuZOYLGIPAj8GagCRgJnK6XmJX/5Go2mM9AKjkaj6WhmAjtitm0D+pnv5wNzgHuB3cDlSqmlCc5VAfwU+D1GvM0qYLZSaqO5/6fm378CORiBwacrpXYAKKU2i8hs4G6MYOVPgJuBp6wfUEotF5Hjgd8A7wF+YAPwcrIXrtFoOg+dRaXRaDoNZ4ZUZ7dFo9F0L3QMjkaj0Wg0mm6HVnA0Go1Go9F0O7SLSqPRaDQaTbdDW3A0Go1Go9F0O7SCo9FoNBqNptuhFRyNRqPRaDTdDq3gaDQajUaj6XZoBUej0Wg0Gk234/8DEAkgzpgTN2cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\", fontsize=14)\n",
    "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
    "save_fig(\"double_dqn_rewards_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dueling Double DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "K = keras.backend\n",
    "input_states = keras.layers.Input(shape=[4])\n",
    "hidden1 = keras.layers.Dense(32, activation=\"elu\")(input_states)\n",
    "hidden2 = keras.layers.Dense(32, activation=\"elu\")(hidden1)\n",
    "state_values = keras.layers.Dense(1)(hidden2)\n",
    "raw_advantages = keras.layers.Dense(n_outputs)(hidden2)\n",
    "advantages = raw_advantages - K.max(raw_advantages, axis=1, keepdims=True)\n",
    "Q_values = state_values + advantages\n",
    "model = keras.models.Model(inputs=[input_states], outputs=[Q_values])\n",
    "\n",
    "target = keras.models.clone_model(model)\n",
    "target.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=1e-2)\n",
    "loss_fn = keras.losses.Huber()\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
    "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
    "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
    "    target_Q_values = (rewards + \n",
    "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 74, eps: 0.0100"
     ]
    }
   ],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = []\n",
    "best_score = 0\n",
    "\n",
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step)\n",
    "    if step > best_score:\n",
    "        best_weights = model.get_weights()\n",
    "        best_score = step\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
    "    if episode > 50:\n",
    "        training_step(batch_size)\n",
    "    if episode % 200 == 0:\n",
    "        target.set_weights(model.get_weights())\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAERCAYAAABVU/GxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXeYG9XVuN/jtusKLutGsbEBG0wLGNNbgAAhhUDoCQk1hI+EhC9fQvLDiRNq6L2GHiBAcEgIxmDTTDdLx93Y2Ma4rHHb4m3a8/tjZrQjraQdydKonfd59Ozs3Jm590oz98wp91xRVQzDMAwjXbrkuwGGYRhGcWICxDAMw8gIEyCGYRhGRpgAMQzDMDLCBIhhGIaRESZADMMwjIwwAWIYhmFkhAkQwzAMIyNMgBiGYRgZ0S3fDcglgwYN0pEjR+a7GYZhGEXF+++/v0ZVqzo7rqQFyMiRI6murs53MwzDMIoKEVkS5DgzYRmGYRgZYQLEMAzDyAgTIIZhGEZGmAAxDMMwMiI0ASIiFSJyn4gsEZFaEflQRI7xlR8uInNFpEFEXhGREXHn3i8iG0VkpYhcHFa7DcMwjMSEqYF0A5YBhwBbABOBJ0VkpIgMAia7+wYA1cATvnMnATsAI4DDgN+KyNHhNd0wDMOIJ7QwXlWtxxEEHv8VkcXAXsBAYJaqPgUgIpOANSIyVlXnAmcAZ6rqOmCdiNwL/BSYGlb7DcMwjFjy5gMRkSHAjsAsYBzwsVfmCpvPgXEi0h8Y7i93t8clue55IlItItU1NTW5ar5hGAVKpE158r1ltEbaOj12UU0dt760gJraphBa5rBgVS23vbyAtfXNodWZK/IiQESkO/Ao8JCrYfQBNsQdtgHo65YRV+6VdUBV71HV8ao6vqqq04mUhmGUGI/NXMpvn/6Eh97ufC7cva8v4vpp83n2469CaJnDDdPmc92L85k2e2VodeaK0AWIiHQBHgGagQvd3XVAv7hD+wG1bhlx5V6ZYRhGDOvcN/v1DZ2/4UfaFICm1s61lWzR0BwBoNWtu5gJVYCIiAD3AUOAE1S1xS2aBezuO643MBrHL7IOWOEvd7dnhdJowzBKlu5dnSGwJYC5y+hI2BrIncBOwHdVdZNv/7+AXUTkBBGpBP4IfOKatwAeBi4Vkf4iMhY4F3gwxHYbhlGCdBEB8iNAtPgVkFDngYwAfgbsAawUkTr3c7qq1gAnAFcA64B9gFN8p/8Jx6m+BHgNuFZVLQLLMIzNQnFG8eZ8CJDQa8w+YYbxLgEkRfl0YGySsibgLPdjGIaRVVpaS2E4Dx9LZWIYRllS29jC8nWOJT0vPpASsGGV9HoghmEYyfjebW+yeE09kCcfSOg1Zh/TQAzDKEs84QH58YGUAiZADMMoe1oi4esDJWDBMgFiGIbREuJEQg8tAQliAsQwjLLHJhJmhgkQwzDKHpsHkhkmQAzDKHtsJnpmmAAxDKPsyYcTvRQwAWIYRtlj80AywwSIYRhlT7NFYWWECRDDMMoei8LKDBMghmGUPeYDyQwTIIZhlD0WhZUZJkAMwyh7zISVGWEvaXuhiFSLSJOIPOjbf7pvgak6EWkQERWRvdzySSLSEnfMqDDbbhhG6RLmmugeWgJxWGFrIF8BlwP3+3eq6qOq2sf7ABcAi4APfIc94T9GVReF12zDMEqZfJiTSsGEFep6IKo6GUBExgNbpzj0J8DDWgpxboaRB259aQFdugj/c9j2+W5KUWBDTWYUnA/EXTv9YODhuKLvishaEZklIj/PQ9MMo2i4ftp8rn1hXr6bUTTkQ3yUgsgqOAECnAG8rqqLffueBHYCqoBzgT+KyKmJThaR81w/S3VNTU3uW2sYRtFjJqzMKFQB8pB/h6rOVtWvVDWiqm8BNwM/THSyqt6jquNVdXxVVVUIzTUMwyhPCkqAiMgBwHDgn50cqoDkvkWGYZQD+YiIsiisNBGRbiJSCXQFuopIpYj4Hfk/AZ5W1dq4874vIv3FYQLwS+Df4bXcMIxSxkxYmRG2BnIpsAm4BPiRu30pgCtYTiLOfOVyCrAQqMVxrv9VVRMdZxiGkTYlMJbnhbDDeCcBk5KUNQJbJilL6DA3DMMw8kdB+UAMwzDyQl5MWMWv95gAMQyj7CkFh3Y+MAFiGEbZY070zDABYhiGkQdKQH6YADEMwyiFwTwfmAAxDKPsyYdD20xYhmEYJUB+kikWvwQxAWIYhmFkhAkQwzDKHovCygwTIIZhGCGicX+LGRMghmEYlMbM8LAxAWIYhhEiUUFVAgLLBIhhGAbhj+fFLz5MgBiGYQClMaCHjQkQwzAMwvOBlJAFK5gAEZFDRGQf3/8/FZE3RORuEemTu+YZhpEOza1ttETa8t0MIwXeBMJymkh4EzAUQETGAHcDnwD7AdcGrUxELhSRahFpEpEHfftHioiKSJ3vM9FXXiEi94vIRhFZKSIXB63TMMqJMROf56C/vpLvZhQlxT+ch0/QFQlHA5+62ycA01T1AlcreRr4ecDrfAVcDhwF9ExQvqWqtibYPwnYARiBI8heEZHZqjo1YL2GURaowsqNjfluRlESlkmp7ExYOMK5q7t9OOAN3CuBgUErU9XJqvoM8HXgFjqcAVymqutUdQ5wL/DTNK9hGIaRlLBMSlEBEkptuSWoAHkPmCgiPwYOAp5394/EESLZYomIfCkiD4jIIAAR6Q8MBz72HfcxMC6L9RqGYRhpElSA/ArYA7gNuEJVP3f3nwi8lYV2rAH2xjFR7QX0BR51yzwn/Qbf8RvcYzogIue5fpbqmpqaLDTNMIxyIDQTludELwEVJJAPRFU/A3ZLUPQbILK5jVDVOqDa/XeViFwIrBCRfkCdu78f0Ojbrk1yrXuAewDGjx9fAj+RYRilRLsJq/iHp82aB6Kqjarakq3G+C/t/hVVXQesAHb3le8OzMpBvYZhlCmloBGETVINREQWE9DPo6qjghwnIt3cOrsCXUWkEmjFMVutBxYA/YFbgFdV1TNbPQxcKiLVwBDgXODMIHUahmEUEtpho3hJZcK6zbfdB7gYmAm87e7bD5gAXJ9GfZcCf/L9/yPgz8A84EpgMLARmAac6jvuT8CdwBJgE/BXC+E1DCObhGZSKqEorKQCRFWjgsGd9PdXVb3Sf4yI/J40oqFUdRLOnI5EPJ7ivCbgLPdjGIaRdcyElT5BfSDHA08m2P8U8L3sNccwDCM/hCU/2qOwil9iBRUg9cChCfYfCjRkqzGGYRilTinNRA+ayuRG4HYRGQ+84+7bF/gJyU1ShmEYRUMpaARhE3QeyDUi8gVwEXCSu3sO8BNVTWTaMgzDKCrCM2GFW18u6VSAuKG33wJeMmFhGEapEl4yxdKZid6pD8TNjjuZJKlDDMMwjPIkqBP9Y2D7XDbEMAwjr4Q7DaSsUplMAq4XkeNEZBsRGeD/5LB9hmEYoRB6Ovfilx+Bo7Cec/9OJlZOC7FrhRiGYRQlpTCgh01QAXJYTlthGIZRJpSSnAoaxvtarhtiGIaRT0Ib2LV0ZqIH1UAAEJHhwLZAD/9+VZ2RzUYZhmGETSkM6GETSIC4guMx4GAcQe35PjzMB2IYRlFjEwnTJ2gU1k04Kw/ujJP76iCc5WznAEfnpmmGYRilRzlGYR0CHKuqc0VEgRpVfVNEmoDLcNbvMAzDKFpKYUAPm6AaSE9gjbu9FmfhJ4DZJF4r3TAMo6gIbR6Il849i/V9XlPHkq/rs3a9oAQVIHOBse72R8D5IjIC+B9gedDKRORCEakWkSZ3kSpv/74iMk1E1opIjYg8JSLDfOWTRKRFROp8n0DL6BqGYQQitFxYsX+zweHXv8Yh176avQsGJKgAuRkY6m7/BSe54iLgAuAPadT3FXA5cH/c/v7APcBIYARQCzwQd8wTqtrH91mURr2GYRhGlgk6D+RR3/YHIjISRyNZqqprkp2X4DqTAdx1Rbb27X/ef5yI3AbY3BPDMEIjtCisEloTPZAG4jcnAahqg6p+kI7wSJODgVlx+77rmrhmicjPk50oIue5ZrLqmpqaHDXPMIxSI7R07iHXl0uCmrCWi8g8EblbRE6NFyjZRER2A/4I/J9v95PATkAVcC7wRxE5NdH5qnqPqo5X1fFVVVW5aqZhGCVGKWTHDZugAmRH4FqgN3ANsQLllGw1RkS2B54HLlLV1739qjpbVb9S1YiqvoXjk/lhtuo1DMMIi/YZ78UvsAIJEFVdqKp/U9Ufqeo2wDjgLeAs4NHUZwfDjeqaDlymqo901iSc2fCGYRhZIWyTUimYsIKmMukCjMfJynsocADwNY7weCVoZe7yuN1wUp90FZFKoBUYArwM3K6qdyU47/vADGA9sDfwS9KL/jIMw0hJCYznoRN0Jvp6oBFnXZB/AOer6pIM6rsU+JPv/x8Bf8b57UYBfxKRaLmq9nE3T8EJ/a0AvgT+qqoPZVC/YRhGQsJKpliOqUw+BfYCJgD1QJ2I1KcbhaWqk3BWN0zEn1Ocl9BhbhiGkYxCH6BLwWkf1AdyAM5kv4twtJFfA8tE5BMRuTmH7TMMwwiF8MJ4i19weAReD0RVNwHTReQznDkaxwIn4zjUL8pN8wzDMDJDCjTMpuxMWCJyIo4D/TCckN5VOE7tX5CGE90wDCMs0h2gQ4/CCre6nBBUA7kFJ7XIzcCrqjo3d00yDMMoXUpBcHgEzYWVs5nnhmEYuSBdE1Zo6dyja6KHUl1OCToTHREZIiK/EZE7RWSQu+8AEdkud80zDMPIjMI3YRW/BAmaTHEvYB5wOnA20M8tOhK4IjdNMwzDCI+w10QvBYJqINcBN6vqN4Am3/4XcGalG4ZhFBSFGoXVno43r63ICkEFyF5AopnfK3DSkBiGYRQU6Zuwwh3RS0B+BBYgm3AmEsYzFlidveYYhmHkBzNhpU9QAfJvnDxVFe7/6q5K+Ffg6Ry0yzAMY7NIOwortDXRNeZvMRNUgPwGGADUAL2AN4CFOGlNLs1N0wzDMNIn85neZsJKl6DzQDYCB4rIN4E9cQTPB6o6PZeNMwzDKDVKQXB4dCpARKQ7jsZxhqq+jLNuh2EYRYCqIgUbjpQbvPkVhWvCCre+XNKpCUtVW4DtKC3BaRhlQSkMUumS6QAd9ldVCj9NUB/IQ8C5uWyIYRjZpxQGqbCwdO7pE1SA9AbOE5GPROQ+EbnF/wlamYhcKCLVItIkIg/GlR0uInNFpEFEXnHXSPfKKkTkfhHZKCIrReTioHUaRjlTCpE+6VLoPW7XkAq9pZ0TNBvvTsAH7vaouLJ0voWvgMuBo4Ce3k43t9Zk4BzgWeAy4AlgX/eQScAOwAhgKPCKiMxW1alp1G0YZUfxD1EZ4IXJptn7sDWDUvhtgkZhHZaNylR1MoCIjAe29hUdD8xS1afc8knAGhEZ66aOPwM4U1XXAetE5F7gp4AJEMNIQQm85GZMoSZTLKXfJHA23hwzDvjY+0dV64HPgXEi0h8Y7i93t8clupCInOeayaprampy2GTDKHxKyd4elEx7HPrAXgI/TaEIkD7Ahrh9G4C+bhlx5V5ZB1T1HlUdr6rjq6qqst5QwygmSultNyhRH0N+m9EppSDcC0WA1NGeIt6jH1DrlhFX7pUZhmEkJP0w3nAXlCoFCkWAzAJ29/4Rkd7AaBy/yDqcrL+7+47f3T3HMIwUlNBYFZhMBUF4Ybzh1pdLkgoQEXlZRLZ0t8/wJVLMGBHpJiKVQFegq4hUikg34F/ALiJyglv+R+AT39rrDwOXikh/ERmLMyflwc1tj2GUOqVgJkmXdhNWYfe9pAUIzkJRvdztB4AtslDfpTip4S8BfuRuX6qqNcAJOKsbrgP2AU7xnfcnHKf6EuA14FoL4TWMzimFQSpjCrTvpfSbpArjnQtcKSKvAAKcJCIbEx2oqg8HqUxVJ+HM6UhUNh1nfZFEZU3AWe7HMIwElJJtfXMo9CgsTzMqdA0pCKkEyM+Bm4Hv4/wmV5P4t1EcE5NhGHkk0QBY/ENU+mQahRX6RMIS+HGSChBVfQvYG0BE2oBRqmqrDxpGgZLw7a4URqkMSbfvNpEwfYJGYW2Hs5iUYRgFSqIBs4TGqsAUumlI4/4WM0FTmSwRkSEi8j/Azjh9nw3coaqrctlAwzCCkVgDCb0Z+adY0rmXwG8TSAMRkQNwlrA9DSdyqhE4HVggIvvlrnmGYQQl4YBUAoNUpqTtAwlpRC8FweERNBvvdcDjwPmq2gYgIl2Au4Drgf1z0zzDMDaHQjfn5IKMo7Cy2oogNRX/bxNUgOwB/NQTHgCq2iYiNwAf5qRlhmGkRSJhUUpvu+lSzn0Pi6BO9A04jvR4tgPWZ685hmFkioXxOnimqLTXA7E10dMmqAbyD+A+Efkt8BbOfXkgztyQx3PUNsMwNpNyDOPNvMsh+UBCrS23BBUgv8WZjX6/75wW4E6ctCSGYeQZ00BiKdQFpUqJoGG8zcBFIvJ7nCy5AixU1YZcNs4wjOCYD8Sh0LscNbGVwI8TVAMBwBUYn+aoLYZhbAaJNZDiH6TSJdNxOaxvqpRMWIWyHohhGJtJwgGpFEapDCnUVCalhAkQwygRSsEkkg3as92meV7IEwlL4ecyAWIYJUwJjFFpU+gDc3uYcfFTMAJEROriPhERudUtGykiGlc+Md9tNoxCwnJhxVLoubBKgcBOdBHpAewCDCZO8KjqlM1tiKr28dXVG1gFPBV32Jaq2rq5dRlGKWJO9FgKdiJhtL7i/20CCRARORJ4BEd4xKM4a5xnkx8Cq4HXs3xdwyhdEgmQ4h+j0ibTgTk0YVtCv0lQE9btwH9xUpf0Anr6Pr1SnJcpPwEe1o53whIR+VJEHhCRQTmo1zCKloTzQPLQjkKhXIRnPjWZoAJkGHClqi5R1UZVbfJ/stkgEdkWOAR4yLd7Dc7qiCOAvYC+wKNJzj9PRKpFpLqmxtbAMsqHhCaschlFfWQ8zyJ0E1aWrpfHnzioAPkv4aVsPwN4Q1UXeztUtU5Vq1W11V3A6kLgWyLSL/5kVb1HVcer6viqqqqQmmwY+cec6A4FP5Eww2SPSa+XlatkRlAn+vnAoyKyF/AZTh6sKKr6cBbbdAZOksZUeN+ZZLFewyhqylHbSEW55MLK5+8eVIAcBRwOfBtoIFboKZAVASIi+wNbERd9JSL74KSNXwD0B24BXlXVDdmo1zBKAdNAHDRzI1YoZN2ElZ3LZERQE9Z1wG1AX1Xto6p9fZ8OZqTN4CfAZFWtjds/CpgK1OJoQE3AqVms1zBKknIM483chFWcM9Hz+ZIQVAPZErhLVetz2RhV/VmS/Y9j644YRkoSO9HDb0ehUDYmrDy+JATVQJ4GjshlQwzD2DwsjNchUxNReNl4s+xELwINZBFwhYgcDHxCRyf6DdlumGEYaWJhvEDhaxKllEwxqAA5C8f/sD8dw3kVMAFiGHmmBMajrJJ+KpPi/AYLXgNR1e1y3RDDMDYPW9LWw1vxL5Ozck+2Y8SKwQdiGEaBY0vaOmTc5yKVIG2FroGIyC2pylX1l9lpjmEYmZJ44CwPCfLRsvX07N6VMUP7RveVR8+LYyLhrnH/dwfGuud/kNUWFREzF69lmwE9GbZFz3w3xTDKeiLhcbe/CcAXVx+bsZM6tHkg2Y7C8m+rIhJego6gPpDD4veJSCVwH2Wccv2ku9+mZ/euzLns6Hw3xTASUibyI4ZMB+bQ1gPJ4URCVQhRfmTuA1HVRuAK4P9lrznFx6aWSL6bYBhAYlNGuWggiSjUBaWyjibcDIXNdaJXAX06PcowjJxjKxI6aJad1Nkml1FYYftDgjrRL47fhbNGyOnAZi9naxjrG5pZW9/MqCp7H8kmRftWvRkUfBCWl849Sz+O5lEDCepE/0Xc/21ADfAAcFVWW2SUJUfeOIOa2ia+uPrYfDelaLFcWLGk2/WinUjo224rRA3EJhK209QaYdWGJrYdmIuVfIPz1fpNbNGzO70rgr4DFDY1tVld2LIsSZwLqzgHxc2h3Umdpg8kB21JVU/WTFjqN2Fl6aIBycgHIiLdRKQsbQ0XP/kxB1/7Co15dp6feNfb3P3a53ltg1FYmAbiUOhCM+tRWNm5TEakFCAicriInBS37xKgDlgvIlNFZMtcNrDQmDZrFRC+qhjP+oZmvq5vzmsbjMKisIfN8EnfhJWTZuSc+DDeMOlMA7kE2Nr7R0QmAFcCjwC/BXanzMJ4myNtAETymT8AiKjS4rbFMMDCeKNk/IYf7peVkyiskPvQmQDZFXjN9/+JwFuqeq6bwv2XwPey1RgReVVEGkWkzv3M85WdJiJLRKReRJ4RkQHZqjcTWiP5fTLb2vLfBsMoRDKOwgrhcdJcqAsFrIFsCaz2/X8AztKyHu/hrGGeTS50l83to6pjAERkHHA38GNgCM667Hdkud60yPfbf0Q1qg0ZBiRJZVLGhq1y6bkm2Q6DzgTICmA0gIhUAN8A3vaV98VZnzzXnA48q6ozVLUOmAgcLyJ9OzkvZ7Tk24TVVjomrGINnyw0zInukOk8izC+qlzM2YhVagrLhPU8cI2IfBP4K1BPbO6r3YCFWW7TVSKyRkTeFJFD3X3jgI+9A1T1c6AZ2DHLdQemNY+Dd5srvFpKxIR19kPV+W5CiZAojLf8KGgTVg7q8wf0FNpEwj8Ck4HpOJFXP1FVf+jPWcC0LLbnd8BsHOFwCvCsiOyBky5lQ9yxG3A0oBhE5DzgPIBtt902i02LJZ+Dd0Q9AVIaGsjLc9utpGFnEy0lEmsg5ShCHNKOwipScZsLoRSUlAJEVdcAB4vIFkCdqsZPfjgRR7BkBVV91/fvQyJyKvBtt45+cYf3w1lmN/4a9wD3AIwfPz5nX2drW/4Gby8CrLm1NASInzaFriY/MiKxD6T8KORcWDGT/rKVzj2PuUyCzkSPf/v39q/NbnM6VoGTd2sWTsgwACIyCqgA5ue4/qS0tObv7mwrMQ3ET6RN6drFJEgmmA/EodxMWLHyowBTmYSBOyFxH5yw4VbgZOBg4Fc47XxbRA7CWcDqL8BkVe2ggYRFSx41EM9/Xyo+ED/5nqBZzCQePMr3+0w7nXuO2hEmBWXCCpnuwOU4Kx1GgLnAcao6D0BEzgceBQbi+GTOzFM7gfzOwYi0lbYGYmSGaSAO7VFYwY4XcY4Nw1+Uk2kgRZCNN+eoag2wd4ryx4DHwmtRagojCqsEBUg5jng5pBy/zXT7LBmckymxs8ZzcM0CC+M1fPh/HP88kIOveYVNzeElV2yPwiq94aHNNJCMMQ0klnLpez41EBMgaeA3r/g1kKVrG1i0JmvBaJ1SyhrIOTYnJGMSpnMvl1HUj5cLK8Vw6v9evLDxcFKZJG7DZl3Ttx22D9EESBr43/jj3/7DjBzKxzyQGfNrOPOBmTnXEKqXrMvp9UuZxEvalh9BnOf+78p7cot2Hkihh/EaDv7Iq/jBu0uIk9/yMQ/knIeqaY600dTaRs8eXUOr19g8ykEBSfYmn6rv/qJin7dayLmwDB8tvgE7fiJhmA+qV3WYPhCvv2E4ucvS7JIFynUmerxSHF2wKc3rhG/CKtxrBsUESBq0tiU3YWXLnPTmwjXc/krq9GL5MGF5XY+EILSaQtCsps1exX1vLM55PWGSyATTWgZBCfGh30EG0RgfCCH6QHKydkcurhkMEyBp4DcZxc8DydaDevrf3uXaF+alPMZ7YFrbNPQ3zDBSuKQSIJ/X1HHllDmb3e9zH67msv/O3qxrFBqJvpJymFeTzHEc1ITlOUGK9ZsyDaRI8AuJ+IE0zHkh/gcm7FDedATlsx9/xQuzVqZdR1Nr8pDoMx94j3tmLGL5+k1pXzcTbn9lIXNWbAylrs0l0S9TDhpI/KDZ/hYerO9hukByvJ6U+UAKGb/JKN6Bne5A3hJp4+rn57K+If11zSMxprRwQ3nTGZB+8fiH/OyR99Ouo6kleZ8a3Pk23bvm/taNtCnXvjCP425/M+d15YpIHlPuhEW8BhLMhJVoXwj+vSTbm0NMOncL4y1c/IN1vGkgXdPO85+t5K7XPuevU+em3Y58CpB8+0C8/mYr3j2VicfThIpl5cdEg0c5aCDJAjtSm7D880C8fcWJmbCKBL+WEf9gppsba1Nza8rzUr1J+AfPoIPbWwvX8PqCmjRamJgwkkimCk/2BEimucja2pQ7Xm0PUmhwf4dU7ehaJHGeib6RcvCBaNztkp4Bq92JHoYE0RxoC/kMtDMBkoSFq2t57pMVMftaU5qw0htYm90BsHu3xD9Bqgc/kiIaLBmn/e1dfnzfzDRa2Hndm8tnyzfw0pxVHfan8oF433Ommte0Oau4Zmp7kEKqFDSeJtSlSNLLJxpISjHdTTzZMmGFQS5MWLG5sLJ00YDYRMIkHHHDDACO3e3Y6D7/234HJ3qaA6snjLonGZxa25RuSebr+atqCXlRqWxmIf7OrW8AUNGtS4zZKrUJy6k/U0HW2BIrMOpTCJBi00ASDUnl6APxCPqG327CKk5hm8/1QEwDSQP/4Bk/kPrfiBtbIjz53rKUN7B3fLckzuBUAik2CitsJ3r266vs3pVtB/SK/h9kHkimb9bxy+WmMmFFNZAikR+Jbrdy84E4z5zzf3ATlnduVpuVEM2FCpLs+iFgAqQTkjms4wewZz/+Krp94/T5/PbpT5g2u6N5Jv78ZNFE8WHBr8xdzeraxgRtSu+OUVWe+XB5xmlQcjEgtUba6OEz5TW1dJ7ZOF6QNbZE+PdHy9O2KzekNGE5ZUVjwkqwryx8IHFO5HRNWNFkilluV+KKE25u3iVzcM2gmABJwBsL1kS3/fb4WCd67AA2fc5q5q505gusr28B4Ov65CG6rVEBktyE1X5sG2c++B6n3essGd+2GVFYUz9bya+e+CjGkZwOuRiQWtuUCr8ASSDc1jc0M2N+exBAvOC87oV5XPSPj3hj4Zr4U2OI/7ZTCZCoCatYBIj7lfjbm8+Fz8LCr5HHhrQmP6dYzVWJsPVAABGpEJH7RGSJiNSKyIcicoxbNlJEVETqfJ+JuWrLi7PbJ7/55yS0pkimCNDoHuu9TQeJJkqWhNH/4Nc3OYPc5zVOyvjIZpjuQo88AAAgAElEQVSw3ln0NQCra5vSOi9IfS2RNl6em1zrSkakTWM0kETf27kPV3PG/TNjzvGzyu3P2hRCOxENTY4Ja9XGRj5etj6mzBNk6xta+Lous+8rTLzBw++zKQcNxN/FiKY7jTBkE1YOBnvTQBy6AcuAQ4AtgInAkyIy0nfMlqrax/1clquG+M1KjTEaSHITFrTbyoMIkE2umSaZT8FfV21Ti3t9pwL/oJDuHIWH3l4S09YgtCaY/7KpOcK7rjDyuO7FeZz1YDUzF69Nq02tbUqPrqk1kHkra5O2CaCb26F0TXqeBnLEDa/x/bgJg/7f75R73knruvmki++pLgcfiF8jD7o0bcwhaTjRm1ojvP35150eF6TeRLV9uHQdGza1JD1/7sqNrNrYGHvNJNcPg4IRIKpar6qTVPULVW1T1f8Ci4G9wm6LX4D4NZAYE1aCgdt7WD1zTKrB3XPeJhvw/EKizn1L9t4ss5HKJJ30840xWYiVuSs3csGj73PyPe/w5bqGaNncFbVue1vSfvP1ayC1jR0foHjnd0vc9T1TYGcpZeK77f1GtY0dnel+QbZgdXgLhmWK943EaiDlFYUVdFZ2opIgg+8Vz83h1HvfiZqrs0mkTfnBHW9x5gPJw+2Pvul19rvqpZh9uZhbEpSCESDxiMgQYEdglm/3EhH5UkQeEJFBSc47T0SqRaS6piaziXM9fH6JpBpIggHSCxH1BsNUzmDvzTeZluLXTOrcwa050kZNbROzv2q/eYOE8a7Y0DFvVDpWfX/oa2tEOfqm13llnvPdeuY1aNeqKrt1Tdu05tdAvkqQ5yp+4I8fGL1ots7qjRds8b+R/wEMc72VbOA13e/0LwsNxG/Cagvm3YjNxhucua4mvL6h40uOqvJRnBm0wzExx8eWefdbZ9fokL4+yXYYFKQAEZHuwKPAQ6o6F1gD7A2MwNFI+rrlHVDVe1R1vKqOr6qqyqj+ZBpIaycaSHvYp3NLNkXaWL2xkTUJ7Ofr3Bsw2YDn1yz8b8d7XzGd616cH/1/0Zq6lCovwH5XvdxhX/wbfSr8k+3qm2Lf1P1vfE1Rs5ymL0B8GkiiRInxrY3XvLoHNGHFl8eby/z/N0fCW+c+G3hDZ7cu5eYD8Wsg6ZtxshWFNfmD5Rx3+5tM+XRF0mNitIW4GjN9YbFUJj5EpAvwCNAMXAigqnWqWq2qraq6yt3/LRHpl4s2+GeH+9++/YNiougWT9h4xzU2R5hw5UuMv3x6h2M/d00iyQZa/4Nf25R8rsKVU+by0xQqbzLSMWH5I9G8UGIP//fjaSBNrW0xA3UQtdovQL5c13mm3fjv39NAOpunEi/44wWIX1imSuoIzm+0eE19p20NjURRWOUgQNoS32uB07kHOKE10sYXa+qjJyY61DNzZnpPNGX8wpJcKOWaghIg4rwK3AcMAU5Q1WSv1t63lJP4yu5JHLoxa6IneDCjyffcc5IN/PVNrdG37GRvzH7BUpfAPu/nw6UdVd519c1sTOBL8PBMQHVNrZ1GGDX6BtLVG2OP9Q+43nFNrZFOAw7i6dyEFftTxwuKbq7ZsTHFoL+xsYWauOiz+LQpfpNlZwEKt7y0gMOue5VFNYXhH/G+Zf/LQVjLDKgqy9Y2dH5gDvA/im3qz4WVwgfiK+oSdaIn56rn53Loda+yyn2BSnTtIO9kQUxYSc9NIuD8/S93DeROYCfgu6oaHUVEZB8RGSMiXURkIHAL8KqqbshFI/w+kNgUGz5tJMGP3RQdQN3BOcnA74/iaI60sWpjI6qxi0O1xjjRU5uodhzSp8O+b1w2jX2vfCnB0Q6etnDUjTPYK05DamqNsM4XDrvJp2WsihuAV2xojGoh3nE1tU0xNuIgs9djnOhNrR0elvhns7m1LUYb8g6vTzGzfPxl07l+2vyYffEPbToayLuLnd9xZVxUTDZYU9eUthkw4TyQkDSQp97/koOueYXqL9KLwMsGfhNWJEeLrHlzw7z7OpEFIt0JjOkKkGQvYmbCAkRkBPAzYA9gpW++x+nAKGAqUAt8BjQBp+aqLTFhvC0RWiNtbNjUEjOoJhoUvbdX78HflMCJPvWzFZzzcHX0/zlfbWSfK1/i4beXxEV5+QRIJxpI74rEKc0amiNJkwWua2hh4eq6hP6Gcx6q5huXTYv+7zdTrY4bLP/3qY85+qYZNLVGor6YPz87m4v+8WH0mCAayIoNscKgodn53tfWN7NhU0uHt7sbp81nwhUvRUMavYdvU3Mk4VyQlkhbQo2igwmrJbgGkitaIm2Mv3w6f5j8aVrneW/FXfPgA/EERz6i1ZJFXqUcTBOUBZl46NWViwCLzu63ZElGU/lVck3BCBBVXaKqoqqVvrkefVT1UVV9XFW3U9XeqjpMVc9Q1fSXugtIvAnrt//8hN3//CJr6psZ0LsHkHhQ9N5YvZsr/g32ky/Xc/7fP4j+P6hPBYtce+n0OauSJmtM5QNx2uL6XFoiNLe2xZiudv7T1Oj2qRO2iW5Pm72KI254rcO1WiNtvO6+bbW1KZE2jRmQ401AAF983cCuk16Meajm+uZtxC/EFZ/QEGCg+7161DW18r9Pfcyel01j9z+/2OH4r1yB45nUvDqerF7GnpdN6zApcP6q2HkkHvG/kb9tnaVUyVUacO+F4d8ffdXJkbHkUwPx6s7HnH3/u1xQc45/oPVMfinDft2ieCtDuqQa4DsTSsnKbR5IgRHvRJ/84XIAln7dQFWfCqB9gL/7x3vFHAu+uQVxA3+8bX/4lpXR7dcXrOGBNxZH/09HA1lX78y7GDtxKjte+jy7TWofcP031E7DkscceG835/lWEGxoifD7yZ9w0T8+aq8ryQqKqW7+1ohGTU7fufV1xk6c2uGYcw8eFfN/bWNrzACazJwUiXsj9Hwgs76KjdPfkCDsEjq+1W1q9vmemmLL2pIMxv7MAN49kEhIJtqXCG/eT6ajccw8kJBSmUSdknmQIDEmLNUMorCcv0EUFu/ZzjhCL0bApReF5X/BDKxp5RgTIAnw+0DqmlqjN9jsFRsZ3M8VIO6DOWZIX175zaGAY8KKtGn0Rlhb3/627rzNx9bTq0dsvvYn318W3Y71gbSy1ZY9GeQKr3iWr9/Ezx6pTljmR0R49w+HM3Jgrw5l9U0RVJWX565ur7exlServ4w5bn0nIcOJaIm0cd4j1Uy44iXmr+po4thmQM8O+abq4oRvQ5LB1wsrjvcXxJsa4q/nEf8m6R/k431Pl0z+hDbXxt7c2ha9LzyhNWN+DWMnTuWWlxYwduLUGC1o1lcbGDtxasL1T+JJNKkxCIkG8bA1kHwQE8brj8gK6EQPQvxg35l/LBM6EyD+Ov33bT7XAzEBkgC/Cevq5+dGf5RIm0bLvLeBrl2E7Qb1pmf3rtz+yueM/sMUnv/Msa75TT9H3vhahwEpPqJp2dp2DcV/M9U1tTKkXwW7b71F0jZPn7M6aZlHW5sypF8l2w/u6HSva2zlD//6LGZfoiiuTG7QxpYIr86LndTpn7vSq3u3Dtf1C19Ibsv3Btt4k6Kq8sKslex46fPMX1WbNGliKh9IvNB5svpLRv1hCtv9fgo7Xvp8VHh5QufT5U5Mxw2uo/7Dpeui53qBE68vSJ3sMVG9QfEGuW6+XCZhzUT3BrF8LGAVb7by2hI8jNczYQU9PrG/wqs3lXlLk2yDM28sFf46Y+Zj+ftfrj6QQiJZinWAyu5OmTegeW973n4//ofp85p6fve04xR97pcHMv3ig6P+j0T8z2MfRAem2sZW+lR2p7J7khWmAuK9jQ7uV9mhrLaphcdnLo3Z960bZ2Rc17At2uvwBlY//gixnj26dnjDO+vBzjUqaB9s4x/aif+exc9cc9y8lbUJB+XK7l14ee7qmHk6fgHSmSbgOf6936lfZWwwQw/fimDetfokCHjYsKmFkZc8x9/fWeL2KX0tD3xhvHmcBxLUTOfxu39+wshLntusOuNTmeTChBVfmEhbiPo+U6youTlRWH4NxP9CZD6QAiOVAJn0vXFAuwnLc8D1SLI0bSLGDu3H9oP7JkwT3tc3wDxZ7Zi06ppa6VvRjYoEQiodvDkBVQlMYXWNrR0GwM3hmF2GRbc934+/u/6BesyQvtGHoGeaQvI3T33MxU98lDLk9RePf8ilz3zWYf/wLXoCxGQK+PN/ZjF24vPc8tKCTrUFzx/k5QqL93k1+wYST+NK5CPwUs3c/6bjA/OETXNrGyMveY7Rf5jCzdMXJG3HAVe/zO/++Ul0JElnJvqlz3zKzn/s6JNKF2/gSpUePxFPuPd4Mv9SEPznxiwuleKcRA7zdHJnJdIyovOgMjRvde4Daf9u/b5Iy8ZbYPToltgTOLqqd3Tw9ZuwINZskIqe3btGz5n264P51wX7x5SPGNTun7hnxiLAGdz7VHQLpIH88vAd2GObLROWeW+jfRMIipPveYchCTQTP+k4SH995A784BtbAbCmzrnZd9u6Y7v2Hz2QP39/XDQB5S5bpZ9cYPKHyzNambFbgrVY6psjNLa0Rc1QqfA0zInPfMbvJ3/SQWOZ9OxsfuKmoPdS8d/68sIO2Yo3boo9L15birQpN05P3J4NDS0sX7+JJ6qXRc0X8RrIn/79GSMveY7/eeyDDuf//Z2lNDRH0po7cczNr/OXZ2cD8N4Xaxl5yXPR/iUKXQ9CqkmvnRFrwvJpIAFNUkFu6w4+kETzwFojMX8T15vcR5OOBuJfa8jWAykwPA2kT0U3eruO7kF9Knj8vH0REbpI+9u8N6hWBNRA/A/YqKo+fGPb/jHlIwf2jm5/uW4Td7y6kJUbG+nWVaJ1nDR+a+44fU8eO2cfpvzyILba0nmTHl3Vm4tSCBDvTW24e/wxuwzlmh/uFi1fsLouei2PfUcNiG53DygkAfpWduf4PR0BstSdoXzkzkMAOHa3du3kO7sNp7J7V0YM7M2dp+/JfT/dm2+5xyWjRwIN8YsM0kckcuhnyuMzl3Hnq5932P/a/BpOuuvtGG3m1pcXsNPEqZx899uc+cDMDv6eVFF3Uz5dwY6XPs/fXl/ECXe+xcn3vB0t+98nPwZiNZCWSFs0hf9znzg5mm57eQEn3vVWzGATL7Qen7mUfa6cHvN2f9SNM7j1pQXMWbExqi151/zkS8dMuak5wryVtYy85DkWppgTcv2L8zj2ltej/6dafK0zNMaEFcwPkO44G+kgQDoKiaYAGkg04aUkMGF1Ng/EV/51nWkgBYsnQIZtUcmJ4525E8fuOpTBfZ039DZtT4bYNaAJa7cUDnA/2wyIjZC6Zuo8ADY2tkad7HuPHMC3dx3G/tsPYufh/Thml6FOG3cbTtcuwiXHjOWq43flm2MHx1yrl2seO3rcUK4+flduPHkPTtxra/YfPTB6TLyZ7GeHjI5uj6rqTSJ++c3to9uPn7svz/3yQAB232ZL+lZ24+W5qxGB8w4exZU/2JUrj9s1erx/RcZjdh1Gv8runLrPttF9vz16THR72BaVvPZ/hyZ80L7a0Pls8PioN2+cvfy4XQDHlJYLZn6xlv69ukf/f33BGja1RHh38VpemVfDglXtedGOuOG1qOkynvvfWMxN0+fT3NrG5c/N4f0l66LzbSq6dYnek37mrIgNZ25siXDdi/N574t1HHfHW9H9XsBHS6SN7932Br+f/CmrNjZFJ5pubGxh3qramJn8qzc2RudFeWxqjjD5Aydyb+pnyZMK3vrywphQ62QLgU39bAUHX/NKyrd6/+AeSRGFtba+mQOufplzHor1r0V9IClG3/gJuYm0hXYNpHNtOFEy03TmgfhfOvLpA8me0buE8ExMChw6pooH3/oiacZbzwfiaQdHjRvCC7PaQzX7VXZjY2MrR40bGn1Li+f5iw7i5bmraWhu5af7b8fIgb0Y1KeCs90b/fCxg/nd0WO4xHXCT9huQMz53g3rRWlVdu/KqRO25Xu7D+eRd5YgOMkGz9hvhNPmLsIpE9oH6d8fsxPfve0Ntx+xg+wIn0Dbd9TAmAmC/zx/P+avquPUCduw54j+DO5byc7D201Q/Sq7c9YB23HzSwtQdQTzaa5w2GrLnixfvymh4N2ypzPYisAJe24dFaLf22M4IwZ2FGJjh/aNaRc4v1t85JffPv/I2RMY2LuClRs3ceiOjqA9atxQXpy9kkfeXtLheuBoftv078XgfhVcOWVu0nti4nd25paXFsSU/+HbO/F///wk4fHeoOyPwgM4+8DtuM+dGzR8i0qunDInqVPcuwcG9O7B7ltvyftLnAiwVW6k36+P2JEbp89nos8X5A8zPuTaV3nk7An06tEt5j496JpXuPmUPTq82IAT6PHeF+ti9j3hE35egsurpszhmY+Ws/OwfrwyryZGA/Xwv1H7uWTyp6xvaGHxmnrGDu3HBY++z57b9uecg9rnDcXmwmo3YcV/VW8sXMPy9ZtYvn4Tb1zTfm8M7F3Bqo1NPFG9jCumzGHCyAE8fPaEqMm4rqk1aob1SOUD8QcSXDN1Lis3NnLDSXsAsUn8OkR2dWbCak1iwvJJjUn/mcWzvzgw5XWyiQmQBLQvcakcsmMVf/n+OA4b0/42f+UPduUP/3IGc0+AOP6DDZyy97bsuW1/lqxt4LF3l3LBYdvTReDMA7Zj160SayE7DesXM8nv5L23jUkZcu2JuzOgdw+uPmFXXl+whm3jHuZfH7kjIwf1jmkjOClOzvdpEMnYZat+XHrsTmzY1MJBO1Rx0t3tZhHP3NW/V3fOPGAkXUT45tjBfPF1PXuN6M/4kY4wOzSubo9TJmzDzS91dADfePIeTJ+zioN26Jhyf8tezlttj65dYrSG/UcPcs/dnTFD+vGTB2ZSU9vEpO+Ni1kx8Du7DeO6E3fnobe+YPvBfbhp+gI+Xb6BXj26RoWIV68n8H60ryNcT99nBPuPHsRh173aoV3/79s7s4WrSTz9/nJmJsn7dPaB2zFqUG/eXbyWu15zzFpH7DSEp3++Pyfc+VbCc+I5eMcqzjt4FPe9sZgJ2w3g3INGca6bAqdH1y5JzR0Tv7MTS76OTWo4bItK9t9+IDdOd3JWAVx42PZMnbUyxsx028sLOWD7jsvsTHzmM3579NiYfXtss2UH4RFPbWMLV06ZE/XlrdroDNqe2cvP+X9/n0N2rOI7uw3j+hfnoyhdRKK5p46+qd3cNeXTlVz+3ByGuHOyVvnC4VXbTUFe8MYNL87jrhmLYgZob7AfN7wfD545gW9e92r0u5j5xVqOufl1LjlmLPe9sZhfHbFDh/Y+9u5SHnu3PWpxSL+KqBZV29jKNVPncofPpDn5g+Vs2at7tD9dugg1tU3RCLSh/Sqjc8zAMX3eO2MR5x08KmYpZ4+H31rCM+4EZ79g+XT5BkZe8hxD+lVw48l7RJ+ZXGECJAH+sD4R4Yz9RsaUn7bPthw6poopn66IDihXn7Ab39h2GYeOqeKwsYO5/L+Ok3Fg7x5RM9jBOwZfn6SqbwW/+daO9OzRLWom2Lp/L071aQ4eA3r34OwDt0uzl+2ISPSNzsuo2qNrFy47bhyV3btyxQ92Yf/RgxgxsDd//O7OABy4Q7Abc9gWPRPun7DdgA6alMcWrgYydItKevVov0UPduv8wTe2BmDUoN7U1DbRr7I7t5+2J0+9v4xX59Ww1ZY9qezeNWp+O2D7QVz7wjz2HtmfAb0rOl0/ZeTAXvzpuztz+NghKMo/3lvG8C0qo781wD6jBiQVIACHjR3MYWMHs++oATS1ttG/dw/26t2DJ3+2H2vqmrjgUceh/c/z9+PpD75k+fpGZsxvfyvu0bUL/V1BusPgPkwY2f5dPXL2BF6cvYr1DS08/UHsRM8tenbn/ENG062LsK6hhfveWMwROw1hzNB281zfym6cecBIxo/sz08feC+6/8Nl6/lo2Xr2Hz2Qw3cawsLVtTw+cxn1zRGumjKHPhXd+ME3tmJw3wq+vdswDr++PRXORYfvwMdfro/R+m5/xRlAtx/chwsP256rnp8TM9gDXPb9cby7eC3//WQFr82v4c2Fa9h2YC8mjBzAC7NSZyvaaVg/+lZ259mP2zMWnP3Qe6yubaJrF2HFhkb2vGwa6xqao1rJxO/szGXus3ny+G341ZE7UNW3gkF9K2Ki6BavqY+GgXt/9xrRP6rZxVPVt4LDxgzms6828N6StQnvDU94HLfHcMYO68fVz8+Nlq3c2BhNytmmRIMv3ou7zoWHbc/IQb07JK3s2kVQiAq1w8YMZmDvxBOPs4mE7bUPk/Hjx2t1dbD5BH4Wr6nnsOteZdzwfjz3y4Myqru2sYV7ZyziF4fvkDIsuNBQVe6esYjv7j68g0M9U6bPdkx6R3TiHPe34d7XF3HUuKGMGNibe2cs4gDX3+Nn9cZGHp+5jF98c3u6dBGumjKHu2cs4sof7Bo1leWKOSs2cszNzlvxY+fsw+raJqqXrOXbuzi+qc6YuXgtX65r4Pg9HWEYaVOumTqXgX160NzaxtG7DGP7wX148M3F7Dd6EGOG9uXeGYtY29DMb48aE7WhP/HeUrbo2Z03F35Nzx5d+c23xkTNghsbW7j95YX8/NDRbNmrR/Rtd8b/Hca2A3sRaVNue3khKzduYt9RA3ltXg24vqqxQ53vesGqWu56bRGqymn7bBvVOMFxtP/eTfj4xdXHsra+mbtnfM4Og/vym6cch/6oQb156vz9GNingubWNm55aQGn77stj7+7lNP2GcHQLSqJtCk3TZ/P/FW19K7oxjkHjmLn4f04+e63eXfxWvYdNYDBfSs5bGwVi2vqOXH8Njw2cym/OmIHunXpwug/TAEcP+Poqj5Udu/Kbltvwe8nf0rfim58e9dh7DWyPyictPc2vPX5GpZ83RDzMvbZ8g28/fnXRFTZeVi/6Fv/kTsPoU9FN4b0q+RH+27LE+8tY+7KWqbNXsWA3j2iWse1P9yNE8dvw4aGFq6eOjc6p+q8g0ex36iB1NQ18VvXhDn/8mPoIs6E06VrG+giwthhfaOm2mN3HcZzSRal+uLqY1PeV1M+XUGbKt/ZbXgnd2BqROR9VR3f6YFeGvFS/Oy1116aCW1tbXrTtPm69Ov6jM438sPauia98rnZ2tjSGkp997+xSN/+fE0odWWDl+as1EffWZLVaz753lJ9fX5Nh/3//mi5/vofH+qa2saMrz13xUa9dupcjUTaUh736DtL9Hf//Fg3Nbf/7nWNLTrpP59lXP8t0+frh0vXJSxbsX6TXvHcbG1qieia2ka97NlZMXWrqtY3tehfnp2l6+qbovtmzF+d9PuPRNr0mqlz9NMv12tbW5veMn2+nvXATL3g7+/rzMVf6wNvLNJ3QrzXgGoNMMaaBmIYhmHEEFQDKR7bimEYhlFQFI0AEZEBIvIvEakXkSUiclq+22QYhlHOFFMU1u1AM8566XsAz4nIx6o6K7/NMgzDKE+KQgMRkd7ACcBEVa1T1TeA/wA/zm/LDMMwypeiECDAjkBEVf0Z5T4GxuWpPYZhGGVPsQiQPkB8HpANQIfkRSJynohUi0h1TU1NfLFhGIaRJYpFgNQB8Xm++wEdEhap6j2qOl5Vx1dVBZ/5bRiGYaRHsQiQ+UA3EfEnpdkdMAe6YRhGniiaiYQi8g+c9FTn4ERhTQH2TxWFJSI1wJIMqxwEdL6AdXFgfSlMSqUvpdIPsL54jFDVTk04xRTGewFwP7Aa+Br4eWchvEG+gGSISHWQmZjFgPWlMCmVvpRKP8D6ki5FI0BUdS1wXL7bYRiGYTgUiw/EMAzDKDBMgCTnnnw3IItYXwqTUulLqfQDrC9pUTROdMMwDKOwMA3EMAzDyAgTIIZhGEZGmACJo5jSxovIhW7aliYReTCu7HARmSsiDSLyioiM8JVViMj9IrJRRFaKyMWhNz62rRUicp/7fdeKyIcicoyvvGj64rbp7yKywm3TfBE5x1dWVH3xEJEdRKRRRP7u23ea+5vVi8gzIjLAV1Zwz5GIvOr2oc79zPOVFVVfAETkFBGZ47brcxE5yN0f3j0WZNnCcvoAjwNP4OTfOhAn59a4fLcrSVuPxwltvhN40Ld/kNvuE4FK4FrgHV/5VcDrQH9gJ2AlcHQe+9EbmASMxHmp+Q5OmpqRxdYXt03jgAp3e6zbpr2KsS++tr3otu3vvj7WAge7z8pjwD98xxfccwS8CpyT5Pcqtr4ciTNJel/3mdnK/YR6j+X9xiykjzuQNQM7+vY9Alyd77Z10u7L4wTIecBbcf3aBIx1/18OfMtXfpn/gSmED/AJTgr/ou4LMAZYAZxUrH0BTgGexBHyngC5EnjMd8xo99npW6jPUQoBUox9eQs4O8H+UO8xM2HFUipp48fhtBsAVa0HPgfGiUh/YLi/nALro4gMwfktZlGkfRGRO0SkAZiLI0CmUIR9EZF+wF+A/40riu/L57gDLYX9HF0lImtE5E0ROdTdV1R9EZGuwHigSkQWisiXInKbiPQk5HvMBEgsgdPGFzip+tHH9398Wd4Rke7Ao8BDqjqXIu2Lql7gtuMgYDLQRHH25TLgPlVdFre/s74U4nP0O2AUjqnnHuBZERlN8fVlCNAd+CHO/bUH8A3gUkK+x0yAxBI4bXyBk6ofdb7/48vyioh0wTEPNAMXuruLsi8AqhpRZ/XMrYGfU2R9EZE9gCOAGxMUd9aXgnuOVPVdVa1V1SZVfQh4E/g2xdeXTe7fW1V1haquAW4gWF8gi/eYCZBYSiVt/CycdgPRJYFHA7NUdR2OSWV33/F576OICHAfztvVCara4hYVXV8S0A23zRRXXw7FCWRYKiIrgd8AJ4jIB3TsyyigAucZKpbnSAGhyPri3itf4rQ/nnDvsXw6ggrxA/wDJ+qiN3AABRBxkaKt3XAiLa7CeXOvdPdVue0+wd33V2IjMa4GXsOJxBjr3mkG5TIAAARdSURBVFT5jly6C3gH6BO3v6j6AgzGcTr3AboCRwH1wPeLsC+9gKG+z3XAP91+jAM24phQegN/JzZyqaCeI2BL97fwnpHT3d9lTLH1xW3TX4D33PutP05k1WVh32N5+wIK9QMMAJ5xb66lwGn5blOKtk7CeQvxfya5ZUfgOHA34USfjPSdV4GTGn8jsAq4OM/9GOG2vRFHzfY+pxdhX6rcB3S926ZPgXN95UXTlyT32999/5/mPiP1wL+BAb6ygnqO3N/lPRxzzXqcl5Uji7Evbpu6A3e4fVkJ3AJUhn2PWS4swzAMIyPMB2IYhmFkhAkQwzAMIyNMgBiGYRgZYQLEMAzDyAgTIIZhGEZGmAAxDMMwMsIEiGFkGRFREflhDq8/3q1jZK7qMIwgmAAxDB8i8qA7OMd/3knjMsOAZ3PVRsMoFLrluwGGUYBMB34ct6856MmqujK7zTGMwsQ0EMPoSJOqroz7rIWoeepCEXnOXTJ0iYj8yH9yvAlLRP7oHtfkLiP6sK+sQkRuEpFV7nKr74jIgXHXO9pdorRRRF7HWaOCuGP2F5HX3DYtF5E73bU8DCNnmAAxjPT5M/AfnHUY7gEeFpHxiQ4UkRNwstheAOyAs1zvTN8h1wAnA2fhrOnwKTBVRIa552+Dk4dpmlvfre45/jp2xVly9j842VWPd4+9f/O7ahjJsVxYhuFDRB4EfoST2NHP7ar6OxFR4G+qeq7vnOnASlX9kfu/Aieq6j9F5GLgZ8Au2p6i3juvN7AOZ5nVh919XXFSiD+uqpeKyJU4CweNUfdhFZFLcTKvbqeqX7gaTYuqnu279h7Ah8AQVV2dnW/HMGIxH4hhdGQGztrSftb7tt+OK3sbODbJtZ4CLgIWi8gLwFTgP6rahLNOQ3echY0AZxEqEXkb2NndtRNOOm7/m158/XsB24vIyb594v4dDZgAMXKCCRDD6EiDqi7MxoVUdZmIjAEOx0mzfT3wJxHZh/ZBPpEZwNsnCcri6QL8jcQrBy5Pr8WGERzzgRhG+uyb4P85yQ5W1UZVfU5Vfw3sjbOA0QHAQpzorqjT3DVh7QfMdnfNBvZxV2xMVv8HOAscLUzw2YRh5AjTQAyjIxUiMjRuX0RVa9zt40XkPZzFen6Io13sk+hCIvJTnOfsXZxFsk4GWoAFqlovIncCV4vIGmAx8GucZX3vcC9xF/C/wE0icgewK3B+XDV/Bd4RkbuAu3EWTRoLfFdVf5Z+9w0jGCZADKMjR+As9elnObC1uz0JZ8nQW4Aa4ExVfS/JtdYDv8NZDrY7jkZxvKoudst/5/59AGfZ1Q9xlhhdAaCqS0XkeOAGHGf8+8AlOMuu4h7ziYgcDFyOsxpiV2AR8K90O24Y6WBRWIaRBv4Iq3y3xTDyjflADMMwjIwwAWIYhmFkhJmwDMMwjIwwDcQwDMPICBMghmEYRkaYADEMwzAywgSIYRiGkREmQAzDMIyMMAFiGIZhZMT/ByZOg53Tm43RAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\")\n",
    "plt.ylabel(\"Sum of rewards\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like a pretty robust agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using TF-Agents to Beat Breakout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use TF-Agents to create an agent that will learn to play Breakout. We will use the Deep Q-Learning algorithm, so you can easily compare the components with the previous implementation, but TF-Agents implements many other (and more sophisticated) algorithms!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-Agents Environments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.TimeLimit at 0x10b3788d0>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.environments import suite_gym\n",
    "\n",
    "env = suite_gym.load(\"Breakout-v4\")\n",
    "env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x10a9d8518>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]]], dtype=float32))"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.seed(42)\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: since TF Agents 0.4.0, there seems to be an issue with passing an integer to the `env.step()` method (it raises an `AttributeError`). You need to wrap it in a NumPy array, as done below. Please see [TF Agents Issue #520](https://github.com/tensorflow/agents/issues/520) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]]], dtype=float32))"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.step(np.array(1)) # Fire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure breakout_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAIcCAYAAACn9OoDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAACVxJREFUeJzt3T1uG1cUgFFOwNrIClK5yBIEL0Bgkc2YK8gKuIwgC0hBuHBpaDFGiiBI4cIvlStRiX84fJ/G55QDQXNBFR8uZvi0jDF2AFD1w+wBAOC/CBUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAafvZA1yyLItznQC+M2OM5dJ1GxUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZCWPOuPdZxOp5vf83g8/u/PVOfa7W4/2+fOVVX+Wz53n/PZbvWzsFEBkCZUAKQJFQBpQgVAmpcpeMQLEF+mOlfZVh/6sw4bFQBpQgVAmlABkCZUAKR5mQK4uWu+WOLFjO2zUQGQJlQApAkVAGmeUQFXc+3nRb4kzW5nowIgTqgASBMqANKECoA0L1PwSPUBtrn6fBaswUYFQJpQAZAmVACkCRUAacsYY/YMjyzL0hsKgFWNMZZL121UAKQJFQBpQgVAmlABkJY8mcK32wH4xEYFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZC2nz3A2o7H4+wRADbrdDqtfg8bFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkLafPcDaHg6H2SMAbNa7G9zDRgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkCZUAKQJFQBpQgVAmlABkLb5f/Px8eVfs0cA4BvYqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABI2/zp6X+++Gf2CAB8AxsVAGlCBUCaUAGQJlQApAkVAGlCBUCaUAGQJlQApAkVAGlCBUCaUAGQJlQApAkVAGnbPz395w+zRwDYrvfr38JGBUCaUAGQJlQApAkVAGlCBUCaUAGQJlQApAkVAGlCBUCaUAGQJlQApAkVAGlCBUCaUAGQtvl/8/Hbx59mjwCwWfc3uIeNCoA0oQIgTagASBMqANKECoA0oQIgTagASBMqANKECoA0oQIgTagASBMqANKECoC0zZ+e/uH3X2ePALBd9+9Wv4WNCoA0oQIgTagASBMqANKECoA0oQIgTagASBMqANKECoA0oQIgTagASBMqANKECoC0zZ+e/vZ8N3sEgM365f60+j1sVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACkCRUAaUIFQJpQAZAmVACk7WcPAN+Lh8Ph0bW783nCJPC82KgASBMqANKECoA0oQIgTagASBMqANKECoA0oQIgTagASHMyBdyIUyjg69ioAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEgTKgDShAqANKECIE2oAEjbzx7gkj9+/Hv2CMAXeDgcZo9w0d35PHuEzXv15s31ftnr1xcv26gASBMqANKECoA0oQIgTagASBMqANKECoA0oQIgLfmFX+B58cVa1mSjAiBNqABIEyoA0oQKgDQvUwDw1a75Is144rqNCoA0oQIgbRnjqWVrnmVZekMBsKoxxnLpuo0KgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0pYxxuwZAOBJNioA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgDShAiBNqABIEyoA0oQKgLR/AcFBaFysro1wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "img = env.render(mode=\"rgb_array\")\n",
    "\n",
    "plt.figure(figsize=(6, 8))\n",
    "plt.imshow(img)\n",
    "plt.axis(\"off\")\n",
    "save_fig(\"breakout_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]]], dtype=float32))"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.current_time_step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Specifications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('float32'), name=None, minimum=[[[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " ...\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]], maximum=[[[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " ...\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_spec()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BoundedArraySpec(shape=(), dtype=dtype('int64'), name=None, minimum=0, maximum=3)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_spec()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=ArraySpec(shape=(), dtype=dtype('int32'), name='step_type'), reward=ArraySpec(shape=(), dtype=dtype('float32'), name='reward'), discount=BoundedArraySpec(shape=(), dtype=dtype('float32'), name='discount', minimum=0.0, maximum=1.0), observation=BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('float32'), name=None, minimum=[[[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " ...\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]\n",
       "\n",
       " [[0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  ...\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]\n",
       "  [0. 0. 0.]]], maximum=[[[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " ...\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]\n",
       "\n",
       " [[255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  ...\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]\n",
       "  [255. 255. 255.]]]))"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.time_step_spec()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Wrappers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can wrap a TF-Agents environments in a TF-Agents wrapper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.ActionRepeat at 0x134eb87b8>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.environments.wrappers import ActionRepeat\n",
    "\n",
    "repeating_env = ActionRepeat(env, times=4)\n",
    "repeating_env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x10a9d8518>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repeating_env.unwrapped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the list of available wrappers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ActionClipWrapper           Wraps an environment and clips actions to spec before applying.\n",
      "ActionDiscretizeWrapper     Wraps an environment with continuous actions and discretizes them.\n",
      "ActionOffsetWrapper         Offsets actions to be zero-based.\n",
      "ActionRepeat                Repeates actions over n-steps while acummulating the received reward.\n",
      "FlattenObservationsWrapper  Wraps an environment and flattens nested multi-dimensional observations.\n",
      "GoalReplayEnvWrapper        Adds a goal to the observation, used for HER (Hindsight Experience Replay).\n",
      "PyEnvironmentBaseWrapper    PyEnvironment wrapper forwards calls to the given environment.\n",
      "RunStats                    Wrapper that accumulates run statistics as the environment iterates.\n",
      "TimeLimit                   End episodes after specified number of steps.\n"
     ]
    }
   ],
   "source": [
    "import tf_agents.environments.wrappers\n",
    "\n",
    "for name in dir(tf_agents.environments.wrappers):\n",
    "    obj = getattr(tf_agents.environments.wrappers, name)\n",
    "    if hasattr(obj, \"__base__\") and issubclass(obj, tf_agents.environments.wrappers.PyEnvironmentBaseWrapper):\n",
    "        print(\"{:27s} {}\".format(name, obj.__doc__.split(\"\\n\")[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `suite_gym.load()` function can create an env and wrap it for you, both with TF-Agents environment wrappers and Gym environment wrappers (the latter are applied first)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from gym.wrappers import TimeLimit\n",
    "\n",
    "limited_repeating_env = suite_gym.load(\n",
    "    \"Breakout-v4\",\n",
    "    gym_env_wrappers=[partial(TimeLimit, max_episode_steps=10000)],\n",
    "    env_wrappers=[partial(ActionRepeat, times=4)],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.ActionRepeat at 0x135fa0400>"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "limited_repeating_env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an Atari Breakout environment, and wrap it to apply the default Atari preprocessing steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x135fa0588>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "limited_repeating_env.unwrapped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.environments import suite_atari\n",
    "from tf_agents.environments.atari_preprocessing import AtariPreprocessing\n",
    "from tf_agents.environments.atari_wrappers import FrameStack4\n",
    "\n",
    "max_episode_steps = 27000 # <=> 108k ALE frames since 1 step = 4 frames\n",
    "environment_name = \"BreakoutNoFrameskip-v4\"\n",
    "\n",
    "env = suite_atari.load(\n",
    "    environment_name,\n",
    "    max_episode_steps=max_episode_steps,\n",
    "    gym_env_wrappers=[AtariPreprocessing, FrameStack4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.atari_wrappers.AtariTimeLimit at 0x135fc08d0>"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play a few steps just to see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "env.reset()\n",
    "time_step = env.step(np.array(1)) # FIRE\n",
    "for _ in range(4):\n",
    "    time_step = env.step(np.array(3)) # LEFT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_observation(obs):\n",
    "    # Since there are only 3 color channels, you cannot display 4 frames\n",
    "    # with one primary color per frame. So this code computes the delta between\n",
    "    # the current frame and the mean of the other frames, and it adds this delta\n",
    "    # to the red and blue channels to get a pink color for the current frame.\n",
    "    obs = obs.astype(np.float32)\n",
    "    img = obs[..., :3]\n",
    "    current_frame_delta = np.maximum(obs[..., 3] - obs[..., :3].mean(axis=-1), 0.)\n",
    "    img[..., 0] += current_frame_delta\n",
    "    img[..., 2] += current_frame_delta\n",
    "    img = np.clip(img / 150, 0, 1)\n",
    "    plt.imshow(img)\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure preprocessed_breakout_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAGoCAYAAADiuSpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADJ9JREFUeJzt3U+InOUBx/F3Jlmwi+AW/6QExMtiKggGeuplCaUiueToXQQFoaBIiUJ7KgQ8CEK9tFBE8KCeRZHmENJCT5LsQU08KVpipVFZZDAsO28vPWTmeZcdJ+87729mPp/b+/Ds+z7ZnZ1vXt533h3UdV0BQJJh3wsAgGniBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGO93HQwWDgk78AVHVdD5rGnTkBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECIE4vH8KlPSdOnCjGPvvssyO/7sqVK0fOefTRR4uxjY2Nie2dnZ1izu7ubjH29ttvT2yfPXu2mPPFF18UY99+++3EdtO/9+TJkxPbb7zxRjHn+eefL8aeeOKJie133323mDMajYqx69evT2xPf0+qqvl7N+2ee+45ck6il156qRg7f/78xPb0z62qqurLL7+c63jT3++qqqpnn312rn0tg9dee60Ye+qppya2L1y4UMx55ZVXOltTH5w5ARBHnACII04AxBEnAOK4IWJNnTlz5sg5n376aTE2ffNBm1599dVi7M0335zYbroY//LLL3e2pqaL8dPfu3lvSlll165dK8aabjiZxc2bN+90OSwhZ04AxBEnAOKIEwBxXHMCfpIPPvigGLtx48Zc+5r+sPIzzzxTzGn6wPh777031/FYHs6cAIgjTgDEEScA4ogTAHHcELGmLl26dOSc+++/v/uF3ObFF18sxqafxtz0gdcunTp1qhib/t41PZV8lZ0+fboYm/45zWpZn8xO95w5ARBHnACII04AxBEnAOIM6rpe+EFPnz69+IMCEOfq1auDpnFnTgDEEScA4ogTAHF6uea0t7e30IOOx+OJ7eGwbPL0nMPmtXH8Wffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05CWva2tpyzQmA5SBOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQJzjfS/gTly/fr0YG41GPawEYHVtbm4WY6dOner0mM6cAIgjTgDEEScA4ogTAHEGdV0v/KB7e3utHHRnZ6cY293dbWPXAPzfY489Voxdvnx5Yns8HhdzhsPhkXO2trYGTcd05gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBECc430v4E489NBDxdhoNCrGxuPxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jU9+OCDxVjX1usVBMBSECcA4ogTAHGW+prT+fPnizHXnPrX97r7Pv46cs1pta85bW5uFmNdW69XEABLQZwAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgzlJ/CPe+++4rxvb393tYCcDq2tjYWPgxnTkBEEecAIgjTgDEEScA4iz1DRFNF+m6fBLwvDyV3FPJV52nkq/2U8mPHTtWjHVtvV5BACwFcQIgjjgBEEecAIiz1DdENF24Ozg4mGnePHPmNe++l/Wict/r7vv462jdv+ez/vvbei9q8z2t7/fHQ4+58CMCwBHECYA44gRAnJW75tT0YbG6rie2B4PBkXMOmzePeffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05s67JNScAqMQJgEDiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGW+gkRDzzwQDHW9IQIf6Z9sfped9/HX0f+TPtq/5n2pr/2cOvWrWKsTev1CgJgKYgTAHHECYA44gRAHHECII44ARBHnACII04AxFnqD+HevHmzGGv6sBgA82t6uMHdd9/d6TGdOQEQR5wAiCNOAMRZ6mtOe3t7xdj+/n4xVtf1xPZgMDhyzmHz5jHvvrtcU5f6Xnffx19Hs/yOrbJZX3NtvRe1NWfWNW1sbBRjrjkBsHbECYA44gRAHHECIM5S3xAxGo2KsR9//LGHlQCsrrvuumvhx3TmBEAccQIgjjgBEEecAIiz1DdEfPLJJ8VY05PKx+PxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jXde++9xdj29nYx1qb1egUBsBTECYA44gRAHHECII44ARBHnACII04AxBEnAOIs9Ydw33rrrWLs448/Lsb8mfbF6nvdfR9/Hfkz7av9Z9ofeeSRYuzcuXPFWJucOQEQR5wAiCNOAMQRJwDiLPUNEV9//XUx9tVXX/WwEoDV1fRU8q45cwIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACIc7zvBUCy3zaM/Xpq+0+LWAisGWdOAMQRJwDiiBMAcVxzYn1tTm3/vJzym3+XY8c6WQxwO2dOAMQRJwDiiBMAccQJgDhuiGB9/WFqe6Oc8uffl2N7nSwGuJ0zJwDiiBMAccQJgDiuObGCdhrGflYO/fDh5PZH5ZQbrawH+KmcOQEQR5wAiCNOAMQRJwDiuCGCFfS7hrGGux0ufFiOARGcOQEQR5wAiCNOAMQRJwDiuCGCaINqMLH9evV6MWe32p3Y/mv1l4Y9/bPNZQEdc+YEQBxxAiCOOAEQxzUnog2n/v/0cPVwMeed6p2pkcsdrghYBGdOAMQRJwDiiBMAccQJgDhuiCDaQXUwsf149XhPKwEWyZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARDneB8HHY1GrexnPB63sp/DDKrBxPbZ6uxc+6mr+sh9t/l1s3i/utYw+stW9l01rLtqad2z+Xs5dGa/HNvsfiVdOHFjcvtXV/pZx+2mf+Kz/rT/M7X9UQtroX37++XvzzfffNPKvre2thrHnTkBEEecAIgjTgDEEScA4vRyQ8T333/fyn4ODg5a2c9hhlPtfqF6Ya79jKvyxo3pfbf5dbN4v/pbw+jTrey7alj3Yv8f9I9y6OmGGyJ+0f1KurB9eXL7hYAbIqZ/4rP+tP81te2GiEy3bt0qxj7//POJ7aYb1IbD4ZFztre3G4/pzAmAOOIEQBxxAiCOOAEQp5cbIuq64ckHgxmemNDwdV06qCZvuHiuem6u/STeEFFV/20Ya+tydN83RPxQDv2xYdpG5wvpxNW9ye35XpXtmveGiIafFFRV5cwJgEDiBEAccQIgzmDR13GqqqqefPLJVg568eLFYuy7775rY9cALEBd1403HDhzAiCOOAEQR5wAiCNOAMTp5YaIwWCw+IMCEMcNEQAsDXECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4gzquu57DQAwwZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA4/wPd8aiZkkg/HgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 6))\n",
    "plot_observation(time_step.observation)\n",
    "save_fig(\"preprocessed_breakout_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the Python environment to a TF environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.environments.tf_py_environment import TFPyEnvironment\n",
    "\n",
    "tf_env = TFPyEnvironment(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a small class to normalize the observations. Images are stored using bytes from 0 to 255 to use less RAM, but we want to pass floats from 0.0 to 1.0 to the neural network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Q-Network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.networks.q_network import QNetwork\n",
    "\n",
    "preprocessing_layer = keras.layers.Lambda(\n",
    "                          lambda obs: tf.cast(obs, np.float32) / 255.)\n",
    "conv_layer_params=[(32, (8, 8), 4), (64, (4, 4), 2), (64, (3, 3), 1)]\n",
    "fc_layer_params=[512]\n",
    "\n",
    "q_net = QNetwork(\n",
    "    tf_env.observation_spec(),\n",
    "    tf_env.action_spec(),\n",
    "    preprocessing_layers=preprocessing_layer,\n",
    "    conv_layer_params=conv_layer_params,\n",
    "    fc_layer_params=fc_layer_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the DQN Agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.agents.dqn.dqn_agent import DqnAgent\n",
    "\n",
    "# see TF-agents issue #113\n",
    "#optimizer = keras.optimizers.RMSprop(lr=2.5e-4, rho=0.95, momentum=0.0,\n",
    "#                                     epsilon=0.00001, centered=True)\n",
    "\n",
    "train_step = tf.Variable(0)\n",
    "update_period = 4 # run a training step every 4 collect steps\n",
    "optimizer = tf.compat.v1.train.RMSPropOptimizer(learning_rate=2.5e-4, decay=0.95, momentum=0.0,\n",
    "                                     epsilon=0.00001, centered=True)\n",
    "epsilon_fn = keras.optimizers.schedules.PolynomialDecay(\n",
    "    initial_learning_rate=1.0, # initial ε\n",
    "    decay_steps=250000 // update_period, # <=> 1,000,000 ALE frames\n",
    "    end_learning_rate=0.01) # final ε\n",
    "agent = DqnAgent(tf_env.time_step_spec(),\n",
    "                 tf_env.action_spec(),\n",
    "                 q_network=q_net,\n",
    "                 optimizer=optimizer,\n",
    "                 target_update_period=2000, # <=> 32,000 ALE frames\n",
    "                 td_errors_loss_fn=keras.losses.Huber(reduction=\"none\"),\n",
    "                 gamma=0.99, # discount factor\n",
    "                 train_step_counter=train_step,\n",
    "                 epsilon_greedy=lambda: epsilon_fn(train_step))\n",
    "agent.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the replay buffer (this may use a lot of RAM, so please reduce the buffer size if you get an out-of-memory error):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.replay_buffers import tf_uniform_replay_buffer\n",
    "\n",
    "replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(\n",
    "    data_spec=agent.collect_data_spec,\n",
    "    batch_size=tf_env.batch_size,\n",
    "    max_length=1000000)\n",
    "\n",
    "replay_buffer_observer = replay_buffer.add_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a simple custom observer that counts and displays the number of times it is called (except when it is passed a trajectory that represents the boundary between two episodes, as this does not count as a step):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ShowProgress:\n",
    "    def __init__(self, total):\n",
    "        self.counter = 0\n",
    "        self.total = total\n",
    "    def __call__(self, trajectory):\n",
    "        if not trajectory.is_boundary():\n",
    "            self.counter += 1\n",
    "        if self.counter % 100 == 0:\n",
    "            print(\"\\r{}/{}\".format(self.counter, self.total), end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add some training metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.metrics import tf_metrics\n",
    "\n",
    "train_metrics = [\n",
    "    tf_metrics.NumberOfEpisodes(),\n",
    "    tf_metrics.EnvironmentSteps(),\n",
    "    tf_metrics.AverageReturnMetric(),\n",
    "    tf_metrics.AverageEpisodeLengthMetric(),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=469, shape=(), dtype=int64, numpy=0>"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_metrics[0].result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "I0528 08:47:44.704986 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 0\n",
      "\t\t EnvironmentSteps = 0\n",
      "\t\t AverageReturn = 0.0\n",
      "\t\t AverageEpisodeLength = 0.0\n"
     ]
    }
   ],
   "source": [
    "from tf_agents.eval.metric_utils import log_metrics\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "log_metrics(train_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the collect driver:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.drivers.dynamic_step_driver import DynamicStepDriver\n",
    "\n",
    "collect_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    agent.collect_policy,\n",
    "    observers=[replay_buffer_observer] + train_metrics,\n",
    "    num_steps=update_period) # collect 4 steps for each training iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the initial experiences, before training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0528 08:47:44.747640 140735810999168 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64\n",
      "W0528 08:47:44.761187 140735810999168 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64\n",
      "W0528 08:47:44.765793 140735810999168 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64\n",
      "W0528 08:47:44.770788 140735810999168 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64\n",
      "W0528 08:47:44.775924 140735810999168 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20000/20000"
     ]
    }
   ],
   "source": [
    "from tf_agents.policies.random_tf_policy import RandomTFPolicy\n",
    "\n",
    "initial_collect_policy = RandomTFPolicy(tf_env.time_step_spec(),\n",
    "                                        tf_env.action_spec())\n",
    "init_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    initial_collect_policy,\n",
    "    observers=[replay_buffer.add_batch, ShowProgress(20000)],\n",
    "    num_steps=20000) # <=> 80,000 ALE frames\n",
    "final_time_step, final_policy_state = init_driver.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's sample 2 sub-episodes, with 3 time steps each and display them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(888) # chosen to show an example of trajectory at the end of an episode\n",
    "\n",
    "trajectories, buffer_info = replay_buffer.get_next(\n",
    "    sample_batch_size=2, num_steps=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('step_type',\n",
       " 'observation',\n",
       " 'action',\n",
       " 'policy_info',\n",
       " 'next_step_type',\n",
       " 'reward',\n",
       " 'discount')"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories._fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 3, 84, 84, 4])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories.observation.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 2, 84, 84, 4])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.trajectories.trajectory import to_transition\n",
    "\n",
    "time_steps, action_steps, next_time_steps = to_transition(trajectories)\n",
    "time_steps.observation.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1],\n",
       "       [1, 1, 1]], dtype=int32)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories.step_type.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sub_episodes_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHgCAYAAAC4vJuwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGSpJREFUeJzt3V+IZGd+HuC3eqaDdiJ7xtZIyopVtJjBY4GCxiHYmJBZEbII2URXRuhuEQu7oEDQri5GwjeODQJdCIslkL8gC5REUSBXIpPFcyEECYG1ohl2Je3ITjyyZi1Zzkja9m7vjNrTlYuuqvm1+s9Ud51Tp6r6eW7qzHdaX33VPa/m7a9OVfX6/X4AAIANS10vAAAAZomCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAADF4S7utNfr+fg+GFO/3+91vQaZhfF1nVl5hfHtlFc7yAAAUCjIAABQKMgAAFAoyAAAUHTyIr15c+eddyZJ3n333S3n3nzzzS1j9913X5JkeXl5NHb69OkkyYULF0ZjL7/8cpLkoYceSpK89957o3Mff/zxpvtOkrvuuitJ8sILL4zGnnjiiSTJgw8+mCR55ZVXRudWV1eTJBcvXhyNDdc0XGN19OjRLWNt+9rXvjY6/s53vpMkOXv2bJLk0Ucfnfp6tvP888+Pjh977LEkyTPPPDMae/bZZ6e+JnYns+2RWZomr+2R1/2zgwwAAIUd5Ak98MADW8beeeedJDd+Gx3Xc889Nzp+8cUXkyRPPfXUaOzpp5/e03zD32rrGnf7TR0OApmF+SGvdMUOMgAAFAoyAAAUCjIAABQKMgAAFF6kN6HXXntty9jtt9++r7mefPLJ0fHwrU7qW9Ds1cmTJ5NsXmN9Wxw4iGQW5oe80hU7yAAAUCjIAABQ9Pr9/tTv9NSpU9O/U5hT58+f73W9BpmF8XWdWXmF8e2UVzvIAABQdLKDvLKysu87XV9fT5IsLS1t+nMd2+sc455rwn7W28YcXWv7+zzJOsb5O7bXsd3mv9nYsWPHOt9BltlMdB8y2+46ZHYzec1E9yGv7a5jXvI6nz95AABoiYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAADF4a4XsJuLFy8mSVZXVzteCTTnyJEjSZKTJ092vJLmySyLaFEzK68soqbyagcZAAAKBRkAAIpev9+f+p2urKyMdaenT59Okly4cKHV9cA03X///UmS119/fTS2vr6eJFlaWtoyduzYsd4Ul7ctmeUgm7fMyisHWVN5tYMMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAIWCDAAAhYIMAACFggwAAMXhrhewm3vuuSdJsrq6OhpbX19PkiwtLW36cx27mc/PMe65JuxnvW3M0bW2v8+TrGOcv2N7Havz33333c09gBkjs+3N0TWZXTzy2t4cXZPXyc3nTx4AAFqiIAMAQDHTl1icOXMmiad/2pijawf56Z8jR4409wBmjMy2N0fXZHbxyGt7c3RNXic3nz95AABoiYIMAACFggwAAIWCDAAAxUy/SO/48eNJkrW1tY5XAs1ZXl7uegmtkVkW0aJmVl5ZRE3l1Q4yAAAUM72DPPwtYK9vD3Iz3oKmewf5LWgOHTrU3AOYMTLb3hxdk9nFI6/tzdE1eZ3cfP7kAQCgJQoyAAAUM32JxXDL/Pr161vGdvrzXubd67kmNDF/10+ZNGFWHsN26xjn71jTY4tCZtubo2uz8hhktjny2t4cXZuVxzDPeZ2N7yAAAMyIudhBrhdc9/v9JEmv19v05zp2M5+fY9xzTdjPetuYo2ttf58nWcc4f8f2Olbnn5Xf7Nsgs+3N0TWZXTzy2t4cXZPXyS1m6gEAYJ8UZAAAKBRkAAAoFGQAAChm+kV6d9xxR5LNLyDwKT8+5aftdbT9KT/Dt1S6du1aQ49idshse3N0TWYXL7Py2t4cXZPXyfM6nz95AABoiYIMAACFggwAAIWCDAAAxUy/SO/KlStJNn9OPMy74Qtibr311o5X0jyZZREtambllUXUVF7tIAMAQDHTO8grKytJkrW1tdGYz4n3OfFtr6Ptz4lfXl5Osni7UYnMtjlH12R28TIrr+3N0TV5tYMMAACNUpABAKCY6UssVldXkyRXr17teCXQnFtuuaXrJbRGZllEi5pZeWURNZVXO8gAAFDM9A7y22+/neTGW9EkPie+qTm6dpA/J/62225Lkpw4caKhRzE7ZLa9Oboms4uXWXltb46uyevkeZ3PnzwAALREQQYAgEJBBgCAQkEGAIBipl+k99JLLyVJ3nrrrdGYT/nxKT9tr6PtT/m59957kyQPP/xwQ49idshse3N0TWYXL7Py2t4cXZPXyfNqBxkAAIqZ3kH+8MMPkySXL1/ueCXQnOFb0CwimWURLWpm5ZVF1FRe7SADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEBxuOsFsL3e4PZflrELg9t/O+W1AAAcJHaQAQCgsIM8S+67cbj0k43bX750Y+w/T3UxwM2UyGYQ2VzqYB0ANMsOMgAAFAoyAAAULrGYBXcMbv/gxtD1f71x+9VL014McDPbRDaDyLrEAmAB2EEGAIDCDvLUHS3H/3zj5mf/beP2v75x49S5qS0I2I3IwtzYJq4ZxDVvBMZnBxkAAAoFGQAACpdYTMmX8qUkyeXcWkZ/fePmrwdPAP2r6a4J2M1GZvPF/3dj6NevJhFZmDk7x3V0iQXshR1kAAAo7CC36P7cPzr+/fx+kuTp/O5o7K08Pjj68ymuCtjZ/eV4I7P5v79zY+jx72/ciix0T1xpkR1kAAAoFGQAAChcYjElbwzegfEv8mdl9JNuFgPkaHnH1EfzaJLk3+R/la8YvGvqZ5dvDHmuFjpS3+H40cHtyzeG3vhxEnGlOXaQAQCgsIPcogu5sO0x0L2v5Cuj49/ObydJXs2ro7Ef5V9MfU3ATr5Sjjfymgs38poLP57qalh8dpABAKCwgwwcSOdzfnT87Xw7SfKj/Kir5QDFb+Q3kiS/l99Lknw1Xy9nvz24lVfaYwcZAAAKBRkAAAqXWAAH0p97EyiYWSdzMknyp/nTJMlyPhidW5NdpsAOMgAAFHaQAYCZ8of5w023MG12kAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKA43MWdrq6ujvV16+vr7S7kjsHtPyhj/cFt70gZfCBJcqiMPLjbvO8Pbr+/9VR/dAdJL73B0cb343j5D76ULydJzufOMefYm7WsjY7/KH+0rznYn7W1je/9Rx99NNbXHzt2rM3ljGVWMrvHyI5tl8imxC2fi2zy2n3l5N/d5R62m2Rnv5q/v2VsKddHx3fmwk3nqD7Kxt+1P84f7+m/Y8O8ZXZW8rqbCeK6q+nG9ecGt/9o66m1cjz4J3a7XG/nlvw4SfIL+T9jff12zuZsks194aBoKq92kAEAoFCQAQCg6OQSi08//XSsr7t+/frNv2gSXx7cfquMDZ9xWvqFMrjxBctbRnbw3we32zxfu54bT2ktjX4/+TBJcirPjc4dz28lSc5t89TN9nPszU/z09GxSyym69q1a0mSS5cujcaGT3UuLS1tGTtx4sT0FreDWcnslwe3Y0Z2bLtENqnPQm+ObPLaPy4n/+ku97DdJDv7rfyTLWP1f9b/MOduOkf1vXwviUss9mveMjsred3NBHHd1XTjOrzoa5tH8NNyPPgndrtcb+cXB7d/b485r76b7yZJrqe7n3FXmsqrHWQAACg62UHu9zcuGu/1tl79Pjw3FT8Y3D5exkbbUfXi7o0vuLZlZAc/3vnU9ru/GxeUH8lfjs79fP5dkuTD/Kcx59ibg/hbJfs3K5ndY2THtktkt99NGr0A5+Vy8uweJ9nZv88Xt4wtlTn+Q/l/xTh+kp/s6euZb7OS191MENddTTeuw5f3bvMItvkndrtcb+dwriZJvpBPxvr67fg3fnJ2kAEAoOh18dvkI488Mtadnju3cf3NJ5/s/7comHf9fn9/7+XXIJmF8XWdWXmF8e2UVzvIAABQKMgAAFB0colFr9ebjVcJwBzo+unaRGZhL7rOrLzC+FxiAQAAY1CQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCg6PX7PrIdAACG7CADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEChIAMAQKEgAwBAoSADAEBxuIs77fV6/S7uF+ZRv9/vdb0GmYXxdZ1ZeYXx7ZRXO8gAAFAoyAAAUCjIAABQKMgAAFB08iK9eXPnnXcmSd59990t5958880tY/fdd1+SZHl5eTR2+vTpJMmFCxdGYy+//HKS5KGHHkqSvPfee6NzH3/88ab7TpK77rorSfLCCy+Mxp544okkyYMPPpgkeeWVV0bnVldXkyQXL14cjQ3XNFxjdfTo0S1jTXrqqadGx2fOnEly43Emyfvvv7/jfzt8DN/85jdbWt3Onn/++dHxY489liR55plnRmPPPvvs1NfE7mS2GTLLNMhrM+S1WXaQAQCgsIM8oQceeGDL2DvvvJPkxm+j43ruuedGxy+++GKSzb8RPv3003uab/gbYV3jbr+pd+GHP/zh6Lj+Zv55V65cmcZyOABkdjIyyzTJ62Tkdf/sIAMAQKEgAwBA4RILpubs2bOj4w8++GDHrxu+uOEb3/jGaGz4Qo1XX321pdUBnyezMD/ktVl2kAEAoLCDPKHXXntty9jtt9++r7mefPLJ0fHwrU7qW9Ds1cmTJ5NsXmN9W5xpO3Xq1Oh4+Pi20/Zb4XCwyez4ZJauyev45LVZdpABAKBQkAEAoOj1+/2p3+mpU6emf6cwp86fP9/reg0yC+PrOrPyCuPbKa92kAEAoOhkB3llZWXfd7q+vp4kWVpa2vTnOrbXOcY914T9rLeNObrW9vd5knWM83dsr2O7zX+zsWPHjnW+gyyzmeg+ZLbddcjsZvKaie5DXttdx7zkdT5/8gAA0BIFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAAisNdL2A3Fy9eTJKsrq52vBJozpEjR5IkJ0+e7HglzZNZFtGiZlZeWURN5dUOMgAAFAoyAAAUvX6/P/U7XVlZGetOT58+nSS5cOFCq+uBabr//vuTJK+//vpobH19PUmytLS0ZezYsWO9KS5vWzLLQTZvmZVXDrKm8moHGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAAAKBRkAAAoFGQAACgUZAACKw10vYDf33HNPkmR1dXU0tr6+niRZWlra9Oc6djOfn2Pcc03Yz3rbmKNrbX+fJ1nHOH/H9jpW57/77rubewAzRmbbm6NrMrt45LW9Obomr5Obz588AAC0REEGAIBipi+xOHPmTBJP/7QxR9cO8tM/R44cae4BzBiZbW+Orsns4pHX9ubomrxObj5/8gAA0BIFGQAACgUZAAAKBRkAAIqZfpHe8ePHkyRra2sdrwSas7y83PUSWiOzLKJFzay8soiayqsdZAAAKGZ6B3n4W8Be3x7kZrwFTfcO8lvQHDp0qLkHMGNktr05uiazi0de25uja/I6ufn8yQMAQEsUZAAAKGb6Eovhlvn169e3jO30573Mu9dzTWhi/q6fMmnCrDyG7dYxzt+xpscWhcy2N0fXZuUxyGxz5LW9Obo2K49hnvM6G99BAACYEXOxg1wvuO73+0mSXq+36c917GY+P8e455qwn/W2MUfX2v4+T7KOcf6O7XWszj8rv9m3QWbbm6NrMrt45LW9Obomr5NbzNQDAMA+KcgAAFAoyAAAUCjIAABQzPSL9O64444km19A4FN+fMpP2+to+1N+hm+pdO3atYYexeyQ2fbm6JrMLl5m5bW9Obomr5PndT5/8gAA0BIFGQAACgUZAAAKBRkAAIqZfpHelStXkmz+nHiYd8MXxNx6660dr6R5MssiWtTMyiuLqKm82kEGAIBipneQV1ZWkiRra2ujMZ8T73Pi215H258Tv7y8nGTxdqMSmW1zjq7J7OJlVl7bm6Nr8moHGQAAGqUgAwBAMdOXWKyuriZJrl692vFKoDm33HJL10tojcyyiBY1s/LKImoqr3aQAQCgmOkd5LfffjvJjbeiSXxOfFNzdO0gf078bbfdliQ5ceJEQ49idshse3N0TWYXL7Py2t4cXZPXyfM6nz95AABoiYIMAACFggwAAIWCDAAAxUy/SO+ll15Kkrz11lujMZ/y41N+2l5H25/yc++99yZJHn744YYexeyQ2fbm6JrMLl5m5bW9Obomr5Pn1Q4yAAAUM72D/OGHHyZJLl++3PFKoDnDt6BZRDLLIlrUzMori6ipvNpBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIBCQQYAgEJBBgCAQkEGAIDicBd3urq6OtbXra+vt7wStvPVcrzc0n30B7fvl7EffHFw8KtN3MNvJkn+Tr645cwXPvjC6Pjomz+XJOnlUFnb9U1jf5Ofjs59P38w8crW1taSJB999NFYX3/s2LGJ73NSMst2mo3sLw0OfmXLqU9yI7N/1v/5jYNe2d/pr28aW//sZ6NTH537jxMvbd4yK6+07Td/aZDXX9ma16rf3/jXvtfr7Tj22Wefjc6dO3du4rU1lVc7yAAAUCjIAABQdHKJxaeffjrW133p+sZT3X9Sxq62sB42+2fl+G+3dB/DJ/bOlrEfnBwcfKuJe9iY5JdzesuZ46//rdHxvW9u3PbK+eHlH8Ox+mRlE5dYXLt2LUly6dKl0djwqc6lpaUtYydOnJj4Pic1bmavDzLLwdBsZH9tcPD1LafeyfHR8X9Z33hKt1ey0h9kZTi29ulfjc41cYnFvGVWXmnbt35tkNevb81rtVtOhmP172sTl1g0lVc7yAAAUHSyg7zdRdufP5ckZwa3l8v577e4LjZ8uxwf2vGrJjPcQf7rOvi9we3jTdzDxiT/O7duObO8cuOlh/9j8DLE3V6kt57PctCNm1kOlmYj+93BwRtbTv2s/FP1yfrGM0C77SD3r/9NEyuaW/JK2x7/7iCvb2zNazXODvKsPpNhBxkAAIpeF79NPvLII2Pd6c8G16L8z08+GY193M6SYGb1+/2t20BTNm5mh9ePfVIyCwdN15mVVxjfTnm1gwwAAIWCDAAARSeXWPR6Pa8SgDF1/XRtIrOwF11nVl5hfC6xAACAMSjIAABQKMgAAFAoyAAAUCjIAABQKMgAAFAoyAAAUCjIAABQKMgAAFAoyAAAUCjIAABQ9Pp9H9kOAABDdpABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoFCQAQCgUJABAKBQkAEAoPj/ih77CmxSybwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x489.6 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6.8))\n",
    "for row in range(2):\n",
    "    for col in range(3):\n",
    "        plt.subplot(2, 3, row * 3 + col + 1)\n",
    "        plot_observation(trajectories.observation[row, col].numpy())\n",
    "plt.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0, wspace=0.02)\n",
    "save_fig(\"sub_episodes_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = replay_buffer.as_dataset(\n",
    "    sample_batch_size=64,\n",
    "    num_steps=2,\n",
    "    num_parallel_calls=3).prefetch(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the main functions to TF Functions for better performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.utils.common import function\n",
    "\n",
    "collect_driver.run = function(collect_driver.run)\n",
    "agent.train = function(agent.train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we are ready to run the main loop!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_agent(n_iterations):\n",
    "    time_step = None\n",
    "    policy_state = agent.collect_policy.get_initial_state(tf_env.batch_size)\n",
    "    iterator = iter(dataset)\n",
    "    for iteration in range(n_iterations):\n",
    "        time_step, policy_state = collect_driver.run(time_step, policy_state)\n",
    "        trajectories, buffer_info = next(iterator)\n",
    "        train_loss = agent.train(trajectories)\n",
    "        print(\"\\r{} loss:{:.5f}\".format(\n",
    "            iteration, train_loss.loss.numpy()), end=\"\")\n",
    "        if iteration % 1000 == 0:\n",
    "            log_metrics(train_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next cell to train the agent for 10,000 steps. Then look at its behavior by running the following cell. You can run these two cells as many times as you wish. The agent will keep improving!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0528 08:49:00.697262 140735810999168 deprecation.py:323] From /Users/ageron/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/learning_rate_schedule.py:409: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "W0528 08:49:01.475340 140735810999168 deprecation.py:323] From /Users/ageron/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1220: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "I0528 08:49:02.463025 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 0\n",
      "\t\t EnvironmentSteps = 4\n",
      "\t\t AverageReturn = 0.0\n",
      "\t\t AverageEpisodeLength = 0.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "997 loss:0.01551"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:50:16.405580 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 24\n",
      "\t\t EnvironmentSteps = 4004\n",
      "\t\t AverageReturn = 1.399999976158142\n",
      "\t\t AverageEpisodeLength = 180.5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 loss:0.00024"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:51:28.353239 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 47\n",
      "\t\t EnvironmentSteps = 8004\n",
      "\t\t AverageReturn = 0.8999999761581421\n",
      "\t\t AverageEpisodeLength = 165.89999389648438\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2997 loss:0.00010"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:52:36.316717 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 69\n",
      "\t\t EnvironmentSteps = 12004\n",
      "\t\t AverageReturn = 0.800000011920929\n",
      "\t\t AverageEpisodeLength = 162.3000030517578\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3999 loss:0.00751"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:53:47.764101 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 92\n",
      "\t\t EnvironmentSteps = 16004\n",
      "\t\t AverageReturn = 0.699999988079071\n",
      "\t\t AverageEpisodeLength = 161.89999389648438\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4997 loss:0.00032"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:54:57.040647 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 111\n",
      "\t\t EnvironmentSteps = 20004\n",
      "\t\t AverageReturn = 1.0\n",
      "\t\t AverageEpisodeLength = 181.60000610351562\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6000 loss:0.00006"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:56:07.210252 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 131\n",
      "\t\t EnvironmentSteps = 24004\n",
      "\t\t AverageReturn = 1.7000000476837158\n",
      "\t\t AverageEpisodeLength = 206.39999389648438\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6999 loss:0.00784"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:57:18.494511 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 154\n",
      "\t\t EnvironmentSteps = 28004\n",
      "\t\t AverageReturn = 1.100000023841858\n",
      "\t\t AverageEpisodeLength = 182.6999969482422\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7997 loss:0.00002"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:58:35.320452 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 175\n",
      "\t\t EnvironmentSteps = 32004\n",
      "\t\t AverageReturn = 1.7999999523162842\n",
      "\t\t AverageEpisodeLength = 196.60000610351562\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8997 loss:0.00749"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0528 08:59:51.332596 140735810999168 metric_utils.py:47]  \n",
      "\t\t NumberOfEpisodes = 195\n",
      "\t\t EnvironmentSteps = 36004\n",
      "\t\t AverageReturn = 1.100000023841858\n",
      "\t\t AverageEpisodeLength = 185.8000030517578\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9999 loss:0.00001"
     ]
    }
   ],
   "source": [
    "train_agent(n_iterations=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = []\n",
    "def save_frames(trajectory):\n",
    "    global frames\n",
    "    frames.append(tf_env.pyenv.envs[0].render(mode=\"rgb_array\"))\n",
    "\n",
    "prev_lives = tf_env.pyenv.envs[0].ale.lives()\n",
    "def reset_and_fire_on_life_lost(trajectory):\n",
    "    global prev_lives\n",
    "    lives = tf_env.pyenv.envs[0].ale.lives()\n",
    "    if prev_lives != lives:\n",
    "        tf_env.reset()\n",
    "        tf_env.pyenv.envs[0].step(np.array(1))\n",
    "        prev_lives = lives\n",
    "\n",
    "watch_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    agent.policy,\n",
    "    observers=[save_frames, reset_and_fire_on_life_lost, ShowProgress(1000)],\n",
    "    num_steps=1000)\n",
    "final_time_step, final_policy_state = watch_driver.run()\n",
    "\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to save an animated GIF to show off your agent to your friends, here's one way to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PIL\n",
    "\n",
    "image_path = os.path.join(\"images\", \"rl\", \"breakout.gif\")\n",
    "frame_images = [PIL.Image.fromarray(frame) for frame in frames[:150]]\n",
    "frame_images[0].save(image_path, format='GIF',\n",
    "                     append_images=frame_images[1:],\n",
    "                     save_all=True,\n",
    "                     duration=30,\n",
    "                     loop=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"images/rl/breakout.gif\" />\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<img src=\"images/rl/breakout.gif\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra material"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deque vs Rotating List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `deque` class offers fast append, but fairly slow random access (for large replay memories):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[121958, 671155, 131932, 365838, 259178]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import deque\n",
    "np.random.seed(42)\n",
    "\n",
    "mem = deque(maxlen=1000000)\n",
    "for i in range(1000000):\n",
    "    mem.append(i)\n",
    "[mem[i] for i in np.random.randint(1000000, size=5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76.8 ns ± 0.31 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "320 µs ± 23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit [mem[i] for i in np.random.randint(1000000, size=5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you could use a rotating list like this `ReplayMemory` class. This would make random access faster for large replay memories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReplayMemory:\n",
    "    def __init__(self, max_size):\n",
    "        self.buffer = np.empty(max_size, dtype=np.object)\n",
    "        self.max_size = max_size\n",
    "        self.index = 0\n",
    "        self.size = 0\n",
    "\n",
    "    def append(self, obj):\n",
    "        self.buffer[self.index] = obj\n",
    "        self.size = min(self.size + 1, self.max_size)\n",
    "        self.index = (self.index + 1) % self.max_size\n",
    "\n",
    "    def sample(self, batch_size):\n",
    "        indices = np.random.randint(self.size, size=batch_size)\n",
    "        return self.buffer[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([757386, 904203, 190588, 595754, 865356], dtype=object)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mem = ReplayMemory(max_size=1000000)\n",
    "for i in range(1000000):\n",
    "    mem.append(i)\n",
    "mem.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "761 ns ± 17.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.97 µs ± 16.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Custom TF-Agents Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a custom TF-Agent environment, you just need to write a class that inherits from the `PyEnvironment` class and implements a few methods. For example, the following minimal environment represents a simple 4x4 grid. The agent starts in one corner (0,0) and must move to the opposite corner (3,3). The episode is done if the agent reaches the goal (it gets a +10 reward) or if the agent goes out of bounds (-1 reward). The actions are up (0), down (1), left (2) and right (3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyEnvironment(tf_agents.environments.py_environment.PyEnvironment):\n",
    "    def __init__(self, discount=1.0):\n",
    "        super().__init__()\n",
    "        self._action_spec = tf_agents.specs.BoundedArraySpec(\n",
    "            shape=(), dtype=np.int32, name=\"action\", minimum=0, maximum=3)\n",
    "        self._observation_spec = tf_agents.specs.BoundedArraySpec(\n",
    "            shape=(4, 4), dtype=np.int32, name=\"observation\", minimum=0, maximum=1)\n",
    "        self.discount = discount\n",
    "\n",
    "    def action_spec(self):\n",
    "        return self._action_spec\n",
    "\n",
    "    def observation_spec(self):\n",
    "        return self._observation_spec\n",
    "\n",
    "    def _reset(self):\n",
    "        self._state = np.zeros(2, dtype=np.int32)\n",
    "        obs = np.zeros((4, 4), dtype=np.int32)\n",
    "        obs[self._state[0], self._state[1]] = 1\n",
    "        return tf_agents.trajectories.time_step.restart(obs)\n",
    "\n",
    "    def _step(self, action):\n",
    "        self._state += [(-1, 0), (+1, 0), (0, -1), (0, +1)][action]\n",
    "        reward = 0\n",
    "        obs = np.zeros((4, 4), dtype=np.int32)\n",
    "        done = (self._state.min() < 0 or self._state.max() > 3)\n",
    "        if not done:\n",
    "            obs[self._state[0], self._state[1]] = 1\n",
    "        if done or np.all(self._state == np.array([3, 3])):\n",
    "            reward = -1 if done else +10\n",
    "            return tf_agents.trajectories.time_step.termination(obs, reward)\n",
    "        else:\n",
    "            return tf_agents.trajectories.time_step.transition(obs, reward,\n",
    "                                                               self.discount)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action and observation specs will generally be instances of the `ArraySpec` or `BoundedArraySpec` classes from the `tf_agents.specs` package (check out the other specs in this package as well). Optionally, you can also define a `render()` method, a `close()` method to free resources, as well as a `time_step_spec()` method if you don't want the `reward` and `discount` to be 32-bit float scalars. Note that the base class takes care of keeping track of the current time step, which is why we must implement `_reset()` and `_step()` rather than `reset()` and `step()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[1, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0]], dtype=int32))"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_env = MyEnvironment()\n",
    "time_step = my_env.reset()\n",
    "time_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[0, 0, 0, 0],\n",
       "       [1, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0]], dtype=int32))"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_step = my_env.step(1)\n",
    "time_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# My exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. - 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These were theoretical questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use policy gradients to solve OpenAI Gym’s LunarLander-v2 environment. You\n",
    "will need to install the Box2D dependencies ( python3 -m pip install -U\n",
    "gym[box2d] )."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make('LunarLander-v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize the environment by calling is `reset()` method. This returns an observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the observation looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00499964,  1.4194578 , -0.506422  ,  0.37943238,  0.00580009,\n",
       "        0.11471219,  0.        ,  0.        ], dtype=float32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_state_ variable (line 306) defines the structure of the observation. (https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: I copied the below code from the official book code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we will set `mode=\"rgb_array\"` to get an image of the environment as a NumPy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 600, 3)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img = env.render(mode=\"rgb_array\")\n",
    "img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_environment(env, figsize=(5,4)):\n",
    "    plt.figure(figsize=figsize)\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    plt.imshow(img)\n",
    "    plt.axis(\"off\")\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASUAAADICAYAAACuyvefAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAEdUlEQVR4nO3c0VXbSABA0dGeVJE6KIM20kf6SBuUQR1pQ/uznCUEjOPY8tPo3k/7YI/MzGMs2Szrug6Ain/uPQCA10QJSBElIEWUgBRRAlJECUj5curOZVl8XuCAHh+/j4ev38bzzx/j6en7L7dd4r3HeX0bx7Ou6/LRfXZK/GaLaDx8/TYeH2/3+OyXKPGLW4fi6en7eP7546bPwb6dfPvGMX20S7p2TB6+fhtP4/fn4diWU18zcU7pWLY83+Pc0rE5p8TZtoyEIPEeOyVgc3ZKwG6IEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooAZta1/Xk/aIEbOazII0hStNa13U8P997FLcz87HN6pwgjTHGlxuPgzt7u3gfHu4zjlt4L0wzHd9Mzg3SGKJ0OLMv5NmPb2/+JEYvROnAZl6sMx/bXlwSpDFEaXozL86Zj23vLg3SGGMsp354WZbLH5m7Wtd1LMty72FwQGcG6cPJ6eobcDV/s0N6IUrAVVwjSGM4pwT8pWvF6IWdEpBipwRc5No7pBeHjdKtXtDPuCLGDG65fqaO0r3Cc8rLmMSJvbr1ujoZpT0soGJ4zrGH1xbe2mK9nbVTumQg115se43PZ8SJvdhqDd7s7dufHsCyLNOG5xziRNmWazPzkYAjB+m1dV29FqRsPR+nPtG9Z68ngt0T93CvP46ZnRIfs3tia/ecb6K0I+LEFu49x0Rph8SJWynMK1HaMXHimipzyYnuCfg4AZeqhOg1UZqIK3a8VYzOZ0RpUnucjJcQ3//N8jsXJXbtqLvDWQL0HlFiGrMGauYAvUeUmNJeA3W0AL1HlJhe+eqkCP1OlDiMwu5JhD4nShzS2zj4/18dogTj73dRInQ9ogRvnBMoEbodUYITxGd7vpALpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKaIEpIgSkCJKQIooASmiBKSIEpAiSkCKKAEpogSkiBKQIkpAiigBKV8+uX/ZZBQA/7FTAlJECUgRJSBFlIAUUQJSRAlI+Re0NjnUCwSS9AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how to interact with an environment. Your agent will need to select an action from an \"action space\" (the set of possible actions). Let's see what this environment's action space looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(4)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Possible actions are Nop, fire left engine, main engine, right engine (https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the lander looks like its right over the flags, let's do nothing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00999966,  1.4274173 , -0.5057368 ,  0.35372102,  0.0114686 ,\n",
       "        0.11338098,  0.        ,  0.        ], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "action = 0 # no op\n",
    "obs, reward, done, info = env.step(action)\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how it looks like now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure cart_pole_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAADvCAYAAADM8A71AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAFW0lEQVR4nO3c23HiWBRA0aupjqLjcBik0Xk4j07DYRBHp6H5mMFlzxjMS2wBa325UBXIAm2fukie5nkeANzeX/UOADwrAQaICDBARIABIgIMEPlxaOM0TS6RYDGbzet4+flrjDHG9s/v8fb2+r/HL/XV827//B5jjPfHYWnzPE9fPW4CJrHZvL7//DGSS3p7ex3bP7+vFne4lACTKUP48vPXpz8CUJgO3YhhCYIl7Ft6+Lj9Wr6arL97fbi2fUsQB9eAYc2Ek3snwGT2TZ9Lh/Xt7XWMzaIvAUexBEFis3k1wfI09i1BCDDAwlyGBrAyAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCAiAADRAQYICLAABEBBogIMEBEgAEiAgwQEWCABc3zvHebAANEftQ7APCIDk2+OyZgzjLP89hu671Yj+12OB68Oya+Y5iAudBX0Xl5uf1+rIXjwbHxHUOAWcDHCImP4/FMTonvGALMwsTnM8fjcZ0a3zEEmAUIy2eOx+M7J75jCDAXEpfPHI/ncm54d6ZDTzBN02XPzsOa53lM01TvBmROjO+XJ4vL0ABOdOnkuyPAACe4VnzHEGCAo10zvmP4Eg7gW9cO744JGCBiAgbYY6nJd0eAV+aYN9zlX7C8peM7hgDvdYuDf66P+ybGcH23Ov8F+F9rDu4hboiA67plCw4GeLcj936C32tcj/Uo7xPUbt2Koybgi+93XjgMjx7YY1magPMVHbnJEsQlv9g0TQJ7BlMxHKfsy+qvAxbfy8zz7BjCHvW5sfoAcx1CDJ+t4XwQ4CcjxLCO+I4hwE9LiHlWa/rcuw74yblygmexpvDumIB5t8YPKDwyEzCfmIh5NGseLEzA7GWdmHu39s+vCZhvmYpZu7WHdh8B5iTusKN0r6HdR4A5y6OdCNfgj9IyHvmzJsBwJZZqLvPIod1HgGEB/42JIP/jGSN7iADDDTzbdCy0xxFguLFH/iJTeE8jwBC596lYbC8nwLACa18zFttlCDCsUDkdi+3tCDCs3C3WjEW3IcBwJ641FYvteggw3KFjYyy26ybAcOdE9n75d5QAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASICDBARYICIAANEBBggIsAAEQEGiAgwQESAASI/vtk+3WQvAJ6QCRggIsAAEQEGiAgwQESAASICDBD5GzAlasXAljuQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "save_fig(\"cart_pole_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like it's doing what we're telling it to do!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment also tells the agent how much reward it got during the last step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19813816635709713"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the game is over, the environment returns `done=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, `info` is an environment-specific dictionary that can provide some extra information that you may find useful for debugging or for training. For example, in some games it may indicate how many lives the agent has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence of steps between the moment the environment is reset until it is done is called an \"episode\". At the end of an episode (i.e., when `step()` returns `done=True`), you should reset the environment before you continue to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "if done:\n",
    "    obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: The code below was copied from the notebook code above and modified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Policies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural network that will take observations as inputs, and output the action to take for each observation. To choose an action, the network will estimate a probability for each action, then we will select an action randomly according to the estimated probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "n_inputs = 8\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(15, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(4, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular environment, the past actions and observations can safely be ignored, since each observation contains the environment's full state. If there were some hidden state then you may need to consider past actions and observations in order to try to infer the hidden state of the environment. Our problem is thus as simple as can be: the current observation is noise-free and contains the environment's full state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may wonder why we plan to pick a random action based on the probability given by the policy network, rather than just picking the action with the highest probability. This approach lets the agent find the right balance between _exploring_ new actions and _exploiting_ the actions that are known to work well. Here's an analogy: suppose you go to a restaurant for the first time, and all the dishes look equally appealing so you randomly pick one. If it turns out to be good, you can increase the probability to order it next time, but you shouldn't increase that probability to 100%, or else you will never try out the other dishes, some of which may be even better than the one you tried."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a small function that will run the model to play one episode, and return the frames so we can display an animation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_policy_net(model, n_max_steps=10000, seed=42):\n",
    "    frames = []\n",
    "    env = gym.make(\"LunarLander-v2\")\n",
    "    env.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    obs = env.reset()\n",
    "    for step in range(n_max_steps):\n",
    "        frames.append(env.render(mode=\"rgb_array\"))\n",
    "        action_probas = model.predict(obs.reshape(1, -1))\n",
    "        action = int(np.argmax(action_probas, axis=1))\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "    env.close()\n",
    "    return frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at how well this randomly initialized policy network performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'plot_animation' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-56cf2ed2775a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mframes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrender_policy_net\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplot_animation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'plot_animation' is not defined"
     ]
    }
   ],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yeah... pretty bad. The neural network will have to learn to do better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train this neural network we will need to define the target probabilities `y`. If an action is good we should increase its probability, and conversely if it is bad we should reduce it. But how do we know whether an action is good or bad? The problem is that most actions have delayed effects, so when you win or lose points in an episode, it is not clear which actions contributed to this result: was it just the last action? Or the last 10? Or just one action 50 steps earlier? This is called the _credit assignment problem_.\n",
    "\n",
    "The _Policy Gradients_ algorithm tackles this problem by first playing multiple episodes, then making the actions in good episodes slightly more likely, while actions in bad episodes are made slightly less likely. First we play, then we go back and think about what we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating a function to play a single step using the model. We will also pretend for now that whatever action it takes is the right one, so we can compute the loss and its gradients (we will just save these gradients for now, and modify them later depending on how good or bad the action turned out to be):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, obs, model, loss_fn):\n",
    "    with tf.GradientTape() as tape:\n",
    "        action_probas = model.predict(obs.reshape(1, -1))\n",
    "        ind = np.argpartition(action_probas, -3)[-3:]\n",
    "        action_index = np.random.choice(ind.shape[0], 1)\n",
    "        action = ind[action_index]\n",
    "        y_target = int(np.argmax(action_probas, axis=1))\n",
    "        loss = tf.reduce_mean(loss_fn(y_target, action_probas[action]))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    obs, reward, done, info = env.step(int(action[0, 0].numpy()))\n",
    "    return obs, reward, done, grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: Below is the function and the paragraph of explanation from the original code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, obs, model, loss_fn):\n",
    "    with tf.GradientTape() as tape:\n",
    "        left_proba = model(obs[np.newaxis])\n",
    "        action = (tf.random.uniform([1, 1]) > left_proba)\n",
    "        y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32)\n",
    "        loss = tf.reduce_mean(loss_fn(y_target, left_proba))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    obs, reward, done, info = env.step(int(action[0, 0].numpy()))\n",
    "    return obs, reward, done, grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `left_proba` is high, then `action` will most likely be `False` (since a random number uniformally sampled between 0 and 1 will probably not be greater than `left_proba`). And `False` means 0 when you cast it to a number, so `y_target` would be equal to 1 - 0 = 1. In other words, we set the target to 1, meaning we pretend that the probability of going left should have been 100% (so we took the right action)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create another function that will rely on the `play_one_step()` function to play multiple episodes, returning all the rewards and gradients, for each episode and each step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn):\n",
    "    all_rewards = []\n",
    "    all_grads = []\n",
    "    for episode in range(n_episodes):\n",
    "        current_rewards = []\n",
    "        current_grads = []\n",
    "        obs = env.reset()\n",
    "        for step in range(n_max_steps):\n",
    "            obs, reward, done, grads = play_one_step(env, obs, model, loss_fn)\n",
    "            current_rewards.append(reward)\n",
    "            current_grads.append(grads)\n",
    "            if done:\n",
    "                break\n",
    "        all_rewards.append(current_rewards)\n",
    "        all_grads.append(current_grads)\n",
    "    return all_rewards, all_grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Policy Gradients algorithm uses the model to play the episode several times (e.g., 10 times), then it goes back and looks at all the rewards, discounts them and normalizes them. So let's create couple functions for that: the first will compute discounted rewards; the second will normalize the discounted rewards across many episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rewards(rewards, discount_rate):\n",
    "    discounted = np.array(rewards)\n",
    "    for step in range(len(rewards) - 2, -1, -1):\n",
    "        discounted[step] += discounted[step + 1] * discount_rate\n",
    "    return discounted\n",
    "\n",
    "def discount_and_normalize_rewards(all_rewards, discount_rate):\n",
    "    all_discounted_rewards = [discount_rewards(rewards, discount_rate)\n",
    "                              for rewards in all_rewards]\n",
    "    flat_rewards = np.concatenate(all_discounted_rewards)\n",
    "    reward_mean = flat_rewards.mean()\n",
    "    reward_std = flat_rewards.std()\n",
    "    return [(discounted_rewards - reward_mean) / reward_std\n",
    "            for discounted_rewards in all_discounted_rewards]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say there were 3 actions, and after each action there was a reward: first 10, then 0, then -50. If we use a discount factor of 80%, then the 3rd action will get -50 (full credit for the last reward), but the 2nd action will only get -40 (80% credit for the last reward), and the 1st action will get 80% of -40 (-32) plus full credit for the first reward (+10), which leads to a discounted reward of -22:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-22, -40, -50])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_rewards([10, 0, -50], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To normalize all discounted rewards across all episodes, we compute the mean and standard deviation of all the discounted rewards, and we subtract the mean from each discounted reward, and divide by the standard deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([-0.28435071, -0.86597718, -1.18910299]),\n",
       " array([1.26665318, 1.0727777 ])]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_and_normalize_rewards([[10, 0, -50], [10, 20]], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iterations = 150\n",
    "n_episodes_per_update = 10\n",
    "n_max_steps = 200\n",
    "discount_rate = 0.95"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.01)\n",
    "loss_fn = keras.losses.binary_crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_inputs = 8\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(15, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(4, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py\u001b[0m in \u001b[0;36m_GradientsHelper\u001b[0;34m(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph)\u001b[0m\n\u001b[1;32m    606\u001b[0m           \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 607\u001b[0;31m             \u001b[0mgrad_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_gradient_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    608\u001b[0m           \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py\u001b[0m in \u001b[0;36mget_gradient_function\u001b[0;34m(op)\u001b[0m\n\u001b[1;32m   2494\u001b[0m     \u001b[0mop_type\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2495\u001b[0;31m   \u001b[0;32mreturn\u001b[0m \u001b[0m_gradient_registry\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlookup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mop_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2496\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/framework/registry.py\u001b[0m in \u001b[0;36mlookup\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m     96\u001b[0m       raise LookupError(\n\u001b[0;32m---> 97\u001b[0;31m           \"%s registry has no entry for: %s\" % (self._name, name))\n\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m: gradient registry has no entry for: IteratorGetNext",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-27-7ba1b8653db8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0miteration\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_iterations\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     all_rewards, all_grads = play_multiple_episodes(\n\u001b[0;32m----> 6\u001b[0;31m         env, n_episodes_per_update, n_max_steps, model, loss_fn)\n\u001b[0m\u001b[1;32m      7\u001b[0m     \u001b[0mtotal_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mall_rewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m                     \u001b[0;31m# Not shown in the book\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     print(\"\\rIteration: {}, mean rewards: {:.1f}\".format(          # Not shown\n",
      "\u001b[0;32m<ipython-input-20-e190097d0502>\u001b[0m in \u001b[0;36mplay_multiple_episodes\u001b[0;34m(env, n_episodes, n_max_steps, model, loss_fn)\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0mobs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_max_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m             \u001b[0mobs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrads\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mplay_one_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m             \u001b[0mcurrent_rewards\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreward\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m             \u001b[0mcurrent_grads\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-19-26f89741ae8e>\u001b[0m in \u001b[0;36mplay_one_step\u001b[0;34m(env, obs, model, loss_fn)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mplay_one_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGradientTape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mtape\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m         \u001b[0maction_probas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m         \u001b[0mind\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margpartition\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction_probas\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m         \u001b[0maction_index\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mind\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1011\u001b[0m         \u001b[0mmax_queue_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_queue_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1012\u001b[0m         \u001b[0mworkers\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1013\u001b[0;31m         use_multiprocessing=use_multiprocessing)\n\u001b[0m\u001b[1;32m   1014\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1015\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mreset_metrics\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, model, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[1;32m    496\u001b[0m         \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mModeKeys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPREDICT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    497\u001b[0m         \u001b[0msteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_queue_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_queue_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 498\u001b[0;31m         workers=workers, use_multiprocessing=use_multiprocessing, **kwargs)\n\u001b[0m\u001b[1;32m    499\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    500\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py\u001b[0m in \u001b[0;36m_model_iteration\u001b[0;34m(self, model, mode, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[1;32m    473\u001b[0m               \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    474\u001b[0m               \u001b[0mtraining_context\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtraining_context\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 475\u001b[0;31m               total_epochs=1)\n\u001b[0m\u001b[1;32m    476\u001b[0m           \u001b[0mcbks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_logs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch_logs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py\u001b[0m in \u001b[0;36mrun_one_epoch\u001b[0;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)\u001b[0m\n\u001b[1;32m    126\u001b[0m         step=step, mode=mode, size=current_batch_size) as batch_logs:\n\u001b[1;32m    127\u001b[0m       \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 128\u001b[0;31m         \u001b[0mbatch_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mexecution_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    129\u001b[0m       \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mStopIteration\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOutOfRangeError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    130\u001b[0m         \u001b[0;31m# TODO(kaftan): File bug about tf function and errors.OutOfRangeError?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py\u001b[0m in \u001b[0;36mexecution_function\u001b[0;34m(input_fn)\u001b[0m\n\u001b[1;32m     96\u001b[0m     \u001b[0;31m# `numpy` translates Tensors to values in Eager mode.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     97\u001b[0m     return nest.map_structure(_non_none_constant_value,\n\u001b[0;32m---> 98\u001b[0;31m                               distributed_function(input_fn))\n\u001b[0m\u001b[1;32m     99\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mexecution_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    566\u001b[0m         \u001b[0mxla_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    567\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 568\u001b[0;31m       \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    569\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    570\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    636\u001b[0m               *args, **kwds)\n\u001b[1;32m    637\u001b[0m       \u001b[0;31m# If we did not create any variables the trace we have is good enough.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 638\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_concrete_stateful_fn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcanon_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcanon_kwds\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    639\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    640\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfn_with_cond\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minner_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0minner_kwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m   1609\u001b[0m          if isinstance(t, (ops.Tensor,\n\u001b[1;32m   1610\u001b[0m                            resource_variable_ops.BaseResourceVariable))),\n\u001b[0;32m-> 1611\u001b[0;31m         self.captured_inputs)\n\u001b[0m\u001b[1;32m   1612\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1613\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1695\u001b[0m         \u001b[0mpossible_gradient_type\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1696\u001b[0m         executing_eagerly)\n\u001b[0;32m-> 1697\u001b[0;31m     \u001b[0mforward_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs_with_tangents\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mforward_backward\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1698\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mexecuting_eagerly\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1699\u001b[0m       flat_outputs = forward_function.call(\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1421\u001b[0m     \u001b[0;34m\"\"\"Builds or retrieves a forward function for this call.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1422\u001b[0m     forward_function = self._functions.forward(\n\u001b[0;32m-> 1423\u001b[0;31m         self._inference_args, self._input_tangents)\n\u001b[0m\u001b[1;32m   1424\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mforward_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_inference_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_tangents\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1425\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, inference_args, input_tangents)\u001b[0m\n\u001b[1;32m   1183\u001b[0m       (self._forward, self._forward_graph, self._backward,\n\u001b[1;32m   1184\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forwardprop_output_indices\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_forwardprop_outputs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1185\u001b[0;31m            self._forward_and_backward_functions(inference_args, input_tangents))\n\u001b[0m\u001b[1;32m   1186\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36m_forward_and_backward_functions\u001b[0;34m(self, inference_args, input_tangents)\u001b[0m\n\u001b[1;32m   1329\u001b[0m     \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func_graph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_inference_outputs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1330\u001b[0m     return self._build_functions_for_outputs(\n\u001b[0;32m-> 1331\u001b[0;31m         outputs, inference_args, input_tangents)\n\u001b[0m\u001b[1;32m   1332\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1333\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py\u001b[0m in \u001b[0;36m_build_functions_for_outputs\u001b[0;34m(self, outputs, inference_args, input_tangents)\u001b[0m\n\u001b[1;32m    888\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func_graph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    889\u001b[0m             \u001b[0mgrad_ys\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgradients_wrt_outputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 890\u001b[0;31m             src_graph=self._func_graph)\n\u001b[0m\u001b[1;32m    891\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    892\u001b[0m       captures_from_forward = [\n",
      "\u001b[0;32m~/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py\u001b[0m in \u001b[0;36m_GradientsHelper\u001b[0;34m(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph)\u001b[0m\n\u001b[1;32m    621\u001b[0m               raise LookupError(\n\u001b[1;32m    622\u001b[0m                   \u001b[0;34m\"No gradient defined for operation '%s' (op type: %s)\"\u001b[0m \u001b[0;34m%\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 623\u001b[0;31m                   (op.name, op.type))\n\u001b[0m\u001b[1;32m    624\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mloop_state\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    625\u001b[0m           \u001b[0mloop_state\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEnterGradWhileContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbefore\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext)"
     ]
    }
   ],
   "source": [
    "env = gym.make(\"LunarLander-v2\")\n",
    "env.seed(42);\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    all_rewards, all_grads = play_multiple_episodes(\n",
    "        env, n_episodes_per_update, n_max_steps, model, loss_fn)\n",
    "    total_rewards = sum(map(sum, all_rewards))                     # Not shown in the book\n",
    "    print(\"\\rIteration: {}, mean rewards: {:.1f}\".format(          # Not shown\n",
    "        iteration, total_rewards / n_episodes_per_update), end=\"\") # Not shown\n",
    "    all_final_rewards = discount_and_normalize_rewards(all_rewards,\n",
    "                                                       discount_rate)\n",
    "    all_mean_grads = []\n",
    "    for var_index in range(len(model.trainable_variables)):\n",
    "        mean_grads = tf.reduce_mean(\n",
    "            [final_reward * all_grads[episode_index][step][var_index]\n",
    "             for episode_index, final_rewards in enumerate(all_final_rewards)\n",
    "                 for step, final_reward in enumerate(final_rewards)], axis=0)\n",
    "        all_mean_grads.append(mean_grads)\n",
    "    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO: Fix the bug above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}