Notebook_Report_P1_Tabular_Methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14cd8caf",
   "metadata": {},
   "source": [
    "#  Pokémon RL - Tabular Methods \n",
    "\n",
    "## **Authors:**\n",
    "\n",
    "* [Bruno César de Oliveira Souza](mailto:b234837@dac.unicamp.br) - [Github](https://github.com/brunocosouza)\n",
    "\n",
    "* [Leonardo de Lellis Rossi](mailto:l261900@dac.unicamp.br) - [Github](https://github.com/leolellisr) \n",
    "\n",
    "* [Maurício Pereira Lopes](mailto:m225242@g.unicamp.br)\n",
    "\n",
    "## Special Thanks:\n",
    "\n",
    "* To our colleague **Henrique Lima Cará de Oliveira** who collaborated in the development of the codes and execution of the experiments\n",
    "\n",
    "\n",
    "## Main info\n",
    "\n",
    "[Git Repository](https://github.com/leolellisr/poke_RL)\n",
    "\n",
    "Notebooks of implemented methods available [here](https://github.com/leolellisr/poke_RL/tree/master/notebooks)\n",
    "\n",
    "Equivalent python files of implemented methods available [here](https://github.com/leolellisr/poke_RL/tree/master/py)\n",
    "\n",
    "json files of trained tabular methods available [here](https://drive.google.com/drive/folders/1GwNQSsOR0PPtPKlbIy9NzWvTnTtMOD8_?usp=sharing)\n",
    "\n",
    "[Graphs MC Control, MC Control FA, Q-Learning, Q-Learning FA, SARSA($\\lambda$) Deterministic and SARSA($\\lambda$) FA](https://app.neptune.ai/leolellisr/rl-pokeenv)\n",
    "\n",
    "[Graphs SARSA($\\lambda$) Stochastic](https://app.neptune.ai/mauricioplopes/poke-env)\n",
    "\n",
    "[Presentation video for Tabular methods](https://youtu.be/uSZE9gnheSI)\n",
    "\n",
    "Images and results are showed with IFrame from IPython.display, but it needs the images stored in the [/images/report](https://github.com/leolellisr/poke_RL/tree/master/images/report) folder, available in our [git repository](https://github.com/leolellisr/poke_RL). \n",
    "\n",
    "Images and Results also are presented with links from imgur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "64667434",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7364fd46",
   "metadata": {},
   "source": [
    "# Goal and Motivation\n",
    "\n",
    "* This project aims to employ different reinforcement learning techniques to train agents in a Pokémon battle simulator;\n",
    "\n",
    "* Our motivation is that the trainer automatically learns, by making decisions through the analysis of states and rewards related to their performance, how to win battles throughout the episodes, noticing:\n",
    "  * the different types between Pokémon;\n",
    "  * which moves cause more damage to the opponent's Pokémon;\n",
    "  * what are the possible strategies using no-damage moves;\n",
    "  * and the best times to switch Pokémon. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64377fad",
   "metadata": {},
   "source": [
    "## **The  problem addressed**\n",
    "* [Pokémon](https://www.pokemon.com) is a popular Japanese RPG (Role Playing Game) which stands a world championship every year; \n",
    "* One single [battle](https://bulbapedia.bulbagarden.net/wiki/Pokémon_battle) of Pokémon has two players. Each player has a 6-Pokémon team; \n",
    "* Each Pokémon has:\n",
    " * 6 [stats](https://bulbapedia.bulbagarden.net/wiki/Stat) (Health Points, Attack, Defense, Special Attack, Special Defense, Speed). The first 5 are used in the damage calculation. The speed defined which Pokémon moves first in the turn.\n",
    "  * The Health Points goes from 100% (healthy) to 0% (fainted);\n",
    " * 4 possible moves (each with a limited number of uses). Each move has an accuracy, a percentage of success or fail;\n",
    " * one [ability](https://bulbapedia.bulbagarden.net/wiki/Ability) that has special effects in the field;\n",
    " * one [nature](https://bulbapedia.bulbagarden.net/wiki/Nature) that specifies which stats are higher and which are lower;\n",
    " * one [item](https://bulbapedia.bulbagarden.net/wiki/Item), that can  restore Health Points or increase the Power of an Attack.\n",
    "* The winner of the battle is the player that makes all Pokémon of the oposing team to faint (all oposing Pokémon with health points equals zero, \"last man standing\" criteria);\n",
    "* Only one Pokémon of each team can be at the battle field at the same time;\n",
    "* Every turn, each players select one action: one of the 4 moves of their active Pokémon or [switching](https://bulbapedia.bulbagarden.net/wiki/Recall) for one of other non-fainted Pokémon of their team;\n",
    "\n",
    "* Pokémon can be summarized as an analyze state (turn) -> take action sequence game. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc2d43a8",
   "metadata": {},
   "source": [
    "* By standard, Pokémon is a stochastic game:\n",
    " * One move can have an accuracy value less than 100%, then this move has a probability to be missed;\n",
    " * The damage moves (attacks) have the following [damage calculation](https://bulbapedia.bulbagarden.net/wiki/Damage):\n",
    "  ![Damage](https://wikimedia.org/api/rest_v1/media/math/render/svg/b8c51fed93bb9a80ae8febc13700a40b8a5da402)\n",
    "  \n",
    " where:\n",
    "  * **[Level](https://bulbapedia.bulbagarden.net/wiki/Level)** (the level of the attacking Pokémon);\n",
    "  * **A** is the effective Attack stat of the attacking Pokémon if the used move is a physical move, or the effective Special Attack stat of the attacking Pokémon if the used move is a special move;\n",
    "  * **D** is the effective Defense stat of the target if the used move is a physical move or a special move that uses the target's Defense stat, or the effective Special Defense of the target if the used move is an other special move;\n",
    "  * **[Power](https://bulbapedia.bulbagarden.net/wiki/Power)** is the effective power of the used move;\n",
    "  * **Weather** is 1.5 if a Water-type move is being used during rain or a Fire-type move during harsh sunlight, and 0.5 if a Water-type move is used during harsh sunlight or a Fire-type move during rain, and 1 otherwise.\n",
    "  * **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** has 6.25% chance of occurs and multiplies the damage by 1.5;\n",
    "  * **random** is a random factor between 0.85 and 1.00 (inclusive):\n",
    "  * **[STAB](https://bulbapedia.bulbagarden.net/wiki/Same-type_attack_bonus)** is the same-type attack bonus. This is equal to 1.5 if the move's type matches any of the user's types, 2 if the user of the move additionally has the ability Adaptability, and 1 if otherwise;\n",
    "  * **[Type](https://bulbapedia.bulbagarden.net/wiki/Type)** is the type effectiveness. This can be 0 (ineffective); 0.25, 0.5 (not very effective); 1 (normally effective); 2, or 4 (super effective), depending on both the move's and target's types;\n",
    "  * **[Burn](https://bulbapedia.bulbagarden.net/wiki/Burn_(status_condition))** is 0.5 (from Generation III onward) if the attacker is burned, its Ability is not Guts, and the used move is a physical move (other than Facade from Generation VI onward), and 1 otherwise.\n",
    "  * **other** is 1 in most cases, and a different multiplier when specific interactions of moves, Abilities, or items take effect. In this work, this is applied just to Pokémon that has the item **Life Orb**, which multiplies the damage by 1.3.\n",
    "  \n",
    "  * **Not** used in this work (equals 1):\n",
    "   * Targets (for Battles with more than two active Pokémon in the field);\n",
    "   * Badge ( just applied in Generation II);\n",
    "   \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f09c8c17",
   "metadata": {},
   "source": [
    "# **MDP formulation and discretization model** \n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "We considered our original (stochastic) MDP as a tuple $M = (S, A, \\phi, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index (0: Venusaur,  1: Pikachu, 2: Tauros, 3: Sirfetch'd, 4: Blastoise, 5: Charizard);\n",
    " * [1] Opponent Active Pokémon index (0: Eevee,  1: Vaporeon, 2: Leafeon, 3: Sylveon, 4: Jolteon, 5: Umbreon);\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8]. One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **$\\phi$** is a stochastic transition function that occurs from state **s** to state **s'**, by taking an action **a**. The following parameters are part of our  stochastic transition function:\n",
    " * Move's accuracy (chance of the move successfully occurs or fail);\n",
    " * Damage calculation: The **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** parameter (6.25% chance of occurs) and the **random** parameter, ranging from 0.85 and 1.00 (inclusive).\n",
    " * Move's effects: Some moves have [additional effects](https://bulbapedia.bulbagarden.net/wiki/Additional_effect). e.g.: Iron Head have 30% chance of flinching the target (target cannot move in the turn).\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of the turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Stochastic Team\n",
    "\n",
    "Our stochastic team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Team](https://imgur.com/KSXvlmO).\n",
    "\n",
    "The stochastic opponent team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Opponent Team](https://imgur.com/rLF5Cli)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "db9931f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c45d909d68>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team stochastic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e37ecd0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Opponent stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c45d92a358>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Opponent stochastic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a11a00",
   "metadata": {},
   "source": [
    "## Deterministic\n",
    "\n",
    "To adapt Pokémon to a deterministic environment, we use Pokémon that cannot receive a critical hit, moves with only 100% accuracy and no side effects, and edit the server code to ignore the random parameter in damage calculation, removing the stochastic transition function $\\phi$ from our MDP. Therefore, now our MDP is a tuple $M = (S, A, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index ;\n",
    " * [1] Opponent Active Pokémon index ;\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8] (len: 9). One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of each turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Deterministic Team\n",
    "\n",
    "Our deterministic team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Team](https://imgur.com/DeRAEQb).\n",
    "\n",
    "The deterministic opponent team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Opponent Team](https://imgur.com/Hltn5OO).\n",
    "\n",
    "We use on both teams only Pokémon with [Battle Armor](https://bulbapedia.bulbagarden.net/wiki/Battle_Armor_(Ability)) or [Shell Armor](https://bulbapedia.bulbagarden.net/wiki/Shell_Armor_(Ability)) abilities, which prevent critical hits from being performed. Also, we use in both teams only moves with 100% accuracy, with no chance of getting it missed, and the moves haven't additional effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ca7e611e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c45d92a320>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fbf3fc29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Opponent deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c45d92a438>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Opponent deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1949157",
   "metadata": {},
   "source": [
    "## Search space\n",
    "\n",
    "The features that integrate our states are shown in [this figure](https://imgur.com/tREjWCG) and below. For a single battle between two players with 6 Pokémon each, we have $1.016.064$ possible states.\n",
    "\n",
    "Since we have 9 possible actions, we total $9.144.576$ possibilities for each battle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0cad37e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"920\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/possible_states.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0b5f8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/possible_states.png\", width=920, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "431dcd3b",
   "metadata": {},
   "source": [
    "# **The environments built**\n",
    "\n",
    "The environment used is [Pokémon Showdown](https://play.pokemonshowdown.com), an [open-source](https://github.com/smogon/pokemon-showdown.git) Pokémon battle simulator.\n",
    "\n",
    "[Example](https://imgur.com/hjHikuc) of one battle in Pokémon Showdown.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e50a7d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/showdownEx.jpg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0b208>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/showdownEx.jpg\", width=1000, height=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b40a65",
   "metadata": {},
   "source": [
    "To communicate our agents with Pokémon Showdown we used [poke-env](https://poke-env.readthedocs.io/en/latest/) a Python environment for interacting in Pokémon Showdown battles.\n",
    "\n",
    "We used separated Python classes (available in our [git repository](https://github.com/leolellisr/poke_RL/tree/master/src)) for define the Players that are trained with each method. These classes communicates with Pokémon Showdown and implements the poke-env methods to:\n",
    "* Create battles;\n",
    "* Accept battles;\n",
    "* Send orders (actions) to Pokémon Showdown.\n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "To speed up the battles, we hosted our own server of Pokemon Showdown in localhost. It requires Node.js v10+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5723a725",
   "metadata": {},
   "outputs": [],
   "source": [
    "git clone https://github.com/smogon/pokemon-showdown.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f7a6b8",
   "metadata": {},
   "source": [
    "After clone the repository, it's needed to create a logs folder in root.\n",
    "\n",
    "To configure the server, we used the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "866e4e79",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd pokemon-showdown\n",
    "npm install\n",
    "cp config/config-example.js config/config.js"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9565e2c6",
   "metadata": {},
   "source": [
    "To start the server, we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8c6300",
   "metadata": {},
   "outputs": [],
   "source": [
    "node pokemon-showdown start --no-security"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84b18b0a",
   "metadata": {},
   "source": [
    "## Deterministic environment\n",
    "\n",
    "To adapt our environment to a deterministic setup, we had to establish the following parameters:\n",
    "\n",
    "* We removed the random component of sim/battle.ts from the Pokémon Showdown simulator;\n",
    "\n",
    "* We use on both teams only Pokémon with Battle Armor or Shell Armor abilities, which prevent critical hits from being performed;\n",
    "\n",
    "* We used in both teams only moves with 100% accuracy, with no chance of getting it missed;\n",
    "\n",
    "* We didn't use any move with additional effect. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2d9c21",
   "metadata": {},
   "source": [
    "# **Characteristics of  the problem**\n",
    "\n",
    "* Both of our environments (stochastic and deterministic) are episodic. One state occurs after another;\n",
    "\n",
    "* Our terminal states are:\n",
    " * When all our Pokémon are fainted (we lose);\n",
    " * When all opponent Pokémon are fainted (we won).\n",
    "\n",
    "* As specified before, a reward **r** is calculated at the end of a turn (beginning of next turn). The value of reward **r** is defined by the sum of:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2afadca",
   "metadata": {},
   "source": [
    "# **Monte Carlo Control**\n",
    "\n",
    "We implemented Monte Carlo Control First Visit. \n",
    "\n",
    "* We defined 2 defaultDicts, $Q(s,a)$ and $N(s,a)$, to store the value function and the number of occurences of action **a** in state **s**, respectively. Both of them are initialized to zero;\n",
    " * Each dict has the keys as the states **s**;\n",
    " * The value of a key is an array of the len of action space (len: 9);\n",
    " * Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s,a]$;\n",
    "\n",
    "* In the first move of a turn, we choose a random action;\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s,:])$. We tested with 3 values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$};\n",
    "\n",
    "* We used the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de3c5fd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def epsilon_greedy_policy(Q, n0, N):\n",
    "    def policy_fn(state):\n",
    "        dimension = Q[state].shape\n",
    "        greedy_action = np.argmax(Q[state])\n",
    "        epsilon = n0/(n0+np.sum(N[state]))\n",
    "        probabilities = np.full(dimension, epsilon / dimension[0])\n",
    "        probabilities[greedy_action] += 1 - epsilon\n",
    "        return probabilities\n",
    "    return policy_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e361d1c1",
   "metadata": {},
   "source": [
    "* In each turn, we collect our observations to define our state with the embed battle function, getting the 12 battle elements concatenated. The state is converted to a string, that is used as a key to our defaultDicts $Q(s,a)$ and $S(s,a)$;\n",
    " \n",
    "* With the state, we select an action using our policy and send it to Showdown;\n",
    "\n",
    "* We store the state and selected action to append to an episode array in the next turn with the computed reward;\n",
    "\n",
    "* At the end of the battle, $Q(s,a)$ and $N(s,a)$ are updated for the first time of each state seen in that battle:\n",
    " * $N(s,a)$ is increased by 1 for each action performed in a particular state;\n",
    " * The value of $Q(s,a)$ is summed to $\\alpha*(G(t)-Q(s,a))$;\n",
    " * The return $G(t)$ is given by $ \\sum \\limits _{i=0} ^{T-1} \\gamma^iR(t)$. We used $\\gamma = 0,75$;\n",
    " * Our policy is then updated with the new values of $Q(s,a)$ and $N(s,a)$;\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "095ff4c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "rewards = [reward for state,action,reward in self.episode]\n",
    "states = [state for state,action,reward in self.episode]\n",
    "actions = [action for state,action,reward in self.episode]\n",
    "t_array = range(len(self.episode)+1)\n",
    "        \n",
    "# Computing Q and N\n",
    "for idx, state in enumerate(states):\n",
    "    action = actions[idx]\n",
    "    if state not in self.visited_states:\n",
    "        self.N[state][action] += 1\n",
    "        returnGt = sum([reward*pow(self.gamma, t) for reward, t in zip(rewards[idx:], t_array[:(-idx-1)])]) \n",
    "\n",
    "        # incremental update of Q value is more efficient than keeping a record of all rewards\n",
    "        # and averaging after every new reward\n",
    "        # step-size: 1./N[state][action]\n",
    "                \n",
    "        self.Q[state][action] += (1/self.N[state][action])*(returnGt-self.Q[state][action]) \n",
    "        self.visited_states.append(state)\n",
    "                \n",
    "self.visited_states = []\n",
    "self.episode = []\n",
    "# Define new policy with updated Q and N\n",
    "self.policy = self.update_epsilon_greedy_policy(self.Q, self.n0, self.N)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cad9a300",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $N$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09f7befc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "62cd015f",
   "metadata": {},
   "source": [
    "## MC_Control - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-84/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### MC_Control - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control - Stochastic - Rewards (per step) - Training](https://imgur.com/NzIOSBP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "87221455",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0bac8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0509af",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Win percentages (per step) - Training\n",
    "[ MC_Control - Stochastic - Win percentages (per step) - Training](https://imgur.com/IJefGsi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "674ed6c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0bc18>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3637ff20",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1262/3333$ battles [this is $37.86\\%$ and took $743.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1063/3333$ battles [this is $31.89\\%$ and took $743.37$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1166/3333$ battles [this is $34.98\\%$ and took $622.97$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3026/3333$ battles [this is $90.79\\%$ and took $841.88$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3113/3333$ battles [this is $93.4\\%$ and took $825.55$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3130/3333$ battles [this is $93.91\\%$ and took $762.68$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acaef4c9",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[MC_Control - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/L7Ti3oY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6858676b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0b438>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acddbeeb",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "We defined two Value Functions to demonstrate the performance of our agents. \n",
    "\n",
    "The first one considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://imgur.com/57UQtyi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cc565728",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0bba8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb31f305",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://imgur.com/agyobiG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "589f48a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0b4e0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1b19e6c",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://imgur.com/ULopP5b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2f42677d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c0b2b0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_Index.pdf\", width=1000, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d187ef62",
   "metadata": {},
   "source": [
    "\n",
    "Our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://imgur.com/L1mFVQd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "97fd71d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7be9630>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29ad4bbf",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://imgur.com/rVQHGn4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bdde7dd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7be9860>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403e5a08",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://imgur.com/JTlHNuj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b247c782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7be9e10>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0669d041",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "116b1173",
   "metadata": {},
   "source": [
    "## MC_Control - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-102/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### MC_Control - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control - Deterministic - Rewards (per step) - Training](https://imgur.com/SK4jm8B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "93253d65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7be9a90>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f0a249d",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Win percentages (per step) - Training\n",
    "[MC_Control - Deterministic - Win percentages (per step) - Training](https://imgur.com/p1zFpu6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4cc7545a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d080>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbe1425",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1380/3333$ battles [this is $41.4\\%$ and took $485$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1359/3333$ battles [this is $40.77\\%$ and took $400.49$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1201/3333$ battles [this is $36.03\\%$ and took $400.07$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3052/3333$ battles [this is $91.57\\%$ and took $680.85$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3007/3333$ battles [this is $90.22\\%$ and took $825.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2997/3333$ battles [this is $89.92\\%$ and took $720$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8c0545",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/dOEC4ay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4dc00e7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d1d0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21103a04",
   "metadata": {},
   "source": [
    "### MC Control - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://imgur.com/tulLv9l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5f3f1d90",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d048>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ebf788c",
   "metadata": {},
   "source": [
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://imgur.com/198LWmn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e6b7bcc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d208>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "650974c7",
   "metadata": {},
   "source": [
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://imgur.com/XkizKnQ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "318d9560",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d2e8>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2131d4de",
   "metadata": {},
   "source": [
    "\n",
    "Our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://imgur.com/Qn6nzV4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "44208ef1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d2b0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6f8143",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://imgur.com/w4svLI6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c9f849cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d198>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f446f43",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://imgur.com/EPkCRgX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e2840ac4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d438>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b0fd8c",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "52a29436",
   "metadata": {},
   "source": [
    "# **Monte Carlo Control - Function Approximation**\n",
    "\n",
    "We implemented the Function Approximation with Monte Carlo Control First Visit. \n",
    "\n",
    "* We defined an array $w$ to store the weights of our approximator and a defaultDict $N(s,a)$ to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly and $N(s,a)$ is initialized to zero;\n",
    " * $N(s,a)$ has the keys as the states **s**. The value of a key is an array of the len of action space (len: 9) and Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s,a]$;\n",
    "\n",
    "* In the first move of a turn, we choose a random action;\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s,:])$. We tested with 3 values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$};\n",
    "\n",
    "* We use the following functions to define our $\\epsilon$-greedy policy and to update the weights $w$, equivalent to: $x(s, a)$ and $\\hat{q}(s, a, w)$, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d37f5e0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature vector\n",
    "def x(state, action):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.append(state, action)\n",
    "\n",
    "def q_approx_fn(state, action, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.dot(x(state, action), w)\n",
    "        \n",
    "def epsilon_greedy_policy(self, w, n0, N):\n",
    "    # epsilon-greedy policy\n",
    "    def policy_fn(state):\n",
    "        epsilon = n0 / (n0 + np.sum(N[str(state)]))\n",
    "        # let's get the greedy action. Ties must be broken arbitrarily\n",
    "        q_approx = np.array([q_approx_fn(state, action, w) for action in range(9)])\n",
    "        greedy_action = np.argmax(q_approx)\n",
    "        action_pick_probability = np.full(9, epsilon / 9)\n",
    "        action_pick_probability[greedy_action] += 1 - epsilon\n",
    "        return action_pick_probability\n",
    "\n",
    "    return policy_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1dae2f",
   "metadata": {},
   "source": [
    "* In each turn, we collect our observations to define our state with the embed battle function, getting the string of 12 battle elements concatenated;\n",
    " \n",
    "* With the state, we select an action using our policy and send it to Showdown;\n",
    "\n",
    "* We store the state and selected action to append to an episode array in the next turn with the computed reward;\n",
    "\n",
    "* At the end of the battle, $w$ and $N$ are updated for the first time of each state seen in that battle:\n",
    " * $N(s,a)$ is increased by 1 for each action performed in a particular state;\n",
    " * The value of $w$ is summed to $\\alpha*\\delta*x(s,a)$;\n",
    " * $\\delta$ is calculated by $G(t) -$ q_approx_fn$(s, a, w)$;\n",
    " * The return $G(t)$ is given by $ \\sum \\limits _{i=0} ^{T-1} \\gamma^iR(t)$. We used $\\gamma = 0,75$;\n",
    " * Our policy are then updated with the new values of $w$ and $N(s,a)$;\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9defe44c",
   "metadata": {},
   "outputs": [],
   "source": [
    "rewards = [reward for state,action,reward in self.episode]\n",
    "states = [state for state,action,reward in self.episode]\n",
    "actions = [action for state,action,reward in self.episode]\n",
    "t_array = range(len(self.episode)+1)\n",
    "\n",
    "# Computing w and N\n",
    "for idx, state in enumerate(states):\n",
    "    action = actions[idx]\n",
    "    if state not in self.visited_states:\n",
    "        self.N[str(state)][action] += 1\n",
    "        returnGt = sum([reward*pow(self.gamma, t) for reward, t in zip(rewards[idx:], t_array[:(-idx-1)])]) \n",
    "\n",
    "        # step-size: 1./N[state][action]\n",
    "        delta = \\\n",
    "            returnGt - self.q_approx(state, action, self.w)\n",
    "            self.w += (1/self.N[str(state)][action])* delta * self.x(state, action)\n",
    "            \n",
    "        self.visited_states.append(state)\n",
    "                \n",
    "self.visited_states = []\n",
    "self.episode = []\n",
    "            \n",
    "# Define new policy with updated w and N\n",
    "self.policy = self.update_epsilon_greedy_policy(self.w, self.n0, self.N)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae5ff3a2",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $w$ and $N(s,a)$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1772841f",
   "metadata": {},
   "source": [
    "\n",
    "## MC_Control Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-93/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### MC_Control  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control  Function Approximation - Stochastic - Rewards (per step) - Training ](https://imgur.com/tcq9QpH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d9c06b1f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d400>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1149a520",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Stochastic - Win percentages (per step) - Training](https://imgur.com/3ZALMFb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "af4e3db4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d470>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "190cfb5b",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2771/3333$ battles [this is $83.14\\%$ and took $521.1$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2796/3333$ battles [this is $83.89\\%$ and took $579.7$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2778/3333$ battles [this is $83.35\\%$ and took $807.72$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $854.3$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3303/3333$ battles [this is $99.1\\%$ and took $765.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.2\\%$ and took $654.11$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deec597d",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/9MHEeI8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "6acbb0e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d6d8>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "460aad86",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "39ad6d1a",
   "metadata": {},
   "source": [
    "## MC_Control Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-116/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### MC_Control Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Rewards (per step) - Training ](https://imgur.com/iGJkPOZ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "70d732aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d780>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beb50ff7",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Win percentages (per step) - Training](https://imgur.com/VRjQZiR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "bca259c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d630>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dc68da2",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2022/3333$ battles [this is $60.67\\%$ and took $456.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1982/3333$ battles [this is $59.47\\%$ and took $508.82$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2000/3333$ battles [this is $60\\%$ and took $561.88$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $532.47$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.19\\%$ and took $723.62$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3310/3333$ battles [this is $99.31\\%$ and took $539.96$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f957e6",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/gIZQgpz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8774c347",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d588>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6086a975",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "156eb28b",
   "metadata": {},
   "source": [
    "# **QLearning**\n",
    "\n",
    "* We have defined 2 defaultDicts, $Q(s,a)$ and $N(s,a)$, to store the value function and the number of occurences of action **a** in state **s**, respectively. Both of them initialized to zero. Each dict has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space);\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've tested using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$}.\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6ffaafd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[state]))\n",
    "    greedy_action = np.random.choice(np.where(self.Q[state] == self.Q[state].max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb85aca5",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionaries $Q(s,a)$ and $N(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=1/N(s, a)$;\n",
    " - calculate $Q(s, a)$ as: $Q(s, a) = Q(s, a) + \\alpha * (R + \\gamma * \\max_{a'}Q(s', a') - Q(s, a))$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec7f0369",
   "metadata": {},
   "outputs": [],
   "source": [
    "# observe R, S'\n",
    "reward = self.compute_reward(battle)\n",
    "next_state = self.embed_battle(battle)\n",
    "# Q-learning\n",
    "self.N[self.state][self.action] += 1\n",
    "alpha = 1.0 / self.N[self.state][self.action]\n",
    "self.Q[self.state][self.action] += \\\n",
    "    alpha * (reward + self.gamma * np.max(self.Q[next_state]) - self.Q[self.state][self.action])\n",
    "# S <- S'\n",
    "self.state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85ad6748",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $N$ in json files for future use.\n",
    "\n",
    "* We validated our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d5590d",
   "metadata": {},
   "source": [
    "## Q-Learning - Stochastic - Results\n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-95/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### Q-Learning - Stochastic - Rewards (per step) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Rewards (per step) - Training](https://imgur.com/yRILJy4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "dfd410c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d748>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de38b557",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Win percentages (per step) - Training](https://imgur.com/Ow5RxLA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2c09d9db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d6a0>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f71bd198",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1510/3333$ battles [this is $45.3\\%$ and took $845.64$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1156/3333$ battles [this is $34.68\\%$ and took $1038.49$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1503/3333$ battles [this is $45.09\\%$ and took $748.07$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3123/3333$ battles [this is $93.7\\%$ and took $957.94$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2859/3333$ battles [this is $85.78\\%$ and took $912.79$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3021/3333$ battles [this is $90.64\\%$ and took $850.87$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0abbfbb",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/2X78uWu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0296ab39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d4a8>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e159772d",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://imgur.com/LOURdLM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "22cc6a25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d908>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "897489a5",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://imgur.com/b4RpLak)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "1926236d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d978>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1a4ccf9",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://imgur.com/JC8EvZr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "5f7d9e39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4db00>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c91379b5",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://imgur.com/1xTGfvm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c9b3e885",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4dba8>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "880d6814",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://imgur.com/2aSqIxU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8552db4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4dc18>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ee36a9",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://imgur.com/i4KX0jY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c14b42cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4deb8>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a27e70",
   "metadata": {},
   "source": [
    "## Q-Learning - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-119/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "\n",
    "### Q-Learning - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning - Deterministic - Rewards (per step) - Training ](https://imgur.com/ajAOTCW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "957a9af8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearning_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c4d518>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearning_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc341c7",
   "metadata": {},
   "source": [
    "### Q-Learning  - Deterministic - Win percentages (per step) - Training\n",
    "[Q-Learning  - Deterministic - Win percentages (per step) - Training](https://imgur.com/5LPy1Po)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "61064d92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearning_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a080>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearning_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23feb3d9",
   "metadata": {},
   "source": [
    "### Q-Learning  - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1446/3333$ battles [this is $43.38\\%$ and took $572.78$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1077/3333$ battles [this is $32.31\\%$ and took $444.55$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1461/3333$ battles [this is $43.83\\%$ and took $582.1$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2801/3333$ battles [this is $84.04\\%$ and took $698.99$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2800/3333$ battles [this is $84.01\\%$ and took $794.75$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2934/3333$ battles [this is $88.03\\%$ and took $601.89$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a90c841",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning - Deterministic - Memory and CPU usage  (absolute time) - Training](https://imgur.com/jzDJY4d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "12097210",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/QLearningMemory _ CPU usage_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a240>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/QLearningMemory _ CPU usage_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa822851",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://imgur.com/ZtAZeiF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "51643bf5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a048>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94d46668",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://imgur.com/gs2WiUa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5a1221a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a128>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be75dc8e",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://imgur.com/QH8auBK)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "6f7ea922",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a198>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d42b9299",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://imgur.com/V0JTDkF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "2d2e74f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a160>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7af15c57",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://imgur.com/gG8erzf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "c325d81d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a320>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2851f35",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://imgur.com/h0Cnty2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "86291580",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a438>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5869604",
   "metadata": {},
   "source": [
    "# **Q-Learning - Function Approximation**\n",
    "\n",
    "* To implement function approximation for Q-Learning, we have first defined array $w$, to store the weights of our linear approximator, and defaultDict $N(s,a)$, to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly and $N(s,a)$ is initialized to zero. $N(s,a)$ has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space).\n",
    "\n",
    "* We used a step-size of $\\alpha = \\alpha_0 / N[s, a]$, where $\\alpha_0 = 0.01$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've tested using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$}.\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4717bf48",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state, w):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[str(state)]))\n",
    "    # let's get the greedy action. Ties must be broken arbitrarily\n",
    "    q_approx = np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)])\n",
    "    greedy_action = np.random.choice(np.where(q_approx == q_approx.max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "982e6cdd",
   "metadata": {},
   "source": [
    "For the function approximation part, we defined the following helpers functions, equivalent to: $x(s, a)$, $\\hat{q}(s, a, w)$ and $\\max_{a}\\hat{q}(s, a, w)$, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c798fc3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def x(state, action):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.append(state, action)\n",
    "\n",
    "def q_approx(self, state, action, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.dot(self.x(state, action), w)\n",
    "\n",
    "def max_q_approx(self, state, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return max(np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6045cc5",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s, w)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action **a**;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha = \\alpha_0 / N[s, a]$, where $\\alpha_0 = 0.01$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma * \\max_{a'}\\hat{q}(s, a', w) - \\hat{q}(s, a, w)$;\n",
    " - then we update $w$ as: $w = w + \\alpha * \\delta * x(s, a)$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s, w)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8b66713",
   "metadata": {},
   "outputs": [],
   "source": [
    "if self.state is not None:\n",
    "    # observe R, S'\n",
    "    reward = self.compute_reward(battle)\n",
    "    next_state = self.embed_battle(battle)\n",
    "    # Q-learning\n",
    "    self.N[str(self.state)][self.action] += 1\n",
    "    alpha = self.alpha0 / self.N[str(self.state)][self.action]\n",
    "    delta = \\\n",
    "        reward + self.gamma * self.max_q_approx(next_state, self.w) - self.q_approx(self.state, self.action, self.w)\n",
    "    self.w += alpha * delta * self.x(self.state, self.action)\n",
    "    # S <- S'\n",
    "    self.state = next_state\n",
    "else:\n",
    "    # S first initialization\n",
    "    self.state = self.embed_battle(battle)\n",
    "\n",
    "# Choose A from S using epsilon-greedy policy\n",
    "self.action = self.pi(self.state, self.w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abea4ad5",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $w$ in json files for future use.\n",
    "\n",
    "* We validated our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3adb12e1",
   "metadata": {},
   "source": [
    "\n",
    "## Q-Learning  Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-126/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### Q-Learning  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Rewards (per step) - Training ](https://imgur.com/GX6fUrw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "5072a5c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a4a8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718c5c40",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Win percentages (per step) - Training](https://imgur.com/qps2Tck)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "daf73328",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a630>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "343cc4d6",
   "metadata": {},
   "source": [
    "\n",
    "### Q-Learning Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2799/3333$ battles [this is $83.98\\%$ and took $934.83$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2812/3333$ battles [this is $84.37\\%$ and took $664.67$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2772/3333$ battles [this is $83.17\\%$ and took $869.2$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $854.3$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3303/3333$ battles [this is $99.1\\%$ and took $765.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.2\\%$ and took $654.11$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd32c021",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/TD3splF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "be3a665f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a668>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c4e7f4",
   "metadata": {},
   "source": [
    "## Q-Learning Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-127/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### Q-Learning Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Rewards (per step) - Training ](https://imgur.com/W65giAJ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4ab99045",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a780>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "362c7a92",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Win percentages (per step) - Training](https://imgur.com/VSDvD36)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "27fd2a49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a710>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02aae2f2",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1925/3333$ battles [this is $57.76\\%$ and took $701.92$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1973/3333$ battles [this is $59.20\\%$ and took $895.72$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1974/3333$ battles [this is $59.23\\%$ and took $508.06$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $700.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3308/3333$ battles [this is $99.25\\%$ and took $914.08$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3313/3333$ battles [this is $99.40\\%$ and took $867.76$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddaaa2c2",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/Pj18r8j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a92d67dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a550>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e858f09",
   "metadata": {},
   "source": [
    "# **SARSA($\\lambda$)**\n",
    "\n",
    "* We defined 3 defaultDicts, $Q(s,a)$, $N(s,a)$ and $E(s,a)$, to store the value function, the number of occurences, and the eligibility trace of action **a** in state **s**, respectively. Both of them initialized to zero;\n",
    " * each dict has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space);\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've trained using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$} and 6 different values for $\\lambda: $ {$0, 0.2, 0.4, 0.6, 0.8, 1$}.\n",
    "\n",
    "* We've validated using the best results obtained with the 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$;\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf8f1457",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[state]))\n",
    "    # let's get the greedy action. Ties must be broken arbitrarily\n",
    "    greedy_action = np.random.choice(np.where(self.Q[state] == self.Q[state].max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "460055cc",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionaries $Q(s,a)$, $N(s,a)$ and $E(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=1/N(s, a)$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma * Q(s', a') - Q(s, a)$;\n",
    " - calculate $E(s, a)$ as: $E(s, a) = E(s, a) + 1$;\n",
    " - calculate $Q(s, a)$ as: $Q(s, a) = Q(s, a) + \\sum \\limits _{s=s_i} ^{s_n} \\alpha * \\delta * E(s, a)$ for each $s \\in S$;\n",
    " - update $E(s, a)$ as: $E(s, a) = E(s, a) * $ $\\prod \\limits _{s=s_i} ^{s_n} \\lambda * \\gamma$ for each $s \\in S$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$.\n",
    " - Our policy are then updated with the new values of $Q(s,a)$ and $N(s,a)$;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d916673",
   "metadata": {},
   "outputs": [],
   "source": [
    "if self.state is not None:\n",
    "    # observe R, next_state and next_action\n",
    "    reward = self.compute_reward(battle)\n",
    "    next_state = self.embed_battle(battle)\n",
    "    next_action = self.choose_action(next_state)\n",
    "            \n",
    "    #alpha\n",
    "    self.N[self.state][self.action] += 1\n",
    "    alpha = 1.0 / self.N[self.state][self.action]\n",
    "            \n",
    "    ########################  Calculate Lambda SARSA\n",
    "    delta = reward + self.gamma*self.Q[next_state][next_action] - self.Q[self.state][self.action]\n",
    "    self.E[self.state][self.action] += 1\n",
    "    for s, _ in self.Q.items():\n",
    "        self.Q[s][:] += alpha * delta * self.E[s][:]\n",
    "        self.E[s][:] *= self.lambda_ * self.gamma\n",
    "            \n",
    "    # S <- S'  A <- A' \n",
    "    self.state = next_state\n",
    "    self.action = next_action\n",
    "    # Update the policy\n",
    "    self.policy = self.update_epsilon_greedy_policy(self.Q, self.n0, self.N)\n",
    "            \n",
    "else:\n",
    "    # S first initialization\n",
    "    self.state = self.embed_battle(battle)\n",
    "    # Choose action\n",
    "    self.action = self.choose_action(self.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fff42f48",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $Q(s,a)$ and $N(s,a)$ in json files for future use.\n",
    "\n",
    "* We validated our solution with the best results against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d371bb0",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Stochastic - Results\n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/mauricioplopes/poke-env/e/POK-41/dashboard/Compare-results-de35633f-644c-4bdd-8688-573933ef0796.\n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Rewards (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Rewards (per step) - Training](https://imgur.com/2egQxZ1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "73d9e7e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsa_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a6a0>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7db3407e",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Win percentages (per step) - Training](https://imgur.com/FkTyPDr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ca99e9f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_sarsa_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a748>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4987f2fd",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1872/3333$ battles [this is $56.17\\%$ and took $468.37$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0$ won $1804/3333$ battles [this is $54.13\\%$ and took $481.42$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0$ won $1802/3333$ battles [this is $54.07\\%$ and took $498.52$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $3151/3333$ battles [this is $94.54\\%$ and took $541.27$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0$ won $2989/3333$ battles [this is $89.68\\%$ and took $664.85$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0$ won $2953/3333$ battles [this is $88.6\\%$ and took $693.19$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6232837",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/YpqqhE2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "ff3bcf06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory sarsa_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a8d0>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd382ffc",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "We've ploted just the best results obtained with SARSA($\\lambda$).\n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75 - lambda:0](https://imgur.com/kJELe1s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "d1522bd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.jsonindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a400>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9659b638",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75 - lambda:0](https://imgur.com/Y29u7H4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "fbc78c82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.jsonindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5ab38>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12d1cf53",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75 - lambda:0.2](https://imgur.com/zUzalm4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "520f38a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.jsonindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a470>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a394885",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75 - lambda:0](https://imgur.com/Y48C3wA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "d022451d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5a978>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74b5309a",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75 - lambda:0](https://imgur.com/Csx9jgc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "390c5cf8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5ac50>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3fcd08",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75 - lambda:0.2](https://imgur.com/P7UCGZC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "333ff82c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5ad68>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79567b78",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-123/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Rewards (per step) - Training ](https://imgur.com/KrLGZZU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "9e6d02a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsa_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5aac8>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abf94b90",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Deterministic - Win percentages (per step) - Training\n",
    "[SARSA($\\lambda$)  - Deterministic - Win percentages (per step) - Training](https://imgur.com/0oASBKv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "360d2130",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_sarsa_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e080>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a356a48",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1739/3333$ battles [this is $52.18\\%$ and took $521.7$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1804/3333$ battles [this is $54.13\\%$ and took $515.35$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1749/3333$ battles [this is $52.48\\%$ and took $577.84$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $2972/3333$ battles [this is $89.17\\%$ and took $690.43$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $2967/3333$ battles [this is $89.02\\%$ and took $669$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0.2$  won $3008/3333$ battles [this is $90.25\\%$ and took $692.09$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c787b4",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Memory and CPU usage  (absolute time) - Training](https://imgur.com/EI6NYaj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "68d1c859",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory sarsa_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e208>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95986dc8",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/rlYy5yV)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "274c523b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e048>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf41a07",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75 - lambda 0.2](https://imgur.com/S8MmNW0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "f13c806a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e128>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86625bed",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75 - lambda: 0.2](https://imgur.com/PPt78on)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "2461752f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.01_0.75_0.2_44.42.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e2e8>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.01_0.75_0.2_44.42.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebbb51a",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/bUg9vH1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "b46a4781",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e198>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2dd0ad0",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/hGJwUyQ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "1da5973a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e3c8>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9577a2",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75 - lambda: 0.2](https://imgur.com/qd8FLvR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "0e5b3503",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e160>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "155fd1f4",
   "metadata": {},
   "source": [
    "# **SARSA($\\lambda$) - Function Approximation**\n",
    "\n",
    "    \n",
    "We implemented the Function Approximation with SARSA($\\lambda$). \n",
    "\n",
    "* We defined an array $w$ to store the weights of our approximator, an array $e$ to store the eligibility trace and a defaultDict $N(s,a)$ to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly, $e$ and $N(s,a)$ are initialized to zero;\n",
    " * $N(s,a)$ has the keys as the states **s**. The value of a key is an array of the len of action space (len: 9) and Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've trained using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$} and 6 different values for $\\lambda: $ {$0, 0.2, 0.4, 0.6, 0.8, 1$}.\n",
    "\n",
    "* We've validated using the best results obtained with the 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$;\n",
    "\n",
    "\n",
    "* We use the following functions to define our $\\epsilon$-greedy policy and to update the weights $w$, equivalent to: $x(s, a)$ and $\\hat{q}(s, a, w)$, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48f85b75",
   "metadata": {},
   "outputs": [],
   "source": [
    "    def x(state, action):\n",
    "        state = np.array(state).astype(float)\n",
    "        return np.append(state, action)\n",
    "    \n",
    "    # q^(S, A, W)\n",
    "    def q_approx(self, state, action, w):\n",
    "        state = np.array(state).astype(float)\n",
    "        return np.dot(self.x(state, action), w)\n",
    "\n",
    "    # epsilon-greedy policy\n",
    "    #Function to choose the next action\n",
    "    def policy_fn(self, state, w):\n",
    "        epsilon = self.n0 / (self.n0 + np.sum(self.N[str(state)]))\n",
    "        # let's get the greedy action. Ties must be broken arbitrarily\n",
    "        q_approx = np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)])\n",
    "        greedy_action = np.argmax(np.where(q_approx == q_approx.max())[0])\n",
    "        action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "        action_pick_probability[greedy_action] += 1 - epsilon\n",
    "        return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe2b87c6",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionary $N(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=0.01/N(s, a)$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma *$ q_approx($s', a', w$) -  q_approx($s, a, w$);\n",
    " - calculate $e$ as: $e = (e + 1)$; \n",
    " - calculate $w$ as: $w = w + \\alpha * \\delta * e$;\n",
    " - update $e$ as: $e = e * \\lambda * \\gamma$; \n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a6be163",
   "metadata": {},
   "outputs": [],
   "source": [
    "        if self.state is not None:\n",
    "            # observe R, next_state and next_action\n",
    "            reward = self.compute_reward(battle)\n",
    "            next_state = self.embed_battle(battle)\n",
    "            next_action = self.policy_fn(next_state, self.w)\n",
    "            delta = reward\n",
    "            #alpha\n",
    "            self.N[str(self.state)][self.action] += 1\n",
    "            alpha = 0.01 / self.N[str(self.state)][self.action]\n",
    "            \n",
    "             #Accumulative\n",
    "            self.e += 1\n",
    "            delta = reward + self.gamma * self.q_approx(next_state, next_action, self.w) - self.q_approx(self.state, self.action, self.w)\n",
    "            \n",
    "            # update theta\n",
    "            self.w += alpha * delta * self.e\n",
    "            \n",
    "            #update e\n",
    "            self.e *= self.gamma * self.lambda_\n",
    "           \n",
    "            # S <- S'\n",
    "            self.state = next_state\n",
    "            # Choose action\n",
    "            self.action = self.policy_fn(self.state, self.w)\n",
    "            \n",
    "        else:\n",
    "            # S first initialization\n",
    "            self.state = self.embed_battle(battle)\n",
    "            self.action = self.policy_fn(self.state, self.w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c78ddb39",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles. At the end, we saved $w$ and $N(s,a)$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "873eb8d6",
   "metadata": {},
   "source": [
    "\n",
    "## SARSA($\\lambda$)  Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-133/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### SARSA($\\lambda$)  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Stochastic - Rewards (per step) - Training ](https://imgur.com/o7zjWeC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "c3d2c68c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsaFA_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e4e0>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0184a124",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[ SARSA($\\lambda$) Function Approximation - Stochastic - Win percentages (per step) - Training](https://imgur.com/KmkqqGW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "57321680",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_sarsaFA_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e400>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4fce9e",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "We've showed just the better results with SARSA($\\lambda$) Function Approximation.\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=1$ won $2063/3333$ battles [this is $83.53\\%$ and took $298.65$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.6$ won $2784/3333$ battles [this is $83.2\\%$ and took $290.17$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=1$ won $2788/3333$ battles [this is $83.65\\%$ and took $307.59$ seconds]\n",
    "\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=1$ won $1914/3333$ battles [this is $57.43\\%$ and took $1288.88$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.6$ won $1912/3333$ battles [this is $57.37\\%$ and took $1272.98$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=1$ won $1904,14/3333$ battles [this is $57.13\\%$ and took $$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42ac31e2",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/7K6JXCs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "be529963",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsaFA_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e5f8>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a87cd4a",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-136/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[ SARSA($\\lambda$) Function Approximation - Deterministic - Rewards (per step) - Training ](https://imgur.com/4Krr5OF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "9b78309e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsaFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e668>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95698e70",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages (per step) - Training](https://imgur.com/vVxO1PA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "54b87ecb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_sarsaFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e550>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d27469d3",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "We've showed just the better results with SARSA($\\lambda$) Function Approximation.\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $2063/3333$ battles [this is $61.9\\%$ and took $513.66$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.4$ won $2027/3333$ battles [this is $60.82\\%$ and took $229.79$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $2030/3333$ battles [this is $60.91\\%$ and took $739$ seconds]\n",
    "\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $3308/3333$ battles [this is $98.95\\%$ and took $222.79$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.4$ won $3302/3333$ battles [this is $98.95\\%$ and took $1296.52$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $3316/3333$ battles [this is $99.49\\%$ and took $222.54$ seconds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "790e6f22",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c1a4b319",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/zsXwrG2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "21d6e3fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory sarsaFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e6a0>"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd94c6c5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6ee81c37",
   "metadata": {},
   "source": [
    "\n",
    "# Limitations\n",
    "\n",
    "The only limitations of our project are in the use of the **deterministic** environment. Given the need to remove **randomness**, our deterministic solutions require the use of modified Pokémon Showdown, without the random parameter in damage calculation, and the Pokémon on both teams with:\n",
    "- Shell Armor or Battle Armor abilities, to prevent critical hits;\n",
    "- Moves with 100% accuracy and no side effects likely to occur.\n",
    "\n",
    "Our **stochastic** solutions can be applied to any case and with any team formation.\n",
    "\n",
    "\n",
    "# Related work\n",
    "\n",
    "Given the stochasticity of the Pokémon game, the environment proves to be a good alternative to validate more robust reinforcement learning systems. However, in recent years, few reinforcement learning works have been willing to use Pokémon battles to validate their models.\n",
    "\n",
    "Regarding tabular methods, the work of [Rill-García, R. 2018](https://ccc.inaoep.mx/~esucar/Clases-mgp/Proyectos/2018/reinforcement-learning-turn%20%281%29.pdf) can be mentioned. The author proposes the use of the Q-Learning algorithm. The author defines as states only the index of his active Pokémon and the opponent's. However, 301 Pokémon are considered to choose from, among the most used in competitive battles. Actions were also defined from 601 moves most used in competitive battles, plus one, referring to an action that cannot be performed. Rewards were set based on the Pokémon that passed out (their or the opponent's), the percentages of damage produced or taken, and based on an \"unable to act\" rating. The author used constant values of $\\alpha = 0.1$ and $\\gamma = 0.8$. The author used 10 different teams for validation and reported an average of $58\\%$ wins against a RandomPlayer (player who selects random actions at each turn).\n",
    "\n",
    "Another tabular method that can be cited is the work of [Kalose, A et. al. 2018](https://web.stanford.edu/class/aa228/reports/2018/final151.pdf). The authors also propose the use of the Q-Learning algorithm, however, it uses two different policies, an $\\epsilon-greedy$ and a Softmax. The authors define as states the Health Points of their Pokémon, the Health Points of the opponent's Pokémon, the types of their Pokémon and the types of opponent's Pokémon. The defined actions are the 4 possible moves of each Pokemon. The authors did not establish how the rewards were defined, or the $\\alpha$ and $\\gamma$ values used. The authors reported an average of $60\\%$ winnings against a RandomPlayer (player who selects random actions each turn) using $\\epsilon-greedy$ policy and $68\\%$ winnings against a RandomPlayer using Softmax policy.\n",
    "\n",
    "\n",
    "# Comparisons\n",
    "\n",
    "One table with performance comparisons in Validation between our methods and the methods proposed by Rill-Garcia and Kalose et. al. are showed below and in [this figure](https://imgur.com/vwUOnDZ).\n",
    "\n",
    "The method with the best performance against both Players (MaxDamagePlayer and RandomPlayer) was Q-Learning Function Approximation in the Stochastic environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "916306b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/comparisons_val.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x23ad7c5e6d8>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/comparisons_val.png\", width=1000, height=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53f2e304",
   "metadata": {},
   "source": [
    "# Video\n",
    "\n",
    "https://youtu.be/uSZE9gnheSI\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "048fd83f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBBAQDxAQEBAQDQ0QEBAQDxAQDg8QDRAQEA4PEA4NEBAODRAODg0ODg0ODRUQEBERExMTDxAWGBYSGBASExIBBQUFCAcIDwkJDxcVEhUVFRUVFxUXFRUVFxcXFRUVFRUVFRUVFRUVFxUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFf/AABEIAWgB4AMBIgACEQEDEQH/xAAdAAABBAMBAQAAAAAAAAAAAAAABQYHCAIDBAEJ/8QAXxAAAgECBAIGBAgICQcLAwIHAQIDBBEABRIhBjEHEyJBUWEUMnGBCCNCUpGhsdQVVGJylcHR8BgkM1OCkpOU1RdDRGOisuEWNFV1g7S1wtLT8XOFs0WkxDV0hKPDJf/EABsBAAEFAQEAAAAAAAAAAAAAAAABAgMEBQYH/8QANxEAAQQBAgQEBAUDBQEBAQAAAQACAxEEEiEFMUFREyJhcQaBkaEUMrHB8ELR4RUjUnLxYpIH/9oADAMBAAIRAxEAPwCmWDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDE2/wAGev8A52k/tJ/u2D+DPX/ztJ/aT/dsS+C/sqP+p43/ADChLBidsv8Agt5g7aRNRg2vvJPbu8KU+OMJPgu5iBcvSgeJee3/AHXDfDddUn/j4K1ahSgzBidKL4L9e7KomowWIAvLPYEmwvalPP8AXhbqfgcZmpsZ6D3S1P3PDXNLealhyI5RbDarhgxZCl+BtmjHaeg98tTb/ueOn+BVmv4xl/8AbVP3LCKZVmwYspU/AwzRRcz0BHlNU/csczfA9zMc56Ef9rU/c8JaFXTBifs8+CdmcUTydZSShFLFI5J2kIG7aVNKupgLtpBubWAJIBj5+iiqsGBidCLh0dmBHO4ATWdvBSfLCFwCglyYoiA9wF8rTCwYlXhToRmqtoqygL/zbS1CSD+g1IG+i+HfTfBIzFvVqKA25jrai49oNHce8YgbmQudoDhfbkfoVa8N2nVW3fmFXvBixEvwQsyH+fof7Wo+541fwScy/n6H+1qPumLQFqAytHMqvmDFgz8EnMv5+h/taj7pjwfBKzL+fof7Wo+6YXSUnjs7qvuDFj8v+Bvmji4noAAbby1P6qM43VHwMM0UXNRQe6ap+5YadlICCLCrVgxZL+Bpmn8/Qf21T9zxrf4HeZDY1OXA+BnqAf8AueGeI3unUVXHBixn8D7MfxrLf7xP9zxt/gb5l+NZb/eKj7nhQ8FFKt2DFkf4G+ZfjWW/3io+54P4G+ZfjWW/3io+54XUEirdgxZH+BvmX41lv94qPueD+BvmX41lv94qPueDUEKt2DFkf4G+ZfjWW/3io+54P4G+ZfjWW/3io+54NQQq3YMWR/gb5l+NZb/eKj7ng/gb5l+NZb/eKj7ng1BCrdgxZH+BvmX41lv94qPueD+BvmX41lv94qPueDUEKt2DFkf4G+ZfjWW/3io+54P4G+ZfjWW/3io+54NQQq3YMWR/gb5l+NZb/eKj7ng/gb5l+NZb/eKj7ng1BCrdgxZH+BvmX41lv94qPueNbfA8zH8ay3+8T/c8IXgJaVcsGLGfwPsx/Gst/vE/3PHTF8C/ND/pGX/21T9ywB7Siiq1YMWXk+Bbmg/0jL/7ap+5Y1fwNM0/n6D+2qfueAvA5oAJVbcGLJH4Gmafz9B/bVP3PHifA1zT+foB7Zqj7nhPEb3RRVbsGLIt8DXNPxig/tqn7njSfgfZn/P0P9rUfc8KHg8kh2VdMGLI0nwNM0a9p6DbxmqfueMqv4GeZqLmoy/+2qbn/wDZYcktVswYsN/BFzL+fof7Wo+6Y8PwRsy/n6H+1qPumBNEjTyKr1gxYT+CPmX8/Q/2tR90xsj+CFmR/wA/Q/2tR9zwlhOsKu+DFiW+CBmQ/wA/Q/2tR9zxq/gj5l/P0P8Aa1H3TBYRYVe8GLBn4JOZbDrqIk8gJai5/wD2mNlV8EDNFFzLRAf/AFpjb22pdsFosKvGDE+/wUcx/nqL+1qPumNkHwSsyY2WWjY+UlQf/wCEwWk1BWgx7fGvVgvjZXmicvCNP6zkc7BT5X7XuuB9GFWipNIK31ISSARyB5r5i9+7vxzcPSXiX3j6GOO8tjPkcS4rq8SJjYm+36pvZvQ9WyunLUCBz0sNwPYbXHsI8MSflFYsiK68mF/fyI9oII92G3lsIclG5MCPO/MEeDAgEHuOMoeGpkGiOpKxjkNAuLm53BF9zfuwjnB4APMKzixOgeXRttp6Ctj806o4gL2Fr88bMcOU0rotnkMp8SAD7Nu7HdiBbDTYsikY482tp5XJ2HtOOzGLoDgSpHloTpBF9XeP38MV96W+EBTyGeIAU0z9tRyimY8wOQinb2aZT39aNNm7YZ/FOVxzLLDIA0cgZGBHcf1g7jzAwwiln8Qwm5MRYfl7qo+fcNxTbsul+YddnBHI3HP7fMY0ZB0hVmXuqzM1TT9xLXlVRsTG7c7d6Pz2ubEHClnuQ1VJNJEzCXQeyH7JeI36uRJATfUAVOsGzKw7sJddmMbqUqI2iB+eLpfxWRbqCPE2xVycSLIbpkF/qPUHmFyWDn5XD5dLXbDmOf2KsxwTxxDWRBo3Vr8iNu18xlO8cndY8+7mML18UUSefL5xJA91IuCN45U+awBKsBy8VO45i9puiPpJjrYgfVkXZ1J7SHuv4qe5vpxTgyZMFwjyHamE01/Udg/9nfVdo10fEGeJCKcBu3v6t/cdFITnGF8ZNjwLjo1lm7T6oIlVQF5d3n5+/njgzSU3t3D6/PG/JKrWgPeNj7R+0b405ue0PZ+vFR62mkaQQsrYjTgngiiqanNZKmjpaqQZgEDz00MrhBlmXkIGkRmChmZtN7XYnvOJJRr4a3RN/L5t/wBZD/wvLcVYPzFTuOyYvF34Hp6mamXIBWSU8KTymlyzL3VUk16dpJI5GY9W3ZVCdtr4cvDvDWQzwQ1EdFlginRHjL0dIjESHSg0tGCGL3jt84EC5wnZv0ayVObVs0klVTUstLSxI9NUCHrmQzdbG+m8lkDrY9n1jYnu5KLolRM6p5FpU/B1LlCQUzuEkEVXHWmSIqsjNJ16Rs0glI5sTq1E4tKNPCn6PMlaRolocraVBd4xSUhkUeLII9SjfvGEvKOE8lenSolyygo0dmULU0lAjBg7IFJTrIiW0FlCyMbEXsbgR9wXwDUQjLUlpVpJMtqZ6mszRpqcx1ERScS6ZBKap/TBIjy9ekYQIQSbLhn8AcG1E+X5VUwpLUQRJm0TRwLl7yq09fKUlVMzR6ZopEQxOy/GBSLalLgiFYPM+j7JYgplocriDmyGSlo0DHuVdUY1HyGOGl4TyNqqSjWgy41MUaSyJ6BTdlXLBNzFYsdDEhblRYm1xdgUnR/U0rU5fL2zWAZOKJad6mlkennErOYZHn9GiaCWORIWqIUJAgAKEabvDgrhGSHN3qGokiimy+kjWWEwvFTzQiUTQ6mMdQRoMUSyLDZ1Vb6QtlVC08RUGQQVlPQtl9A9XUvpWOOgpWMY6tpA8x0ARIwQhQbu/NVIVirsh6L8oJIGXZcSpswFFSkqSLgECLYkb745uk3hySWqyySKPUsNcZqhhpGlPRJ4g7XIL2Z0TbUQPIYcfDkAE1URS+jlpELTfF/xoiJQJewS/wAWPivjAD2dtrHAhQlm2e5HE1SWyENT0dQ1PU1SZXl708Tpo1uQshnMarIjkiEkKeXMYkmTgHJAyIaHK1eUXiU0lGHkB5FFMepx7AcMnhroZ9IkzP01quOnqMymlSnjqjHTVEBSHRJIkLajrZWUgsrWRdhYHCb0i9H9Sz5rEtAKtswNP6FWdZTrHSKkUcYWXrJFqIBRujVCejxy69W1muMCE8cp4dySSesg/BlDG1E8KSvJQ0axM08Kyp1baSTZWAOoJ2thfnjqPBmT+l+iDK6AydT1zN6HRBVGrSqaSvWs7bv2UKqouzLqQNG2b9GNcM2mrzG1XBDWUEqUzGLRUhaFaaetUNIB6VSOetiEugAq+k3YHGeQ9GFYuYhpY6llXNpa5amOXLkpupbWyFnMD5o8jIRRvSkrHp3EgRVwIUoPwPkYJBo8pBCGQg01FcRqSGkPY2RSCC3IEHfbGdTwFkixiVqLKlhNtMrUtGIjq9WzmPSdXdY74ivgrohkUZOZqKPXBXZjPVllgYqkrVBpnc6jrufRioXUVKpcDRtnwl0fVVMuXySUBq4KV81RqNGptcIqqppKapiSaZKdx1Q6or1isiy3APaAQFClSr6O8lRdT0OVopUuGakpFXQACXuYwNIBBLchceOCp6OcnETTDLcveMIZLpRUjBlClrqer0m4G29j44jXgvosqFmyY1FMhhpzm0skWqOSKjFUyPSU25tJ1QugMasisuxsFYvToW4VnpsmNLJF1Uw9OCxXQ2WWpqHhAKsUAZJFIF9r723wHkhMvIs6ySQUrvkIpaeteKOnqJsry/qHeb+RUmCSaROtOwZkCjvIG+HjwFw1ktb6T1WVUa+i1k9FJ1mX0Y1SwaQ7poV7xHWNJbSxsbqO+O+F+iKqplyadkqK0U4hFXl89Y0kVLKEISvpUecQLJSsSvVjWulh1aqw1h1dFJraGTMY3y2qmSpzasqopYpKLqupndBGxElYkoNkLkaL2I77gOSJT6Z+jLLY8szCSPL6GORKGrdHSjp1dHWmkKurLEGVlYBgwIIIBGJUylLRp+at/wCqMNfp2/8A5Tmf/V9Z/wB1lwvUs10TuGlfsGIpHBu5TgLW+plv7MaRgJx4Bik5xJtSjZenGM0oGNM9UBy3P1Y4me+HtZe55JjpK2C2TTX/AGY1k446jMo12LC/hufsBx35HKsjCzK1tzY7+VxzG578LHlQudoY9pPYEWoyx/MgpSyQHfbbbnhJzefU58ASAPZsfpOHNNKACTsBhn1k+pi1rXP7+/F0KvkHYALEtjC+MCcZIPcMMcbTWM0hbIkvjolktyxqEwA+z9pxzu9+eEUl0s2bCnw5Tq+q4vaw8t739+ww3aip7h9P7MLXBmYKupWIUk3BOwO1re3b33woQ07pSrI4o72UhmHMHtAeRY7e7CRIoB2vbuuLHC3nqj1tiVG3jcmwJ8huR54bxb9/1YlAUGQ7dc8iKLjv8d9vLwxtrJ9MaIDYEFmt8oliBe3gBy9nhjJsJ8kBHnh7GttVJJX6aTLG/njzV7vbjgqOHFkGuFu/1WHI233NiPoOEVszlgkCSa9BYDlqVVut2NwwK21sCoF7BeZ3yYPiZmRvCAfQmnfop5fgoxt80hB9tv1T1pcydRZWsPCwI5+Y2x3VuZmSMDYNq7QtzHMEdw3G4538ubRpM6ve68iQrAaQwBIuVLMVBA1c+/lthxUERZb2AHdY37vYMTj4jwL/AN12lw5g7/cWFTk+GOKQimt1NI2IP7GiuqkzmQWHhy25+087jxFsPPhji1T2JDpPcxPZ9hPd5X9l+WGPJTH240k408bJxM1twvB9jv8ARUCc3h7h4rSP+3I/NSPm3GUaHSoMp7yCAo8gbG/uGPcg4wWRgrL1bH1d7qT4XsLHw8foBja+Pb4s/hGVST/Wp9erau3+VOCnHuELgjMTJCpO7LdWPiRyPtKkX874XcZzm6TRXVRSCRgeORCDht5olnPnuPfhyYTc8VSNyA3d+vlhpT3BRd0vcOdfB1ii80N2W3Nk/wA7H57DWo+coG2o4gu+LU4q1mUWmR1+a7r/AFWI/VhoXGfEWO1r2yDmdj8k0uK8hZlPVWtfUY+6/e0fgWHNfVPO1wLsrgviB6OoWRbhb2de4r3gjxU38+fjiW74aXH+RK6mQDtDd7d6/O/OXnfwve9hiLIgZNGY3iwRRVXhfE348jTe4OxVqODM8WeJWU3BAI33t4e1Tt9Hjhd1Yq18HPjQxMaZzsDeMnlY7W9ncfC45acWgpn1KGHqn6Qe8HwI/fnijwXLcwnCmPnZ+Un+pvQ+45FdvxCASMGVEPK7mB/S7qP7Je4VlIYjuI3+vfHbmcl2PlthJyjMRH8m9+/v/ZiOcu4hlmq+paulgeoqa2OmVKOmaILSsToaSRCwlaMMwBvq0t4b7Mrd1FikFlWpVikthnZZSZhTz1jU8VDPDVVIqFM1ZUQyr/FKaAoyR5fOuxpi1xIbhhsLYQ5cyYGpX8JVJalligYLl9Kxlmm7MccAWImU9ZeJjYKjI+ohUZh38MRVUsphetq6OpCdaIaijy7U8WoKZUanM8Eio7KrBZC0ZdNar1iaoC08wrbfVOD8MZv+K5b+kqv/AAjB+GM3/Fct/SVX/hGEzOYpop6anfNJevqmkWJRRUZ/k4ZJnZj1VlXTEwBO5YgAGzFU6izCVnKtmNVCmmdkmly+kSCRaZtNQyMYrgRncGRU6xQXj1oCwbUncJ+ycVTWZpIrI1HlbowKsrZjVFWB2KlWyexBGxBxnS1OaooVKPK0RQAqrmNUFUDkABk4AA8BhG4VjqqgkCurIToSRevy6lj6yJ76JFvEbeqdUb6ZUuutE1LdxjhWt/6Tl/ulH/7OHDX1pJstX4Rzf8Uy39JVf+EYPwjm/wCKZb+kqv8AwjDLi4kqCpkNZXLTLM8DVJoMvMKvHUNTOxCFphGs6MpcxWAGo2W7Ydwyqp6zqvwueu06uq9Hoet0/P0dXq0+drYXzeiFu/CGb/imW/pKr/wjB+Ec3/FMt/SVX/hGE+lpqgrEWzZonmClI5IMvEhZ1DCMBVZXcA8o2cHmCRvhPoMwldqgfhZ446Z1ikmkp8vSDrCLlFYjVdLhSWVVJNlLWax5kmycH4Rzf8Uy39JVf+EYPwjm/wCKZb+kqv8AwjGmXKqgFAc3IaSwjBp6EFyQSAgMd2JCsRpvcAnuONMVJMUeQZyDHGSJHENAUjK+sHYJpQr3hiLYPMhdn4Rzf8Uy39JVf+EYPwhm/wCKZb+kqv8AwjHJUUcyoJGznTEwDK5goBGykqAwYx6SCXUXBtdlHeMbIssqD1gGbljFtKBT0JMZte0lo+wbb9q22DzJVv8Awjm/4plv6Sq/8Ixg+a5uP9Fy39JVf+EY5xRzGPrRnPxJBIl6ig6ohbliH6vQQoUk77WPhjlSmmMyQDN2aWWJp0ApaIholZVLgiKxBLgi3MBjyBwh19KRslAZvm/4rlv6Rq/8Ix62bZuP9Fy39JVf+EY4smy+pllqIkzObXTSJFLeiowNTwRTrpPVbjq5kudt7jux0Q5JUNI0QzctMgDPGKehMig8mZBHqUG4sSLHCVJ6Jdln+F83/Fct/SVX/hGD8MZv+K5b+kqv/CMc1Flsz69Gc6+rOmTTBQNoYmwV7RnSxO1jY4xhoZT1ds6B63+StDQHrNtXxdo+32SG7N9jfCVJ6I8q4eOkzappKmmMGWxiop5oC/4Qq20ddE0evT+CRq06tVri9rXGHzE5CqLgAAAt42Fjpvz9uGbNlc4XVLmzwL1jxgSwZeoJSVogQQpX4xluoJ1bgMFa6jdVcNzK4jbNysrAsqNBQh2Uc2CmPUVHeQLDAYy78yS+ydZqgNhv5nGiSYnmf2YR/wDkLV/9JS/3Sj/9nCVwJUza62KaX0g01Z1KSGOONihoqOexWNQtw9Q4vblbwwojATHFyXc8zRIYnlkJEcalmtubDkALgFmJCgEjcjcYqj0g9N1aTI8Mvo6myxoqRuqi+38rG2p7ElnI3tYBVAGJe+FBmRWkVAwAeSzqD2jZQVBA+TYlt7bhfLFPeLsyUgIp1uDey79x5nlfyxUnAkPhkWOvZK3ytsc1YPox6ZVnST0vRDNEFbUurTMGJB0ISziRWAuoLCzAiwBsj8R/CRFOx6mMLIp7Opi722vqjXSimx9Uyn34rvmOdueynxMfgPXY9+ptjf2e++EJoBv54zIfh3EZN42nsQAdh7Kf8W8t0kqxdR8MKsZQDHA1u8wv9JCVag/ViTvg7dMFZmbuJKSNII766mN3RAxF0iEUnWl5G5kLKAi7k7qGqHwLw7FNNGJC3U6wsqpLHFMVa9mSSZDAtjv8YQLKdx6wv/0XZFT01KkdLCaeG5bSzK7szAapGkSWVJC1gNSyMtlAWyqoHQAjkFVkaBuQnXgLYwL4xd7c8ChtZlscU9RfYcvtxpqKi/swUq3IHiQPpNsMkdpaSkBs0ts9Kw5gjwuNjj3L/XS/LWt/ZqF/qwo5jmbK7AHs3tpO67bWsfZ5Yzy+GOY20lCBc6SNBAIvz9W/likzJk5uGymMbboFKGdFld77q4FvYALfQwJ9/nhMbCnmDdYbqVYAbAHtfQbYTpoyOYI9oxdjyozte6qTxuskcl5fHPU1NthzxuJxyw0pdrDf7APE+WJ/FY0anHZVS17tm81HuQZgiL6rl9729U+HNrX87X54a/SJxeqAdYy61N1iQgy2JF2IJHIb3JUG22+2IkzHiOqkWzSyafybJf8AswpIw3Zqbe4I3O9xax8z4nx+vHnuDwyGF4k1HUPkF1s/xhjzlzYxV9Xfsp94f4gjmAZWFx8oePgykAo3ttf6MP2hG10Ysp5gksfcWJsfLFQqWoeJtSkow7x9h5gjyNx5YkDhLpNZLCW47taXt/SX/wBNx+SOeH8U4S3JGuGg7r2P+Vo8P4214DZTt35qwNLV2bTuQSQboV3HmFCm/u9px3VEIPtt7x5efjhp8McWxSAHUG/LU3B/OAOx/fbDvyMG/wAkg3OrbtAjkw5GwA7Q3sPC+OYgfkYU9tJa4fz5haWbjw5MNPAc0/T3H8tJkikc/wDhjDVjfnyFe2ljbd0HJ177bXDra479rb7Y5YJA66kN1+se3HqXAvipmXUeRTX8gehP7H0XmPGvhWTFb40FuZzPdv8AceqdHB2YyRglQHQntJezXA9YE7Xt3HY2HLDyy/imN30ENGxtbWLAnwBuRfu8+6+I6yCtsdG255997cvqt3fXhYq4Awsfce8HuIxvzMGvcKlg5T2xDQbrof0UlBsJGcULFtQ32At3/XtbDZ4fzRw6sxY6iqOpN1vsgdR8knZjbY3PkQ/Riq9lbLdx5xM2xsmHxV16ROYER6i3YSVzHGTcXJZUY7C5AsASACyglhTvjPOq2nndKinSOUsWIMllbUSdaSG0bofFTzuDYggX2zKjDjz7j+/ccMLizhqCqTqqiJZVBNtQ7SN85G5qdu7Y8iCNsRgd1U4hhNnAJFkd+SptHxwUbTUQvCSAQRfdTezgMBqQ22dSwNtr4dGW5lHKt0YOvfbmPJgdx7CBjb06dExo062JjJRlrNG5u0TMbBl5dksQhZbMCV3bUdEIENG2pCVtyZSQR5XG/wCo4Tkudn4ZGTQ8p+ycfFVI1NKHTYXvGe6x5ofNTb6j34tB0IcfRyRxqzjU8asQb9lrWIJIsdLakJ+UFDeNql5txPLLGI5NLAMG1Ws2wIsbdnv7gMWg+CfliGkLMoY2VRcXtfUWtfkTtjF4pjudJE+LZ4Ox9KNg+i6r4fkcyKSKbdtbgd7ABHYqbpKcHy/fwxGGRcOVEnXzQJ1j09RXyRASInWVEWY0s8dNqc3QVUcMtOXI0qrtci4xKES2AHgAN+ewt9OGvP0f0xd3/jKNI7SOIsxzCFC7m7sI4atI1LHc6VAxth7qGpPa1rXEhcVLwFVRFZhGJZoJKOqKq8ca1c/VVi5gq6mIR3eumnj60qhl6sFwup1d+U001RXRVTwSUcNPTTwoszQmaWSpkpndtMEsyJFCtIqgs+p3duyFRWkaeacCRLGxj9Mkkt2VOcZqu/t9OIuBuARuRba+I64bpk0DrXrzpYI00ma16Ru+ykDRXjSvWHRqAN9uW4xRzeJw4leJe/KhfotbE4dLkxl8dUNjvupSzngWu9Pp6lZ6aVPTzNITRyCaGnWhrYYojIcy0yRxipaJVjhS01Q1QVI61H4+MeEauueRWg9DlanrqSWpMyy0s1PUQSRQpBEJWmjJl9GqpWeKBgYGjDShtWNlNwLSsAQ9aQeR/C2a25edcD9XlzGNh6Pqf59b+ls0+/YuNeHCwqBNGilbovyOVJ3c00lDF1EcciSVKTmoqVYlqoaJZRYL2evlKTz6/jEXqku98rpJVZy83Wqxui9WqdWLk6dSm77EC5+bfvxGP+T6n+fW/pbNPv2Pf8n1P8+t/S2affsPtN1BJdP0XypTrNomlqY8wqql6J62VqOphfMZ5Y1EBqvQo5hC8dVESqr16r1tiXZd2VcA1K1t3FS6fhCWtEwbLEpQsjOUXUaR81adIXWiKaxG0SkCZI7RDvTgGm+dWH/7tmv6q/G5eA6TvNd7s3zU/wD8eMNL66J2oFN7IOjedaGRHpl9K/BmV06XaEt19JG3WIr6yFMMulg+oKWsVJtcLcnCcsVR6SKT0lFrKuTqENMsrCeKJIquPrpI4TJFpmi0ySRsEnlIuey/YnR/R/OrR7c2zb7/AI3J0bUh5NWH/wC75r9/xGZgOYKcG9im/kvRjP1U8ZVIJJMvqoKeRSCKRqqrq5o6aMqRIqUkctPGDGFS0K6PVAGHEvCdROkbRUBy/wBHkpGZImy9qmpWnjqlWKFZBPQ9TTPURzwGq0MWRuxTsscgcn+TSl8a39L5r9/wf5M6X51b+l81+/4T8Q31TtBSBwr0eyiWCSWF2QQ5sSKhqJ5I5ayWh6sFKSGGmjedIal3ECyIrPJqlYyXPBmHRhP6JFFFEIHTL8tikCmn1O9JVxzzU15BLE7sglVWlWSBnchyyu93c3RpSfOrf0vmv3/GiTo/ox8qtPszfNfv9sO8cHkCmkUkbhvgWczU0skMxX8JNVTCrfLzMFXKqimSd46CGOmVzOYUAjad7LHIzLYqnf0ecJzU9TTs9N2B+Fo+sVoCKdJszepo7jrQ/VSU50qsKuYywV1jGojKTgKl7jWj25vmv3/Gtej+m+dWn/7tmv6q/DweqYXBKuVSVNNW17eg1NRFU1EEkc0MlCI9K0NLA2pZ66GYFZIXBHVm4AIvcYZ+RcAViVG6Tt1M9fUJJJJlyZfI1SlTo/kKX8KytKahRLFMyIpVnEkvVQh1n/J9T/Orf0tmn37B/k+p/nVv6WzT79hbRrCalFwFWSaleklWFqSGnMcxypI16uspnanjioTp9GEIlKddJKxCyKerJQSvfibgd3avdKdC8s2XtA3xQZlpmhdiCWBTqnEjgNpOq5W5Ivx/5Pqf51b+ls0+/YP8n1P86t/S2affsCTUFjknAMvWSGanVlMedKuoxOP47mss8K21G3XU7KxuLAHS2k3XDMzyLqUkp5oY56maXJzraemMlM6R0MK0zI03pLP10M08Jp45Y3ad7snxhL1/yfU/zq39LZp9+xpk6M6QsHPpZkUEK5zTMy6g8wG9NuAbnYHC2jUFMuIq4Ot6Vmt+QzEX/RmW/r2xy/5P6f59b+ls0+/YY/TGgoqUUlEsxqcwmkt8dNPPJKYUhDdZUSySaurVbEsFQxqdgCcMe6giwVBPT3xX+Ecx0wG8Z+KiAYBZe0QpvexEhuy76SHXkS+GdwrwyZ5UhSIPJLuFaRYQrRKxmDGRlsCqatIu1y2xKDGXEfBzrWJRp8bOSkR0GytKWkDFCdNo0KkaiRZVJ2HKY4Oj2pNTR1ekKXlikml0sAWkUa5JYSNcPXAmCcH1JVdwCGKik+TSLUjIyTf1UH1fCyh5EIZJYmZZYSV66NlYhgbXWQAjYggEWIbfDProdB7JDLzB8ftsR4b+04lj4XuQmHNpnC9Wjx07x9qxcLAkTSJYDcPGysAbgjVycYiOWRiLt2r/ACibn+kdz/W38POeLcar5pHto0E5sjp47g7hJBsFIBDgdqNSbhb2DLcMFKkb8jbroI4b6qHXDUyTwvY2bSIzzvZBdop0PYclu0wPYUacUpyCrCsAwVoyQTqNgCA1mB20m55/Zzxc7oQzGlaH+LdeikC4lUBSVFuzIiLFM6iyltRk0hAwACAKG+eym5Eg00FKz1v0+3b6RjjlmJ5406sF8WKVGytmrCnw8nbBPJbufYu/22wk3w6OFst1o+5XVZbjnYbm1/HYe7FTMdTdPdTQC3Jv1U1ySeZJJ9+5x3ZRVaUmt6xVQPG2uzfUwwuzZLsQkSFR8qRn1se8i1rDzuPIWscNSqi0kj7GDC3tHd7fpOLkENABVsmTwwSuikqbXvhQo84YGxNwe5t/Zz3thNyqDU1uelS5B7wovY+RNvdhKyqcsgYm7Ncse8m9ifq5cgLAbDEsuHFLdhUGZUkYBtPQzRP39Wf9n9n2Y15hOsa6F5mxLeIP77eR88NtZcDS4zhwfzi3HT2Vo8TGk0N+6q/VmMnRHG+q9lYymR2PL1QgUhudlA8j3YWck4AMwYyMYSrFLBQW1AA7nVYqLgW379xh38J8MrAuqwacjmdwtx6o8B84jc7jcbFeoodKhbk25k8yTuzHzYknbbfbHmWRxLSCIj81z0WPvblBmf8AB8sUQm2aLkxBsU7ehbq25VzaxXVz3sN8NiMLvcc/Du8xiTOlPMzPURUUZsA6dYeYDNyuLi6wxEyEX7+4rhmcYcLPT6Wv11PIoaGoVSI3DAW5k6H39Uk3G4JGO44RiGaASTbF24A7dL9Vr4n+0RRonp3SNCzRtqjYgjkVNm7/AN//AJw9cp6VaiMAMoO57QLJfxuAdOrcXsB7MJvRhRiSqjDAEKGcgi4JUdnbyYhvdiU8/wAhbrDPE2mUot1J+JmA30SDv1q2kPzXSDjG4lNAybwZmWe5+y1Mbj2TjXp779vodlzcLdLayHSziN+4SadJ8g/ZF/JrHwvbC3w7mZSzDdTsR3EX2t3cuX/HChk/CkNRD6ioN1MbxghCD2kZCNOn5Suh0lSpA3GGBxTkU9A/WxRFqUj46NWLKLcpUBuVULa5Xsi3aCjcY8UMeSXxQNIcN6u+V8uv85ruMLjsDo2vmcKOzhVHevka9PopeaxAdfVO48v3P0HDropLqpPMqCfoxG/Rpn8U8R0HUDcjexB2upHcy87d978iCX3klYGZYz2G2UHuJtYd23dt/wDGO4+HeKy5Ubsef87O/Mt9fZcd8QcJiwsjxYPyP5dr57e4StTygMpOyhlJJ5AAgk+4C+JBikBAINwdwRyPniOquAqSrDcbEcx/xBGF/o+nsjxk30NdRfcRsAV9wbUvu9mOgeLFqjhylr9B6/qnScNfPR8YbeX2Yc0q3BGGrm9OVa53vvfz7/2+w4gK0pOSjL4ReYxR5bOJOcgWOIfKMrG62/MCtI35KsO/FNCcWh+ENwTXVZR4lhqYYdeiDU8Mt3EepyetKTkaCBbqGUEgB73xWfOp5adyklIkEg+TKk5YX5HTJMUa/MXQj24SiR6LDzIy6S0i11Lbccvs/wCGLa/BBqb0UgHyZE+uJf8AzKw9t8VBmnLbk/UAPoAAHuGLJ/AwzoWnhJ3JUqPEjUfsJH0DFLJIa5jj/wAq+uyu8OB8zeun9N1ZXDbkzmSWRo6cKEQ2knbcA96xr8p/NrjvtYqWcV8Rn1ksE8iK14usaRSjWUaifi3XkzKLKRY8gdjsufxvO/Csa516SfNVaq9L/bddBwjFGQ5wFagPLfL+e+ye8WR98kssp83Kp/UWyj3WxrzjhqOSFYReKJfVEYUWHzQSpKjzQqfO1xjuyerMkaORYsoO3LfkR5EbjyOOvF2HDx3MDg27F2bJ+p3VV2bkMf8Amog9Kqx6DZasvpQiKi30qoUXZmNgNrsxLE+ZJON+McGLwAAoKm55cSSdysr4L4xvgvhU21mGxmJR3gH2bH9n1Y1Xx5fCUlBXUqqe8j2i/wBYxl6Ge4g+w44749VrYaQehTg4dQu0dYPH6j+3A1W3eLfSDjmNS3zj9ONZbDQzuAlMnYlZu9/H3m+Mb4xvgviRMtZXwXxjfBfCpFlfBfGN8F8CFlfBfGN8F8CVZXx6MdEdH3uQg8/W9y88blzBU2jS5+c53+gb/WuFpHLmt1DlJO7dkeHf/wAP35YRaqjhWsFTYSOkQihA5RlmYzPfcFnXQoIuQA4v2zjpqa12BDNse4AAft+vCVPKBsvvP6vZ9uFoJrpHf07e6rJkOSMOJIQd+pEruT33gnde61z6Qht3XxaagReyDzNyB3EXuT4XuQfHv8cM/wD5J6q+Os5BaaWFx3ljJE0TD2L1ysfzMZdKLOIlMZKsGAVlbSQSCrbjuMZdP6Vx44z8qL/ZdfTcLQwXOfOxo/qofVMLpNzDJaqpZcwlIaFwsYj68SHUqagWhQosBCq/aYOxO2jSOs8z74O2UVqE5dKkEoG4WVp4z5vHJIZYyfnAgfkk4jXjXMIY4yZgr3uFQgFmPgveLG122A8cRnwpxs0TASFggtodNpIiBa4IIZk9h1Du1bLiLh2QJGUW0OhW7xrgrcd3lfqJ5jqFIOb/AAfK2nv/ABRplXfrIXEt/MKjdd//AIwcHR7mtRQSq0ZAh1hZ4ZQzwrq7J62IurQyqDqDdkmy3LAWEhcD9OFSij4xK6L8tryW8OtXthvESq7Dy5YcPF1ZDXRdcFEkTBlkjfeopWJsQkyETLA5sV0uE5KygEINfSOa5CbGc021x+e6wzLjiWcmni6ilrTeSkdoiaaoQXDUspDjq5VLIetWwI0tpVWkCJHQlx/NWVD0s6JHVAtdNo9OgAMqrJIWIWxZlUyOLlhdN1a82StGaZusjCUsgdXcHUyrsI2AsNRXYsG7VvUHLEV8ecQyNOtZGTDLGw0yISH7LMUkv/ODWBf5uxuObH3zCsYzWuZon/N3H6+yvZlPD7dYokF0Oo7agOztY3VSN7eFxyuMPaggA2UBQOQAsMQd8Gzp2Fegp6oqlcB2H2VKoAEllAACTqAS0YsCO0m2pI53plxRefElbXIKx4Hg2FtfHLJQIWDFQWFrEjfa9voucdjHEQ8U8QTSVVqWR3GldKx30g27WoEadjzLbC9trY042F3JUsrIbE0Ei9+XVKnSTxdYtBFa9tMj99mFjGvnvue7lz3DMyWvIUrbYXsfO/q+zvvfGWf5Q0Slqi/pEjEqupTte7yuVJHM6VG25J7scdAtl9u+NGCNtbLlc/IkL7dt2HZL9PIxsTa3Pzt7O7HRfCWlbZbd/LHtRW3Fh78PMZJULZ2hvNRp0n8ZCkiuN53BESnkLc5G/JW42+U1hyuRHld0kVEkaRwfF6URZJmt1jsFAZl2KxhiC2wZtwbriPZYnlZmZjJKRqZmJLMQRtcnzsBytttjo4azBVBVyR2r8t7W3HtFvr8scHhfD+PBGA4anA2Se/t6LSkjqMuZuRzTyyKl6oNKCzTFGGpj2iz31Pckm7LePn6rtfc4nLovjvRRxSLcKHiZHW6tGHYR3B2ZTEUxCeU1Qf1fUBCg+NgCx37t9PtB54mTgOclAAV0DZVtZ1FgVI/JO4tzv392NWzyWK/IdpLXc7BvqPT7ptcSdDLdYXo5erRgQ8TSOrAHmscgVtStt2ZLAd7keqs0mbp1QLHq2VkhKMblXLiMISNiQxC35EWPcbSBTz2w2uKeDo29JnF9bRhwnyRJEQ5kHO7OEC8rjXJv29sji2AMxoDzuDdjr6K1HmvcN9+62cMylJQbXLKVNuZXd1PmUF9u/U21yLM7jjj6WhrTG1p6FgHEdh1ihmZH6t7+vHLHIApNrdns2DK7uG6pWeIqQwJNiOWkg2395xXnpPqJDW1Jlvq66TSCbgRliYrbkAGIobee+98c/wDDGF4+S/xOTW12IN7UehC3cdwMdX/Oqm/K+G4jJHXUDjqZGvNGB2HU3DHR/mpo7m9gBfUCASWxIHDWWiolZQSp0Fla2wZStrjwNz3g8sVx6BuJmhqeqLFY5+yB8kSj1GsdhqAMe3MlPmi1kMqQ60kiPVsGGoXIW1xqNvm6bkjla/IjHQY2XHgcRMeT+ZzaY/kCL/qHfar+ys5L5ZIGRDdjXXXX5HtvdJ/cawiyN330n6L/AFEH6cIORuRPGQ2m50N4MpBOg+eoAg+O3fYu3iWk1x7b23+rnhlS0l9jy8RzFuR8iDuD3Y62M2FBltLZQ4ehUmk4b3FEouB3g39mxH0m/Lyxnw9VF49LPd9xqGxPOx2AGoC3vGErMaNkPa3vyPj/AMcQuFLS8XUywm7x1xIlJTy1Em6xre3iTsF95xXDO+KEzWNXnTUoZlAARZIiD6gdAGsykOFZ2U3vuRfE79M+QNVZfUwJ/KOl03A7SMHAudgDp0k9wJxROgr5YWbSWjb1XUjvB9VlYEalN+Yup5WOM/NxZJW+R5aelfvSlxM+LHfUsYc3rYB9qtdHE2WdTM8d9QUjS3IlSAVv+VY2PdcG21sOXoU4p9Eq0e9layt7L3H0EX8yAO/DSq5y93clnJF2PfsAPYABb6McisRuNiNx7fHCuhMsBjed6on17j5qi2cQ5AljG12B6dj8lb3izjOqYbNaAi4eIEIy/n2uPAqWYe3HRwfwrVVFusLU1Ne+11kcdwS/bA29djYcwG5COOjPjXqYQSQGdLKpcaBJcHVpY2PMXHM3t33w7Mn4udnvKxZib6r9oeannt4X5csefyyPbIXZLTJpO1nnX1+y9HhYJY6xyGWL5b7qfYUAAA2AAAHgALAe4YzvhmZbxmoTVKbqLWcWG3LtXIF72F9vPlu2uJenmghuA5lfwQFt/A6NQ+kjHbYfGsbIjDo7/wCtGx6bBchlcKngeRJQ9SQB91J+Z1BVGYAEgXszaV9paxtYb2tvy2vcRLn/AE/U8G0iFpN+wjAtt4kXVf6RH7Im476fp5jphjEa/JaQXI81QHSDbkWZz5d2IkaC+7EsfM7/AL9/mbnvw9oyJ5dZtjR02s+/OvraqTZmNjRaQA9x670PQcr/AEU8Zv8ACjkP8lSqv/1HN/8AZuMI6/CZrf5qI+/7OwPrviIKGnB5DUbkDa5/f2YetD0WV0ya46ZpUte4Kr/VLlA//ZlsXfBB5k/U/wB1lf6nJqoAf/kf2KfdJ8J6b5dOv9Fh+tRhZy34T8dx1lPKvmvVsPo1ofoxEfRv0eNNmEdHUrJT6g7FXRkkKoCzAB171DWIBUm2LR5V0I5Wi2MDSnxeZ7/7BRf9nFV3D2HcOePZ7v7q4ziE7/6Wn3aB+gtcnDfTrQTAXkMJP84pX7dvoY4kXKs1jlUNE6yKeRVgQfHke7DDn6EstPKAp7JGP+/qGNmR9EtPTtrp5Kinba+mSMo1vnIYbN4X5juIxIIp4x5XavR3P6ivuEvjlx8zAPVp/Y/3UiXwXxzUiMBZiGPzgNN/Hs3NvpP7d2Lcbi5tkV6FDtjSzvgvjDBhyS1nfBfGGDAi1nfBfGGDAi1mTj1JiOW3s2P088a8a3lA54EWtwxrmqAPb4fvyxxTVnhsPHv/AOGOdBfzOFpNLuy3zVJPkPDG6lpu8/R+3GgMF8C31D9pxzVVVYXY/wDH9WFTbrmlGqqu4G3ie4DDb4ln61Ci8hupPew3BPgL/UT7tVTWFvZ3C/73ONd8Dow5ukpGZDo3h7eYIP0VbulDhfrWdl1alJunfsTcKOQI3BXke7fnENXTEbHcjv8AEeP79/M4tZ0kZdpcSgbPs1vnjv8A6Q+sHxxDHH2Q/wCdjF1O7Act/lD8lvoB35HbmopHY8phf8l6tLDHxDEbmQ8yN/cc/ooxyvNGibUhKm9+Zsbewgg+YOJS4W6YSh+NW1xYsgCsbix3FlcEbWIW4O+IuzWmAAYew/tI7jtY+eE4NjZjk22XITQAk3se4U8Z1na1B1xtrjKi2/LxFj6p8ftIw1+II0KhBYEkns222G/he9ifIYY/CoOpiG0oFu3I8+XMWvsTe3d54cJUhGIBL8h3tzseXLSxIsLDsg4sg6gscYTo5b1bc/VN2KvemmFmKkMHjYH1WU3UjwIIDAHkfEWJt50OfCnR3WGvHVq1glUANIY81nVAFC6r/HIqqLjUiAF8VErKQabOpu17XuNNvleJNztfbY3BvhPpXKGzcvH7G8r94xFIzR5mrQge2Y6JPl6L62oyuAQQykbEG6lWsbixsQRyPh7ccOVZLDAGMaLGDbUR5Xtck8hc4ol0F9P9Rlg6l0NXRfJiL6ZIbnfqXIYaOZMTDTcghk7Wqymb9L9PW0qmjckSFkmDKVkjsqlomB72EgGpSykBgCe6aB/iGgqWfF+FaZHiwORSPxxnvpE7P8gdhPzFJsT5sSW8rgd18J9JWWFiNvHCeDg1Y2WDSKC4GZxlcXO6pfjkvy3wPJbnthHpUbuuPPkP+OOqGk72N/388Tgk9FVLQOqq7kBHa8dh9v2nCXn20hI23HvNtz+3El0WSx1D5dCNNNO9O8EzaOw00Z1ROSpCs0xZiGLaiqWPaAXEUV0hLG40m5uDzBvuD9mOcY3zLqYcR8c5J5UpO4OkjCxCQhY7qshGwFwO0TY2JBD6m23FziVOGcqlhkR1KS0zrocgkMtr6JQu6lTZQ1rMNRuCEBFeuFc5sere2lgFv5gaVv7QAvuB8byzwvnU8Lp6r08urYALZ1FurjVFCl2YKgjsGLMpDAPcwObpO6w8jGdHI5vU8r5G+3YqaYXx2085+r7P3A92Ghl2bCQGRCWjAUpo3Zx8tGjbcMrDY9nZu4q2O/M5Zgr9ToeVtoxIxWNOzbU2gFmGrcgAHuvsMQvbaz2B7HUdvdI/SfRrSwTVcLGF9JTq1F4zJMRGsqqCBHKhfXqsQQPVvua2MhJuSSWN2JN2JJuWJO5Lbkk7kk3xZ7i6m654Y5ArxIGlkUrdJJLdXGCGJ7Ca5JLNftdWb9k4hafhQvWTQqpVFMjKB3LoLxgE37LFo079m8sXMTHbDH4h5uPzPZakWV4bNI90zJCQwZTpYWIYcwwN1YeYIHvti2PRhxJ1scUw2LqCR3CRSQ6+zUGXzBGKpyx728vtw6+jvi56Z9yepf1wPkNyEqjuZTuQPWXbchbYvxDw450THwnztNt/t+/yWhhcQ8Omu5K/dDMHUMORAP8Aw/VhLzfKQdwPo5/8RhC6E82aajidvWaKGQ+RkiV2Hs1FsPeVARY7g8xjV4fkGaFrzz6+42P3C6EtbK0Jn08LRt5fvY+WFbO31wv84C4tz2O5HfyuNsNviSlqYW1RkyRcgtjIV5mxBBIXuup9tsc2W8dWFpIg2/NWtt32DX38tQHsxeO6qMIjJadguHrT4n6cVp+FHBl+slHtmS9WHjTUYzGb/wAoNBjjkVSCFDo2kglSCDicOljjlIEPo1LNNLIp6kq8PVB9DE9ZG0nXqqMFJKI0VmF3XezG4I6AllpvSq8PWVFTaZ3WR10Le6gdWUY61AJYLp0kKmkLcsI6JrmE8t1WCXJJhEk5icQOWVJCp0MVNms1rGxBHuPhjkootTqoGoswFgbE3NrX3t7SNvdi1vS5VGWJcqoY1Msq2J06aWmhh0MwLhGVXsY4wqAtH1qk6S8ev3M+iw9VRQRdQkcCsZpWUmdpnVFeVRo+MuFYBWdFUEbEIoxXl1hp0i+yIomFw1Gh1UH1mT6HIjDvGqanJXsgC/eFAIA3Nr238DjTRVzIysrtoBF11HRYnnY7KB3+HPuxPlPwGITMGkf0NIzKZZirNqIYTMCgU/yK6WiKpEAUszsZAse9GXDcBgVqhQVl0qzG/wAX1p0J39nY7MN9RG+wI5XJD8cf7wuyB7rqsaMTyasZ1UNRvpXTZPHhbTUwSRNurqVYHmNalTt4gi/kb+GGrS/B1qjCrianEjKG6s9YoFwDpL6D2he3qW2577dvR7TSU1XNSSm8kI0hhe0iMFkikFx8pAG8iSOd8T7R1F44QO8i4v8AJU/qOn6MVOFZBwcmeE8iA4e96fvYVjjmMziEMM557tP0v9lXTo/6AKiWYms/i0ETWIUq0spIBvGw1RiOxA1nUb3Gm4NpS4g6BKB49ESNTuSt5RJNI4UEFwqyTGMO4GnUysFBJC3tiVb4L47ulxbYGAVSYXA3RBQ0h1IjTSbWechytvmqqJGN976Cb232GH/fGN8F8Cla1reQWqqokcozKrNG2qNiO0jFSpZG5qWVmQ2I1KzKbgkHffGN8F8CXZZXwXxjfBfAltZXwXxjfBfAi1lfBfGN8F8CLWV8F8dmT5f1h8FHM/YB5472rKQN1epNfK1+1fw1ePvw4NJ5JdgNyB7pFxrkmA5nHfmscCHQ0rB+dlUsQO6+lGt9WEOoMY3XVIOQLErcgC5K2DaSTtuO/na+F0lRl7QasLKau8PpOORnvz3xgzeVvLf9ZJ+vHl8FJhNrPVjYqNa4Dab2uAdN/C/L3YVeE8n60lm9RTYjvY2vbyFiCfH6cNrj3pM6kOYyI4I9gQoLOb6RZTsLsbKPO5t3ODLBPQbkperWgEucaaBuSSt9TUhfb3Dv/wDjzws8GZjGsU0jgM6EW2BIVtlVb8rsDf3XxC9X02SF4giJVg2LRyxOZVf5kTA7m1/kOtxex5YlWnzpZYkZ/SIw4uY2WEFSDYghOzcEcxseY2O9FmdE51N33r39lr8S4FmcOY2SdunUCQCRY6bi9t/kt78TNIAkiRshNm0rZt9rqbmzLzBw3KiIqxU81JU+0Gx+zCyjQAggygjcdmM7j+kMd2Z1sEu73VuZdYlViLb6rSkH2kbYsSZI6NKwIQXbPeCeiZWa0QkRkbkw+g9ze0HfEGVteiF1IJVWZG0qWDPch1UDYgG+pjZb9nc30znxRntNEpKSmdwL2VLKu1+0+si696rc32NueK6V9Mrl7SHWb7CQqFv6oIU3Av38zuSSd8YHEJoZnCuY5r1H4OxsvHik8QU00Wg9T1q+6YfEUQcmyCIMTYDle/Pc3JB52Fh5YZ0sdvdh7ZyZwrCaMMQTok1dobnT2lN3Uf6xb78xhIo8oM9SkIH8pJ2vJT2nb+imo+62L2OKFLM4i65yaIvoRRHySjwnRfFgEWL9s/m7af6JBU+x74Vc2g1AjkCNvb/8/Thy8Txp6RJoFlRY0sB2QVQCy+QWykeK+WEWtXGkBQC57xwZyOnJMqoZo7qQCL3FxcX8QfMD6B5Y56akaS9lLu7BUVVJZn5kKqgsbJc2HcAMdeaTmR7LuLlV8CflN7NufgL95w/egTLlkqXk5x00eiM25yyk6pAfHQjr7HXlhGAyvoKXIkbiQmQ79v56qJ6kEG1zYd3h4g+Y5eWHX0W8ZGkm1G7RONMijmVFypF9taEki/MFl21XEvdJPRrHVapI7RVR3LEnRIQAAHG+k2AGtB7Q21q+Z5lUkEhjlQxyLzB8O5gRsynuYEg7+BwkkL4HX903Gz4M+MtPUUWn+fdW/wArzJJUEkbB0a9mG48CPIg3BB3BHcRhQy9LtvyG+Kl8CcWy0sgdSxj1DrI9RCSC1jcbjWBya1xZe4EGzvAXEkVSpkiJt6pDCzKwAJU917MDsSLHGri5LZTR5rjuL8Mdhm27tPI/3Tt1YNWNOvBrxpaVztqn8GfyxjsuRY6l3N1YG4ZSCCpB32PPzwpcJ06zgxOQpQDQVA1W5EW7wott4ewYa1Qdjj2mlKnUCVYG4INiPYRjmAF6idypKzroonEYlpg1VGRuqr8evj2AO3a3IAP+SeeOPhfjqSDVFNGs8RI6yKUMr3A2NzvHIBtcqSQSD3EOXoy6TJaZoFcl43DKW71IY2v3MDcDexHccTbnuR0GaoOuULMRZKiMhZlPhqtaRb81e/68RGQO2Kq5OBHMPMFEGQ5tl2m9Q4uVKaVSctzF2YoltNtgLkG73Gww4suz2jQK1O07C5s2uaOMA6VbsMVLBdN9LKRe+nTqJxE3SlwBNQS6JCJYifiplBCsPmstzokAPIkg9xO4VX4YmDQrbuBU+4n7QQffiMRCIAhYHHBLetxO/QflHyU/0GXKhaxd2NgWklklYhb2AMjNpUXJsthuTjelMuovYByApa3a0gkgeYBJNsZU7XVT85Fb+soP242AY6iPS5gIG3Rc4Sb3Ua9P3Rw9JMaiIF6GYh0cbiMvu0b+A1EsjHaxC813j3O8paIhWFtUaSobW1JKodG333uVPgVK/Jxezo9USUiK4Dr20IYAqVDsACCLEW2thkdO/RGtVToaZFSpp1CxLeweEC3o9ztddjGW2Ugi6h2I5PLw3scSw8jsPn/KXSScM1x+LH1ANfqkH4P1bVJlxc8kSKSNm7SyQs0qCM+BhWNRZSCBpB57SPlXSEh2lQofnL2l+j1h7AGxydHuQs2UUsJ1QyCmjDKV0sHCgtG6kXFnuG5HbzILIr6F4zZ0ZD3alIB9h5H3E4vQRtaymiuv13WlqfE1tdgpros1il9SRWPgCNXvU7jw5DDR6QOGCS86W2F3S25t6zg356bEi3cd7nEdXx2x5xKLWlksCCB1jEC3LYm1vK1sT6KSOymvbTh9FzpIRuLjwI23HgfEYdGR8ayps51jxIDEf7SE+0sfZhFmz6RojE2llLa7lRqBvc2IsBc+V9z444hIt91IFhsrd9hdu0GJHM6bjnz2wulQNeW/lKd0/EFLKdUsBWS1usS2q3dchlbzsdQ9uOM0isfiJlk8I5bpJf5oJAVyfaMM/Mq5I0Z5GEcai7O2yqLgaieQAJ3J2HM7AnEb13SBM4WKAQzVJIKyU8sVRSugNmLoH66G4NwjD1l2d1DYhllbELcp4tcxoC/snf0pcTAh6AIXq5kZWXYrCLfyslwRsRqCfKsLkAgsqcJcJxLSejudYkjMcm9m0ldJ02sQwG+oWIbcWsMJPAnDno4Ls3W1Eh1SyNZize8EEDkBa3gAAAJBzLPklUdZEplBF3U6Cyj5Jsrbnlfu7rYoRY7p5PGlFAflHX/sf2V+TKZBEYYnbn8x6H/5HoOvdQFlcsprFlfeWO9FUjlZ4iVjmH+rmjN1+adK76heccml2QG1gtgDzJa7H6NII9hxEnSlXJS5hDPEjLT1Cn0m/aIMVgZSRySKMxuWYcozfkMSxl0BL9ldbKpNtQFgPlC5tyNvYccdxDGfFxeFteVxq+4BDq+Wy6aHLZNwx5H5hZPuRV/NLCTEd5+nG5a1vI+79lsJzzMOcbgeOklfp5HGC5gnj9R/Zj0Olw+v1SyuY+X142DMB4H9/fhF9LX5w+nGxZh4j6RhKTtaWRWr4/UcZCpXxGEa+C+DSjxEuCUeI+nGQbCDfHmE0pfES404HePpxgaxfH6j+zCPfBfBpS60rGuXx+o47RPCu0juGsCNCFhYi43VX38tvfhuXwA4cGjqml7q2S9mXEIMZjhRo7/Lcry79gxa55bgbeGG5BRALY7m9yeRve4IPMWxtvj3ViZsmkUAqc2O+U6nG/0XsMYHLv5km5J8ydzjdTQs5soLHyBNv2DHOWxJ/DdOqxJptuoYnxJFyT++3Luwx77VjDxKPNMKPJ5SwXQwJ33FhYczc7bYVsh4Z1G7spjHzGvqPhccgO+2+HpVQBlKnkQQbbbEWONdPEEUKNgBYe7ELnhosrRbjgFewQhQFUBVHcMQ/wBIuUxF5opIhUQzMjPENmA031o1wRKrlnAQghQxvfSrSzVTgAsxCqNySbAD34hWvzkySvJ3M2wPcBbSPaoC8u8YgbHJlRvDO33UeRmxYc0T38w69ug6lJHCPAVLTF3iklvJa3WjU8a23jF9DAE7m4DbWYmwsndOOd9RSoIZGSVpFAa4DlRdpCtuSg6AbdxsSdRuqZhG7MSCQD+UPfyN8Rb8IOmaOmjlJvpkK95btLqsL+PVkfRjIjxcnbVYA9KWxl8UgyHlxOpzhVk3sm9kXSPWdeIzMXXqmc6lQ8mVR8i97tfnjvz7P6mUHVIGHzWBWP3qhAJ8yMQRkNS8k1y7KWVrFWIIFwdA/JFuXkMOmh4aeU2BlmN/VRSx9hsGN+/3YlyoZHOA17Vy5rW4RxDCwobdCC+z5tgQPek5OIcyYxMjyIuxsElBLbGy6DGGYX+Tq38Rhi5ZlQcXZbAHlcKP91jb6PbiVeF+iSoPNFp1PN5GDPbyVSWJ8mKDzGJT4Q6O4KezMPSJfnyKNI/MS7Kp8yWYdzDli3icOlqgKHfl9lj8a+KIZX6hzrob+/JVslyFVsFiGthca9ei1rljf5IBvsPDcXGFHopqEj66qkN33jiXvJNmkY9yoOwuo2AAcDewL/8AhDQSiUaIyRULo61QSFCga1Y9zstrXIGkX3KnERGNU+L6xXATUdDb6ie0uwIkC8+yxNuSnezYwY5C1/Q7f3WgWB+IyfH8xLRqJPInoO9eid0A7N76ixLFvnFt9X9IWOEDirMNK6Qe01x5hflN/wCUeZv3HHLQ1bxKrEFqd2KKw9XWoVmVG5agsiuUvuGBHecJBl1uzty5keCjZU9pNrjxY4uyO22WBgY3nJfyG5Xsg6tPB2G35Kf8eX/xiXfg4keiyWtcylie/dVCg+wL9uIRzioLe1jb2Dvt7AMSz8HSqt10fiquPYhsf/yjE0BEb2t/nJLxJpyceV1cgCPSiP2Uy3w0Okzgla1F7RSWMP1Z20ksBZX7JbRqUHs27+eHXfG6mS/uxrOjDxpK4eLIdA7xGGiFULinKpaY9VNG0T+JtpIv6yMOyynlqHs5iwsp8FLh8rRNLNqHpEuqO2x6tF0hiCvJ21Ed5CqdwcPCogRipZVZluVLKCVJ2JUkEgkWvb9WFI5jYJYkFRYj5B322v4GxB8Lg4zMnh0zRUDqN8+3/vJdBD8Q48wP4tl7cuh3/bmu7NaJVFwSfIgbi9tQsfHa2E3XjrzavDItrCxN1+ULjx712+zywmX7/dzF/o528+WL/DDMYR4x81m7WFxpuO3JIxh5aBFe1qm0/I4zpHsb87XP24xrBzxtyfd7eY+tiDjHJ2XedU58r7SKfBgw9hH7R9eHfkWdyQtdDt3r3H3eOGTwjJ2SvejMnuBuPqNvdhwA4zcgUbVoflU+cB5jHmNFUxqB1/VMoV97ONQGxJsmsA7XBDd3LFeOEjJFLJTSKUdCdStsyutgVIPO62YHwFxcG4cvQ5mLelGSEtFJEmhlG6yNuA58FZIe0OWoIL3NzLOc0lPXjrJIxFmMK21AhOujB3QMVYK3MAsrGMk7FW3seK2yzrQKyuI4ZnhNC+yXeHt6eA3v8VGL8vkD7LY7AMNLo9rJxTxh1LogPWO1y6E9URHsugxws9QgckKyU6qhZyFZ5aMb3D3f7WnsuLy8N8DqcE8+jnigR2hewQklX5aSdyGv8knv7j5cpDgzWJjZZEY+AdSfoBxBOnGSnvGxG4PeCORHgRh8uK1xsK9icXfCwMcLA/RWBUY4c+y4SxuhF7g28mt2WHgQd8aeFM0E0Sv32sw8GGzfTzHkRhUOM0ijS6prhI2xyIVepbg2bssNiDsQRzBHkdsF8T8aFO12V7Xrdkb7W323223wjHg+nN7xL7tS/wC6Rb3YeHqg7Bd0Kh2niLEKouzEAAcyTyGH1lPR0xUmV9DfJCdq351wL+xfpw8aDhqCNg6RhXF7Hc2vztckXsbXwsYQv7KWLDA/Puogz/gmaIFhaVBz031W7yVPd42Jw0qemVblVVSdzpUC58TYC58zixbDEE8U0PVTyIOQbs+SsAyj3KwHuwop3NQ5MPh7tSXUz6e4tvbsi5Hna/K+3lfwucbA3u8tv1XGMcJGa5hIkqAIzxWJkKoWZd7ACzAG5NyACQAdjfDZHiMWeSZjY7sk+Gyronc1ddPft3UfdO6yNJCFDdmOY9kntRlAJ42C2JSVAUYX06QdVg20k8Pzho1KjslQQSLc+YsQCPHcX5ggEWxydWkjiVNzpEcmoOjiLUzMFDBWR2N1DGwHO/Zx2SVkUYupAg3AckKFKC/VtGWLx2W5G2iw57qDg5bWfjYZXHYF1HobFc+66ONsn4F8DGHVTb6EEG9x7duaUo2sbg2Plt9mMpZieZLe0k/bjTfBjo1yRtZg4L4xUYGH7gg/ZtgpItig91vpA/Xjemvu1e43+w448GEpKCu7XJ+V9F/1Y99Jk8/6v/DHEshHIke/GwVTfOP2/bgpLqXUlc/hf+if1Wxujrm+Zf2X/ZjiGYP4/UP2YzGZN5H3fsOCk4O9UrROTzFvff8AVjO+Ef8ACbeC/Qf/AFYyXM28B9f7cNpO8QJVZsaDM3cn0sB9l8ckeYm+4A+n9uCbMiDsB9eCkusLtVm8FHvJ/UMP3o4WTQxZvi72RbcjzY7k7bge2+IwbM28APpP68SN0Y5vqiKH10Yn2qxuD7iSPo8cMkcGNsqxiOt9J5SSYS6vMlDpGTZnuQNuQ7zfxPZHecdqrfc45RkaFzIw1ubWvyUDkAPbdvaTyvikwGY27ktKTUANPNNvpeqilOANtcgU/mhWY/WoxFVChYhRuTy8vO/hiYelObTT6tCSEOoGtQwW+2oKQQW+T7/LEb5Jw9UyuGSPqvFmQxx+2xHusi22Gw3ON/CLWRdlyfGInyZQqzsNgEpz5NHFEWdtT22PIX+ao5n2n6sRx0h5AtZTtAzmK7IyuF1FSrb9nUoOpSybm1mPgMPDjSnEUgjaQzSgDWbaY0uLhFG51WsxNwOWxJNkGR72xYbEJGb72s6ecwyANFFvz39VGHCPQpHFPHK03XKhJ6tqddLXUrY3mf5191O+Jgpo1UaUAVRyCiyj2AWA92E8PjaJMI3DjZyCbJxKaUU4/TZKEbYzdsc0DY2UtCanrYoKqCGpUALrAlZST2j1IkRm0rfe9gxW4IuMMmdoCmwoTO8NHzVdOljpKjedgqSMYwYtLdgKysesB5kEuLGw3CrvcbRBnPEjte2lTq1XCgsXNjcG22kcrctrWxZTjz4NNNGbLXzQzMSddbGPRpiblh6THGBFKTdzrEhseTEHEF8c9GdTROFniFmBaJ0ZZopEBt1iPHe695LBGFxcC9sc8YA15e4WSvS2cQkkibjscGtaKAG3+bPdR6FZyTux7z/x9uHO9MdIZXMgOnWSLHXbtAjUxsDqCufWFjZSdA50iPgdudt7e0C9vfjzqwfHwP7ML4+/JN/BWNjutUhu3kNgPbufqt9eJQ6DZLVAvtdJFH+y1v8AZP0Yj2AKN7XPny/afZsPbiQuhYlqkNe50Sey1gLAcuZw1shfMz3CsSYzIsKazvod+im6+N9KeZ937/v34WuG+CKmoCskdomvaR2ASw5m1y5F9gQpufK5GGecPPTypHOQgNiWTtgIWILAbEnY7bXx00b26qteUzQSiPVp2PVJxkwAk3O5Atc9wvyv4X7sKWcTUoBWFZZG5dZKwC272CIFNxyGrbvIPLD26IcjWpUyTANHEwSOIACLVpDO7KB23IZe01yd73sLPkn0N1EJkGEZZfCDhfpy+qZGWJqFoopJpbbnTrVL94RVO/cGdiO/SDa3NmNBJHbrI3jvy1oy39lwL4lLpI41FNenp1VJLXZgoCx6twAoFjIRvvsLjnyxFUmfzEMrSu6t6yuxdT52e4B7wRYjuOGQue/zVt681JmQwwnw9RJHYCh+6p/mY5+zBlNwwPmV+skfYR78bc2X9n14zyiK6k+DX9wa5HvFxjn3mmr0Kt128Py/GTfnKfpLX/Vh1IcNCi2qJF8Qp94CH/zHDtTkDihkbkKePlS1dF9eYq8g+q5dW8ACwZWP1KPz8T1XZarG/I95Hf8A8fP7cV8ol+OlZLXFP1lz/qp4i1vyl6rV3dlXxYPIqrrIYn5a40Y+1lBP1nGfxG7a8dlNByIStwzmckLgI/WBg46t7FCEVSdxYoe0ovZvMG2zmqO0SbWub28L7kYRYcrvJDINrBlYb76l2IHK9+flbww42ixt8DedJJPZcz8QNstHutEKLyIPtB7Q9gJ0nxsQCfEY9zTLjGbGxBUMpHJlYXDC+49h5EHGRTD94iooZKNmQ36iJyj/AChojvpa479IuLeYxtPm0G1i4+F47SBzHL91q6MsyRV6khlkYs4uOy2w5HxCj7cJuZdJ5WRwsS6VZ0BaQqzGNipY6Y2CJcGxN9iDte2EWDiaONkYfGOhuFHK5UggtyHM+JBtscMcpKWku6aZGdhZDrUSMWKjtabLqIBIbYC4OLnDsETkvlbt0vZScQ4icVjYoXgkHeq5evTmp14X46hnkMQ1RzAX0OLXFidSMCVZdj4Hstt2Ws6S2K+ZbJomimXaSK9lJ7LKQQUfYm3gw3BvzBIMl8H9dUETySERhjoRLqpKsVO3MqGBXtElrHktr187h4hOpp8vrzvsrnDeLnIbpIt3pyruU4M/4gSDTrDWY2BAuOW553222tffa++M8u4jhkOlJFLHkDdSfYGAJ92M8/yZJk0uPzW+Up8R+zkcQtINLGx3VtmB7wdmB91wcU4YmyD1VjOzZcZ4NAtP19VPd8Qz0iUMi1EjspCMQVbmpAVRzGw5cjY4evDvG6MESS4kNlJt2CxNh33F9uYsL88O9lBFjuDzHcfLEJaWHcK1qjymeUqu+DG/NECySKOSu6j2BiB9Qxz3xKso7GkjcR5dK/qMoUizBi3LvNgCCR4Gw58rk47aDL1WNYtKlAAD2QdZt2i172BJbsXIANt8dmrBfGZNwnHlkD3i6N1e1+y12ccymQ+C0gDuBv8AX9+a9wY8vgvjTWQvcGPL49vgQjHqjHVk9KHcKTYd/wCz34k2DIolTkLfv9J9uGl1KWOEv3UUotzbvwvtw2wTUSBtyt+vxxnXQoHJvbSdtx9f792NNbxEzC3uGEtAa1t6kgnBj3HmHqJGMo3tjHBgQvXa+PBgGOzLaEubC2wuSTZVA5sx7lHj9pIGI5JAwWUrWlxoLHL6NnbSoufoAA5sSdgoG5J5YeHA1QoqEjTcENre27sEJAF/VjHMDmTue4BYyXh2NYryErHzYN2S9vVL2OoJexWLbu1XY2HVS8UQoQqIUjF9woA8iFBvbvPf5YpNY6Y6jyWg1rYCNTgClHP8+WJlW177t+Sv6z3+7zGFmGoBsQQQQCN+YPI+zDD4phDy6lYFWVSLeG4389v/AItvywVLxujaiwW+3IaflL7+7z3xbpuwHNOE8gc4uHlvZPnNaLU0ZYjq4yXKnkWAGhib20p2msflaT8nCHxHx9Tw2GrrSe6Mq1h4k6rD7cIfTjxA0cSxIbGQkSEcwlvV8tf2A+OIVU4t4+MHjU5Z3EOJmF5ZEN+pS3xFmZmmeQ/LNx5C1gPcBbHJG9scitjajY1W7CguWeS4kldKNjKeqCjUxCqOZJAA952wlZ3m6Qprf2Ko5sfAfrPcPcDF2e8RPKw1Ha/ZUHsr7PFrd53Ps2xTzM9sG3M9lf4fwl+UbOze/wDZO7ivjUtdISVXvk5O3knIoPPZj+T3w9ncodmZraY/VPylK9p3DCxVgbWIN+zh0GXET8Q5m4vCNrXElhux1H6AdifEnw2xzz5Xzvsn/C7TGxY8dmmMfz1T9oOnqviiEAneqiKgSR1gSoIYM19EpAkZNOmyS9Yqm4s2kE8bdKCOCGp2pyW1N6NMepdjzdqaa8Kyn+ci6t9yL7m8ZCO3vGPcWA4pHMad1KU+bRT2s8Mx5LqCw1CnyE1w3tVjc92EmvyMC7MCg8WiYD23Vt/G/LDAeMHGyJ2T1XZb/NYj/dIwho80rC5nI/z5J0nKu++tR8oXt77jbc94w8eiZgtSoUWLBgTckWtc2W9gdhuPoxFKZjLvaRt/yr/bhxdHPEjwzo7HWtytj4sOQO25AsO65B8brGGtcCmzeK9rhq5gilerh3pZeNERoVdUQICHKsSosCeyQLgb2Hn5YaGe5pNW1BbSWkaypGgLaVF7KPIXLFjYXJOw2CDG4IBG4IBB8QRcH3jEgdAxArDsTeJwCBcL2kNz4A6dN/Egd+NkBsYLwN6XJF8uS5sD3bXSV+FuiMvHqndonYdlE0nT4FybhvHStvzvB7cIcNJQQynW8t+25CkmyjYJGuolrHkLsxsPABwZ/nMVPG0szrHEguzMdvIDvLE7BRck2ABOGjFxNIxSWYPTwsbwU/8ApEigb1FTcgQRqO0Iri22sl3EQycriGlpMrgB68l1mHweOOnsby2vqfT+clD/AB7DOJnmqITTCZi8YZlYhdgA7L2FkVbalBOnxI3w2KGt1jUAQnySdtQ+cBzCfNJ9bmBbSWf/AEyxtIUmdxIGLRmK3Yg7COqEMfXeOQMxKg2tewIUMIPjU4bM6aFr+nS+Z9T2XLccihhyHtaNzR2um2LoXufdVnzQfq/3hjv4YiBX6Qf39hxyZqvL2j7b/qx18MbG3ioI9o2P6sYU35F3A5rTFEUqt99S7HxsoHsvdDh0VUlorjudAfYXUH6mwhcUjSEkAv1bi/jpOx+k2Hvwp05LxT237T6beSKAR/SUnFN++lye3Y0knMx2VYGxE08beYckkewozKfInE9dE9frpUF+1GWjPuOpf9hlHuOIPFPekZ+8ymUeA7YU+61z78PnogzxY3dGvaQAi3it9ufOzE+wHEOczXEa6FSROp3urEZIbhPaPqBwuumG9whKGVCORP7RY+eHW6YvcFNRn3WJxlup49kl1bhQSxsBiO+KOkzQrxxsdDbMoayHuOojncbaUvcbE4bfSlx31rFIzaBSQtjvKRsXPhGfk+I3PPSGNleWyTuFUFmPIAbD9/8A5x1j5sbAi8Sfd5Fhvb3XKFskz9MZoctuq7sx4pnlPZYqO7TdfoAP2k+7C3kXE1THbVIki94dSWAvzDqQb9121D24cEvBKUkamZg08nqRg7m3Nrc9C8rna9hucSB0VZRPNADEECdsTRsbRSWkdNLqQwk1ooB1CwHKwtjDj41l5ry5j9Lem36K6/hAiYNTCbPTmuaCItHHNbVE2kiROaHUOxIOYUsANQup3B0WBaUuhhj1UwudImuoPJdUaFwPIvqc/lMx78JfCPACxnVTySRUrs4mo50Eiht1IjcsGSzDe5lV1v4gh4cCcMiljZAzPqbV2jfTsFCA2BIUKN2ux3JJxKzPnlhMWQBqBFOHUevY/qtLF4Y2GZssR8tGx/b09+SX3S+2I5zngpkD6O1FpLDVbWjKL2BtuGAKe8XHZBxJBwjcTiUxssShiwIN2sQDsdItYm3iRbEcby07K9lwMkZbhdXVc1C+Jj4LBNNHqOu6nc+Go2U3+aOz7sREtOdWg9ltWk6tgDexvfkB34mKKaOmhQMwVFAXVubnxAFzvufIeQxbyzsAsXgjSHuJ5VShbOYAksifNdlFzvYMQPfbHJjo4keNpmMchkVmJLONJ1E3Y7C2i5uDYHy2ucc1yZ4yvqyB/VaJusVj80EC+rytv3XscVwp3N3NLlY4NWNBfHrSYVMpbtWM4xfHLrxkJcCFuJwasaNeDXgQlPKUYuNJsfHCtmuaTDslvo/fb7cN6krSvLnjKeuLbnc4RODqGyyJxhrxpabHmvCpq36sGrGjVj0PgSLdqx7fGgNhTyqh1XZjojX1m5+xVHynPcvvNgCcRSyhgsp7GFxoLPK6Avc3Cou7ufVUefeWPIKN2PLvOHFwo6yTxxKLQgliDbVIUUlWe23rWIT1QNtzcltZrmeqyqNEa+qt/pdj8pz3nu5Cw2xlwxmnUzxudgGGq4PqNsxtz9U3HnbFZkTpTqf9FZa9rCAPmVIXGFUZJerB7CWv+dbc+dr2Hvxxx0qgWtf27nHmk9bLfnqJ9xJIPsIION+HyuIOkclaxY2vuRwsknn0WiOnC3IuPLuxprZttsdMzYTKh8OiaXGyo8yRsTNLdklZ3RCQdonZgxJ35c7353F+eI4kIubbi5t7L7YfnF1Zoia3NuyPfzP0YjxTjZgvSuUmouXSjYReK+KFgFtmlI7KeA+c1uS+A5nu7yEzjHiwQ9hLNNbfvVL958WI5L7ztYNGNTOWYsxLMxuSeZP7/RyxRzuIiPyR8/0Wlw7hPinxJPy9B3/wuviDPmcl5Hv59wHzVA5ewfbhB4crDI7v3LZVHgTz9rWAv7bYQOKKxt77aSVC9w8G89XP3W9qhwDlVXMmimhMm+p32CqWsd3chL23t2m8sY2hzhqPMrqmhrBQFBOXMM1WMXY91wO828B+vlhg5fQvPIwUXd2Lkdy3e5ufBb2+gd4GO3PsmnjkMcqsJj3HtXF7XBF9gbjy5bcsdvCWcCmZiRcMAD47G/7+wYkY3SonyJH4nyJoSobe4t9ZwiW/f3nDh464g9IlDAaVUaVHsZrt7SfqA88ICsLbg+0H9RxMkWOPGxnYdx+nb7dvrxiRgQtIwpZY+3mDf9h+rCe4wscJ5PJM4WNb3IF+7c295v3DCO5KR+7dlbP4NGSfhCKxfQkFlc/LKtdowo5eqCtzy0HY4sLkHBVPSy9ZGX6zQQFaTs8xcnYbnYbnT32uLiJvgn9HU+X6pJmK9egUwkWsAdUbPfcSAmQAbWEhB35SdxHmS9aU1AMw5d5t6x8dKkgXO1yBzIxhfEPxHJwvGaQ3UTtXpXP+ckYXCYpZS4DfnfZNtaSWeZair0tKp/i1Op1QUpOyv4TVZvYykaVJ0xj5Tc3HPGWh0p6ZlqMzndYwbdYlOL2aVwvZZ4xqbTyWzMRZe0yuNulIRMRAVkmF1Vh2oodiCwPKSc3IuOyg2BPb1M7o76QBRGWVYRUVkm3XSPaNEJuwCKtyWIUk6lGwGwXflOGulypRkZ7t+YbvpZ7Dq7sTy6L1DE+GMqWHxzHYAAYzYWe5vkwcz1d87Uw9MsMVPTU9KNUkoJlMrN2iTfrZZPnPM7Mx5C4PzQMQ0c0BOlO23fb1B7W5fRfGvPM4qa9+tqnNrAKqoEXTckBV56QSSC+om/PCVS8d01JKQ0JqerW6RqwWNprnszudxGlgToWQuTY6QDq7NvxFLKRi4Td+rjvXrS5zM+A8PC1ZvF5NZ6Rs2BPYu5n1qgO6hvPU2H7+OMaOo0lG7uR9htf6CB9eOjOBsPM2+m+E6o2YDmGvt9vvONarFLmXCindUwhlKnkQQff+zHLwFPayHYhSCPNGIb6S2MsmqNSDvI2Pu7/eN8cORi1UR+Ufok0N9t8UNOzmlSXuCnHUUmimdO4K9vpYqffcYTeH0IhEo9dXLr5qr6XX+i2kn8lj54WszhPoodjoVkjOpu8EKSVHN+/1QQO8qN8aOEoA1IFUXIeQrfvOpgVJ30h1Zoza9gx3JFzG13kN9XJ5G4rspv6KM2vKiXLJIV0W3s17257X3BtfcDDw6auJfRKVjuJZT1MY5MCwOt/LQl9+5injhl/BLAaV1O4SMSLcc9QCgkHcHQ6t5HHJ8NFGWelc/wAm0UgXw6xXXrD7WVof6vli/wANa2Em/dZPFmlzbb7fVRrwxlL1MwjQXYkX8FH72AHmPHFnOH+GocupXkCCSZY2diT2mKqW0A2OlduYB955t34MXDSpTekMtnlJ0se8AkFh+Te6g+AJ+VhR4izjrY6579gQMkY8F0Si/tdiWPuHyRjNc2XiuS9zj5G7u/YKCBsWCxhO73Gm+ncqEXz+SepM0zanc+xVFjpRB8lF5Ae0kkkk2b+DQv8AFX3v8axtbkCAb+dyT9GKoVVGyKj7BX1abHfsEA3Hd6wxZb4NEpNI41aNSE6zbs6XkUvvt2bg77bb41IdIFN5clqytvcrV0l/COp6Ss9Djp5KuVWRZWSRFRCwDFFuGLuiEMwOlRy1XDWlzg3iBKqFZkDoCWUpImiWN0YrJG63NnR1KmxINrgkEHFF+Geiic1NTGkwWWG5laQMUqOskciQKfjzH1egFnPWNK0i6lG+Lc9DXEDmLq6hIoKnrGBETXSTTZRKByXrAuoKD7QhOkPYSXkDoonODGguAF8jfPvspHvjFhj0HHuJEKMuknKNMnWAdl+fhqH7RY/ThEybNGQ2Fyvhf9XLExVEAYEMAwPMEXB9oOK+8dSdVVSpE7BFYWAY9m6qzL5hWJA8OXdi5HPbdJCwMzALJPFYatSNAtPN68aavNQD/WAH240VPAEDbqWQ+TXH+1fEZ0vFM68pWH0H7QcdkHG9VewkufDq4yT9Ed8Rkb7KaN+1P3KdFb0bPzSUH84b/SD+rCVPwDUjkqv+a4v9BtjSnSJVLsdF/Bo7H6ipx1w9Kc4+RCf6Lj//AGHCUU6oj3STUcMVK84JPcur/dvjiky2Uc4pB7Y3/wDTh1r0qyd8SH2Mw+0HHr9JurnDb2SftTBukLIujkzHiYc1Yf0T+zGsyYeS8dR96SD2aD/5hjP/AJaw+En9Rf8A3MG6b4bf+SZIlH7nHvW4fMPF9OTuHt+YP1NjuXiyi7wx/wCzP6yMFnslETf+SjkSY914kVuK6D+aY/8AZL/68an4voP5hz/2afrlwX6I8Fv/ACCj/rMeiTD3bjGi/Fm+hB/5zjOj4pp2IVKQlmICjUouSbAcjhr36RZSiEE1qCa2UUmq5Y6Y1F3bnYX2AHe7HZV7z4AEjPM82DdkABFFo1uexvcnYgM7/KYjfusAAHJmvGsSEpHArKDs3WEAnkWsFBI5gG4Nt9rkYTo+PJCbLDDfwCys30CXf6MVY2GQ63fJSO0MGkH3Sbl+ZOqsqxodQsWMZZ+dxYkkC3LYDzuQDjqzjMJ5zdwCdrARAEWBAAIXXbc7Fjv7BZbhqa+T1Y1iHiY41/8AyBn+gHDiyLhypYq0091UglE7Ia2+klQm3jscW7ATWxudsLXJw3lsrxx9kh1AUlriyjZQb77Jp28sLdfk7ILkgjuADFie7a1h7SbDDsjjtyx5JADa/d9GIC0E7rQYwtG3NRvVUkltRQ28gTb32thLLYlXMarRGzkeqrNb2C9vfiI0nvv47n3/AKsWoRY5LD4izw3AF1kpN4zZRTyF2VFUA6mbSoNxa58zYAd5IHfiCOKOMgo0wkM55vzVfZ85vbsPO+HB8JjNG1QQco9BmPgzFmRf6gV/6/kMQ6uKeTnPYTG369VYwuGseBK/f06LJmuSSbk3JJ3JPeT3kk95xz1k4UXPd+/2kY3HCRxMbKCSFW9iTfvINhpDEnsju7jyxlsGo0tw7DZcnGNJddQ3uLHz71P1W+jFhfg71aJRxrIAiqCCB67N3kW2sdiWYjc27het8WcalCC2lQBqIJc2tuFvYXI+UbWx05bxVMqaVYoBcbc+X1W/44uxgtFFROf1UmdKXGUTVTgALba3rNYci1u8j5Ivb33MQZ1VBizctRuB+/0nnjmEl7uTuTuTzt4+04VuEeD56x+wumMGzSNcRoPtZ7ckXc3BOkG+JGtLjQUDnAbuNBNq5Ow3PIe09wHeSfDE4dCfR0qXnrIgzEDqYpFDab7mR0bYNyCqwuvaJANsOrgjgCnpLMo6yb+dcDUPzByjHsu3cWOJO4f4MqJjsvVqCql5ewoLAFQARqZiGUgAb6l8caUOOI/M/wCix8rPdMPCgB9/7dlBfSrwzAHR1hjQOGBCIqi6kEE6QBch7f0R4YjDOsiCDUl9PylPaA8997eJ3Ixb74SPR/BBR60lJmR4mEbMC5WzJK1gL27ayHkAEO5uBithxk5chbMS3kVs4LHNhDZNyNudqN3iFxq2FxcjcWvv5g29uJ76D+KKOOshEjokSi68ralX4sbctz3/ALcQ7m1AFbT8k7r7Pm+0YQaynsbc/wB+/DmPsK0Whp25K8XSz8ICBIWjpXWSse6pYghB3ynu1BblFb1mA20hjiGsr4kqJIinWO+s9s3Jlk2FkeQkuyKOS3C7m43xX2B9J8P35+3EkcFcQnexs9rbW3uO0wFrLy593djK4zjGWOx0XcfAuZHHm+FIAQ7lY/qG4KfNVAkQvI3a7o1Iv/SPyRjryykJAkcLEl7jVtYfkqebf6yS5HyV+VhpwMwcPfUwNxfcA+O97nzP1YUaycMwZi8m+6t2W9xFwAfK368cw6IgUDv1P9h+5XtUmNPN+YkD05n07Aff1SjxlxH2dMUgTcapN7jwVTtuSOe+1/biNK2NQbK2rkSwvcki53IuLE2PfcHcjEo5vNDJD2IrFPVslyhNtVyIKggGwuxjsQOfhF+axW1HrKdzcnSs0Ste/LSwjAt4WHkN8bvAGtZHQFb9a39TS8e+OQTMWBnQbmyfbsPkL9V0ZzRkafzh+vCBmx7S+R/ViZZOOY57LWwie9/4xHpirV5C+sARVFgT2ahGJ+evPDW414EOg1NM4q6Rbl5EUrJBtstVCSXg7wHu8LW2k3tjYjnINPC4V4TZyeqAIPyWtq+zV7jsfL2YcmYSCnVJlVDPI7BWdA+lYRCeyr3QFmltr06hoIVlIOGxwzlrvaMLqZn0Kuw1FyABc2Aux5kgDCnxzXK0kCI2pIV6lX5CQBg7TW7llkZ5VvuFcKd1OAgF+yQcl0cZVzNTxszM7MilmZizN8TzZmJYnluTfHX0ZTH0ex+e5H09oH2Eg/0h4YQ+KX/i8A/1IJ90af8AHHd0cy/EEd4kYjy7KftsfK4xDIweB81KD5/krIfBxiCzzOL7wx38PXIJHl2Rt3W8CMOv4S3DyVVLCW5Q1Mbta9yjdiSPbkHVufcQrdxw0vg0T6mqPJEA8uRP1k4dnH9b1rinB9VXdvDUF0gf0Wcf7Xhhwdpj37KGRuqwlXM87AoltYM46oAbAWHaItyGj6CwxFNXn4/jFOrbmAswte+hlvv3W1qPPUfm46KnO+woPqxxFiPyzYn6RYYYXBSPJPUPuSI1jv3app05/wBmW/peeNbHxxg8Fc/+p5v77fZcyJTl8XaOjNh9N1pz6Tswr4I58u1I36gP3GFx+l70KlhpoNJqJFmE7MocRQNI9rKbqZXJDAOGARGup6xSE/pJzmlSQDUA+99JZlJuSyBVBAIJud13Jvvc4afEHB8LwTVyzNHeVUjE0ZRmHVRxuNIJbYrI8YAJZVA5tc5GJONAG4v9V2EsTmPsjluQe3spBPHYhphM0RWocmDroSo6ySFYXE5Vt4nkjkZJUGpX7JBGplWROHeKxVhnIRZVK6xHcIwdA8U6gm4SZCHH5Qcc1OKtZxxYahEjSPqoYSxAJ7TyyCMSzSN3yMsUahFULGqhQWJZ2kX4OVLK9RLIZLxR06whC+/amLqAvzFZ5Gv3FyNtVsaeI7wiHPPPmsvjMEM7XjHaaBtg6gC9vnf6Ky+S1xhUys7AFTfc8u7mee2x7gT54kTgjMjLTxyHm4Y+7W1vqGID6R85t8UOS2v7f32+nxw6ui7jNKaBRM9oyt4xe7GS5+KQd7OASBcAFSdrk4fZkcSoGacaNjD12+dKaDiAekHg2aKWR0iYwF7qynXbUeR31jtG1yO8bnniYMrzyRnIeJY1AOwl1yg3GjUFj6oagGPZla1l56jpWoakHyPniFuSwGrCnkx/EbuoQ/yVVOhWvHqJGpNXaQE8720sVG5APdtqxMfD+SRQIEjQILWJAGpj85jzZj54U9OPQMTFxKbHA1nJMzpX4ejlp5JCAskSNIr237I1Mp2uQwBHtse7FfhJi2NREGBVgCpBBBFwQdiCORBG1sN3OeCaaWLqurVAPVKAKyHxUgfSDse++FY+lFPjazYVcOsx7rxK69Dyfz7/ANVcb4+hyL+flPsVB9oOJNYVT8LJ2UQ9Zj3XiZB0Q043Mkx/pIB/+PHNJwfl0fruv9Op0/7rrg1hH4Zw5qJNeDrcSlK+Vx8jC39aX/145xxjRJ6i/wBSEL/vBMLqSeCBzITAp4Hb1Ud/zUZvsBwpU/DFQ3KFh+dpT/fIw5ajpNj+TE5/OZV+zXhOqOkyQ+rEi/nFm+zRgso0RjmV7R8ATn1jGg/OLH6Atv8Aaw58k4IWMEmQszKVBVQpVTs5Fy27C6X7gW8QcNnL+LJ3u8jhIUtr0KoLE30xKWDHW9jvfsqGb5Niy6nOqieFqqetp6WOSrqKeCFoKueWTqJmj+Khp5C8gAUX6uM6RzPfijKTI7Te3VWoo21bRupZbJqKL1yl/wDWSXP9S4B/q4xl44pYhaJS3lHGEX/a0/UDhh8HdH1RVxCamrqGaIkrcU1SpVl9ZHR51eORe9HVWHeBhX/yMZh+M0X93n+8YtDSBSTw5OgAXXX9JTn1I0T84lz9WgfbiS+jLPGmp1eTSG1Mgsba9O5Nu4+tcDwvyxFX+RjMPxmj/u9R94wt8J9HuZ07q4qKKQKH0q0FTpVpAoZhao5lUtgcW1snRMla63KZsGGV1Wa/zmX/ANjU/eMHVZt/OZf/AGNT94xGricfE0BeGVRzKMB7dOw9+IdpnuMP8w5r/OZf/Y1X3jDazTgfMXfWJMvjJ5hYKmxPjb0jY+NuftxYhlDQQVk8SwXzkOZzGygr4TFE38Wkt2B1kZbv1tpZVO3eqORY9zbDa8JVEmkg/JNgfI/JPsb1T56fPFzOKOiuvqIJIHmodMigahBUalYMGV1vUW1KR9o7ziNZfgoVhBBraUgixBpZrEHmP+cYzp4i+QuHIq/iMdHEGu5hQNhI4mkGkDa5ufcCtz9YHvOLCP8ABCrCNJzGAj/+me+3Lfrbm3mSfG+NX8Dmq78wgY2tc08pIFwdvjvED9ycRsxnA2VYduNlViWHe42P1HHMdVztz5gb8xi1UnwMKk/6fB/d5PP/AF3njBPgW1I//UIeVv8Am8n/ALuLAYVAGFQf0ScDCrLtI+mGJlDRr68hIJA1X7CbbsASeQ03DCwdBRqoVEUKo2VVAAA8B3Dxv7zh70PQNVxKEhmooYlACoIKg8gAWZjUXZ3ILsx3LE45OI+gDMJomj9MpYw2zFaee5XvTeo9VuRHeLjkTi/DJHG31WRl4eRkSUdm+/3UTZfxuJa1KeHS8XxhaW/raI2YtFuB1asFGs31C5AAsxceVcYxzu0cdQZXhKvYSMQCbgSRkmzW0gGRL2um/aXGEvwNaotq/CEANrbU8neCD/nu8Eg+RPicewfA2qlvavp9+d6aTf2/HXwjcp3XdSv4Qw/lJG38JSZnOciWbq+sDEXL3Yl5TbSV7ydIa5LbX2vfTeMKqHSzKeasV+g2/VixPBnwZKqm1/xmll1gDeGoQi3cClQDbmdrG/fsLclf8Fusdi3plItzey00+kbd2qpZvPckk4wH4878l8juRArf9B0rkuumyMX8DDjxNosu/W+ZJ6knf05dFVjjis09WBz7R92w+s/ZhqTz6iTyxbzMvgeVMhBatprgWv6NLyuT3zkcyccD/AoqD/p8A9lPJb/8uL8cZaAFlGlU1h3YUeH6so4I7t/K3eD5HFnz8Ceo/wCkIf7vJ/7uMh8Cqp5fhCEDyp5Pr+N3w50eoUU+OZ8bg9hpwIII6EJpZHlEb06StIgUoH1qQqgEdpWL/MIK6iR33thoz8S0YJAmkNja4guPcesF8T1l/wAFCqSJYfTado1OreCa+rUx1D46yntAbDfSL9+OHM/gmVpJkbMYGKqbXpCbKLkLu/IXPO+MOHglOcZHEizQFCh06FehS/8A9CzGxsELhYA1WNVnr2UFrxxDGwaIySMO4oEUi24PxjGx25A+zlhqcQ8Syzs7MQFawICRg2BuFZlRWaxJ3OE6ozEvYtbxsEQf7qgn34xMBKllHZXTqO1hqNluOdyRbbGnj4MUJto379VyPGfibO4m4Cd9gdhpH2Tor5e17APrJ/VbGvJ+KZqeVZYZGilUGzqd7HmpBuGRrDUjAqw2IIxpqnuWPn9gA/UccOXZc0syxrYFyFufVQWJaRvCONbyO3yUVidhicsBG6zXHdTfwvXUlREZXEeXVk4kjR0V/QpPVWWYxKrvSF1dqfXHrhJMxEcZQlY+6S+EJoCnWJYMpMcilXhlQHZ45ULRyDcnssSLgMAdsNvMM+Uy3jusEdooQdiIVFgzAbCSRmeoktzlkkPfh6ZDxvJAjIQs9Mx1S00w108lvlabgxyjms0RSRSB2rCxpGJ0b9TeSUEEJo5zUlo1B+RHpx1dH02zD8on/ZUfqw6OJOG4pqVqqj1GBQOvgdg89IS1gHYAddSkiyVIUbWWQI+7M3hEFWa/l+v9mLDS17C1OB8wViPg95qImrGv2iI1Xv7bKNP1DVvtZcPThRdbSTm/bOhL98aE9v2u5dvZbEUdB0BkM4DFQ0xRiBvpSniIt56mO/nib6VFUBF2CqoA8F3C/wC6foxn5ktDQPmnsbvahfO6jSsinY9cU38FYi/ssgN/C3jhhUU2qRkMhhRpQ8spLWWONDuVTtObM5WMXLNpUbm+HJ0mTaZ50Fxpkdt/GVusv7NLgjyOG/wvTLJJJqNlUayL72K6dfmsTmNnFvU1Xt39XxWQDBiZ0oH7Lj+HtczMe4c7P6rmzqtDyARBqaEjTEzMesYXIWadkYAsW3Kr8XEuyrJpLSOzjXK3ahgWNCyRUzPO2oWRiY3BszAu+mJybBioZTyaxa+eP1vVmxQytpZrAoHuoaRdzckMGKkc+8hhjq46eeONabrxKkyxtIwAVroiuYnVT2WEZi3GzqF58o+Z0l2mtt11eNxCRpc+Tf190wcrqbvIb31OzeHMk3t5gn6BiX+gqnkR2nVGaP1JWBChQq6kILbFtT2K2a6sRbtHDKTgN0AIZWPeACBcAE29a5Uj6R3csSLwAJIYGhk7GqUyG7LoCmNBrZgeyAI3Jv8AN7jz0cgNDPNsFDiZgDj4dE0Rv0Hceo6JxcRZw7yq4WKNercOxBfXKWjCMY2FkRU6wsQzMSRsCA2F3jPNVGWsI1sJ5aZV1WusYleckbEXJhVfIN5YZlHn4JeRQOoRCkdwLyu5tra4uAdLFVFrIrk31Wwi9KfHgjhSiWPU6w08nWdZsrsNRRkUAn4pgQdY3cdkgbpgSsGpx2AoD16qtxUvkaxgrmT07VdqQ+EOm2OlMqyNdFIF3uzHSu/I6mdd17IYmwBGwuuS9L0xlLavirdgKu99t9JbSUte1yx5eO1ZqyvNTToDbWj2e1tyUbQ/lrewPiS3gMKvAmda4QhPajsvtX5B9wGnz0788Usp4cS9rQN9+vzU+LCQ0Mc4nt0V++jrixKyLrEWRQraCXTSHYKCzJuQyXJGx2IIIGHNio/wUeNnjq2pWf4iRpAQ7bCRI+sSVSflPGArX9Y271F554r42dHeOML2bDWd97Amw5bXtvfcHbGljEzDyhQ5M7MZtvKf2GV0m9I9NQIDM4MjepEpGtue5ubKgsdz4GwJFsR5xR0sT0sRJKzSOCsQZQCrW/lDosHRe9bAklRq3Nq68QZ60swklczSaXkdnIPbJUAsBawRAwUbAAiwAUWpcQy/Af4Y3NWfT/K0OGQjLZ4vJt0L6/4UycSdPUrQSSInVFStkUsJSrjbUzrdANyT1asLDuIvzZr0vyzUdFURs1NNJO9PNHrMiaVDFJwzKA4Cxm9156wSSgLVtzvPCXDRkhlL3YgWYMsQ0kb7Ex3333HLHsOcSTSxdfMI0BKjTEixJ1q6CdCBVJa4Qu3at37DDYpHSxBhvUf56JJnxxTEj8oPupzz7iuaQ2eQygdx7KH2Bbi/mQb+IwvdGWTpVTxxu2mNgxOki7aQToBK7M3ftcAHkcRtw7ksjkL1wFiVZx2o9SmxILMq2PO99jdeYIwtZvkNVS2kV0aO4IqImJRTcDtabMpuRyB35Em+KMbuI4x87SRf83WlLj8Oym2xwBq+32VmuN+C4PQ5FjhAaKNmi0KDLqUagNVi76yLMCSWv42OIBagkAuY5Ao3J6t7ADmSdNgBiSOi7pZHoyrUNLNOg0KY4JnMioLCQyFSpkYDWxkYbncDEndHeavPSQySbyMpEmwA1KzI2wJA3U7Y6GOQ1uFzs2KHO2P0VYNWN1IpJAG5JsAOZJ5AeZw5elvhpaWp0x7ROiui3uV5qy772utwT4+WFfoa4VaWTriLRRnYkbM/gPzB2r9zaee+FmkDWqiyE69KkLgbghFVGkUPo9RTuus21zEcmZiAqX9WNE5MTiK8iWSnq0rjFPNTLPn9KzQQNUPSzS5xLJHUGGMNKyOsbRFo0a1gGsGF7IxrhkdCX/N5/wDrLNf/ABSrxXjbQWqABsFDWWZLWVccKzxVSQTZ0zSvHStl089J6BIrz1SUpWWNJ5lEZklKsydXuLqB2Zvw3UrVTIkNZ6cMxpGy6pQ1HoUOWRinDxPKJPR1jWJKpJYJO3M7q2mUlWFk8GJUqrvkbVgqaeBoa68WeVk0spim9G9ElSramPXfyckJDxgAErGyhXCEoGbvwbYZpRlEyJW6hT1hzOeVpmhnhYOtJGskjNHK6zaGRIjqiVXDab2a1WODh/JoqeJIYY1hhjFkjQWVQSTYDuFyT78HSkLh6PlQUsHVpPFH1a6I6kSCpQdyy9azSdYO/WScLuGRxDxnNHLUJHTLMlLDHNKzVIiYrIJTpjVoihZRA27yRrci5AuQo/8ALylFg0uhjCk7KyveOJ1LI0tlIi1ANbWV1FXAuVNhCc2DCJnXFlNCWEsyRlTGpBPa1S6upQAAlnl0MEVbliLAEkA8kvH1GI0k64FJDIqAJI0paIlZh1SoZlMDAiXUg6o+vpwITmwYQc/4qiipvSgeviYRmLqirdeZ2RKdIm1BGM8ksaIxYJ2wSwFzjhy7iWdWYVlPHSRrE83Xx1JmpkSLT1izySU9P1EoVusA0vGVWQiTsEYEJ2YMM7JOkKB4pJJGWLRKYwg6x5mVmYUzdV1Kza6mNetVER9r2Z9LEb6rpEolRJDOpSRZHXQru2iJtEzssaM8aQyfFyM4URv2WKnbAhOrBhGpOKad5uoSVXmCq5VLtpRl1IzMoKIJFuyamGsK2m+hrJWfdINPFMkAcSTtPFTlASAry2bTr0mMypGwmMIbrOr7Vgu+BCd2DCFk3GFNNIYopleQa9hqs3VOElMbEBJhDIRHIYywjYhWsSBhOzfiiczyQUlOlQ0IQzvNUmniUyAssKFKeoeSfQFkKskaBZI/jLsQBCd2DDDy3pJSTq9KdXq0q4mcRmOX09aCWmbSsgM0c5ZF0kpI6qAwVxIHG/FVMI0lM0axSI8quzBUMcaa5JCTYKkaDUzNYL32wISzgw26TjOCQAxSKx62ONlfXE69bcodEkYc6wCUuoWQA2bbHLH0lUJRXWcOj36vRHK5lCqrO8SpGzSxRiRNckYZELAMQdsCE7sGG5xPxhFDSGsBE0JWJo2V1EbiZ0SJzIewkBMiu0p7KR6nOy45st4qkRJHrYoqSONUcTR1JqKV1dioVZHgp5OuDBRo6mzdZHoZyWVRCdmDCHlXF1NLo0SqS8rQKpDJJ1yxGZoWjdVkjlEKmbQ6qdFmtYgnlzHj6jjUu86Kg6y72YoOpkeKXUwUqvVyRSI1yLFTgQnNjRmPqP8Amt9hwijjal64QdcOtLCO2l9AkZBIsJk09Us5jIcRFg5Ug6dxhazH1H/Nb7DgQvkXQRiwLb7Cyj1j+oe/3A4elfTWoW7Ijvoa1xcnWm/6l7yN+/GXAPB6skcjHrLgEKB2R+cTztyI2HMbjD04yy5RSTBiLlCw/OjvIoH9TFeSdrCB1KqOeA5RifDv3+3HXA/VQNJbt1LPDGfmwoB6Q/tlYpTKQDdEqlNrjGmjp2dgqDU7kIgva7uQqLc8rswHvxx8Y16tIVjOqGELDCRyaOIkGXa29RI0lURvZpiLm18P5rQJSDStuw8/3+zC9HVXibxCMD/VNj7/ALcN2JrOfP8A+cbql7D27e79mHloKYE8uCuJZaU6430yRsVvsVYG2pGVhpeJ1JDIwKsDuMPiryaGojaspFEaC3pVMCT6K5Nlliv2jQzG+kkkwv8AFkkFDiE/SuxpvuWufYFUD6/sw5+j7i2SlkEsZXUAyFXGqJ0caWilQkCSGQHSyHmN7ggEVpIq8zVIHdFNXwXZjqqNWysomNxy1Od/EfFhQfzB4YdvRnxuKp+s3GtY0IO2/VKwNu7U+sD2nDJz60VLUVFB2aeWL0eohJLS0bSHQva2ZoCGYQz+J0SdtRqafRNmBjkRb2DoF9joLofcQyj87FF8YlDndVK11UE4PhARFKxzc/GpE4HgFXq9v6UR+k4YFFWNE6Sp6ykHyPO6n8l1JQ+ROJT6fafr3oyo3ljkXyBRlJH9HW30YYPE3DLxAsh6yIc721qPPuYeY3HhYXxv4jTNjNLhYqvpsucmaI8h1d7+q6cyiS6PGbxP24780IILwN+VEdI81KnfnhOzGYySx6u0dKqfMakiUG3M6CVvzso78cGTZmUumxjYgkG1gw2DqT6rgEi45gkG+1lDgym62sXTuokVztt1cQ7RI7tTsFHmRjPbAWSV06KV5tpI7KTOKSVjjAbSVZGZV7JId1j07EEJeRhccrKNrjGziLhOWplvqK08aABFbd3LEuvMW7Oi7N7t74ZvGteHrI7b6ZEjXluQ4BFz/rHP0eWHLmHG7U1YQ12pSsYlAFzGxuetWwubKU1LvqW1twBhmUXujJZzvZNwWxtkZr7G12ZtQNGsUbKI13NxcrfZQpIv2kVS1hfZj7MRp0i5WA8kzTqzu66YtBDlLaVYdo9iNUVLkC/lsDYcyxzR3UrLG4NipDKdrbEG1xfxHuxVrjTLXhndXJa7Fgx+UNRG9wNwQVIsLWGwBAxnYUznjQdq6Ley42/mAWjKMxMZJHIixHj4H2g7j/jjdT1RifWvqlipHdp7LqB5lW28CDhJgIJsTbz7ge6/lfn4c8OGlyzUgVgQxB7xbUtxGbrf5D+O4UYuu0t59VBCCdk+ODq8JV08lxpaWNCdtNnOkE37isjC/wCb4YshbFR+AsxAKBv83IjEfkhwT9BVh7LYtucanB9g9p6Fc98RjzRv7gqFulviQCpIRQ7IoQlxqRbb2VTtquTcsDz2HI4a+bBaxAR1cFUgCbvojlj3JHKysrchv61uXq8PGivJVVU0cbtC07KHAJVjHZHKm3a5atr2BHlhqVgLEm3Z2t4e36feMRTiMsLtO5J/9VnGklaWsLvKAPb/ANTsynghUY+lydTHcBRCBLLIzb9nayqCbXIYk/Jt2sa+K8jpYgIkkknqNYZy8LwiNB1gCaGuDrYI+sM2rQpBQFlZEM0gKxu3WXIWNQWc2awVUAW7aiAoVQSSQBfYYeUWZR1FOVmkVZoyBFKeZ1EBQ2kHUoOz25KAxF11GfExIqDuvPdLNlOsho25Wn7wlWK8EZW1gqqQPksqgFbd3iPIg9+FNhfbmCwNj6ur5LHna1z2rbC523xDfAWYgS2Z5IUkUg6bEMw9UaSr3bc2ZV1A2sbXvJ89YH0hWZTrXYWuyhwW23sNK7kgbNb1iQN2KVsjKWbTmG1NPQJw7aSt60LNolRFZhe7gM0jWNwCQ0YPOxFu7ExxQqosoCqOQAAA7zsNtyb4rxlDMFSRGKOFFiptv38t97e/zxN3CVVI8EZkv1hB1XFjsxAJFhuQAeQ545uWEYrNOq9z781tYvE/xkhGmq+m2yQekTgxauWA+rp1CRx6xTYrGPO+ogm+m7eNi8croFjRURQiKLKByAxx57nEVOoaRrAmwsCWJ8gN+XuxxZRxxTSuEWTttyDI67+FyoW/v3wyOCR41uCmdkQsfoLgCfXdOUYYvQl/zef/AKyzX/xSrw+b4Y3Ql/zef/rLNf8AxSrxKp0+sGDBgQjBgwYEKJeP+jp56uaY0GWV6SwQxI1bIyywmPrtZQDLqi6t1qnsyxm692xHWnR1P6NUwtOss01FSU61EmsvJNTxOrTTLctpkkYObSO1i1zfcqD8XyJmVVAUmmhSlopI0iiV9DyyVyyszCzdsQxAAkgaDYC7XatD0xNHDAHEUtSaf0mbr6hKRjG000caQgQOktSxgkURnqkGkapF1i4hLkHCFZJU+kzimjJnpJOrimllCpTx1SMOsemi6x2M6OD1cYF2X5AeTxODquGf0mAU00uuvXqppZYkMVXPBMjiZKaZkkjanAaMRMr6/XXqwTlJ0psXcpTaqSKpoqeWZp9Et69KNoHjg6ltYRq6MSh5Iiqi69YbovNXdKBUColjaGjSeuQGOVJJJVoIa/0kyxPTfFoGotUQhn1uxXUVUMjlboS3UcCv+DY6QSqJ4uokjl0HqvSKeeOpjYxhtQgM8SgoH1CMlQ198cef5DX1sc8M5gooHppYgkM0lQZJ5NGiZ3NPSyRxQaGASNg8vWkloyi38q+kSaElamkET/xMr1dSJUIrK2OkUFjBGRLC0muRQpW1grvckecWdKiwPLEI4+tjq1pVM9QIKck0MVa8ryiKVo0WOXqgqxys0mnYKzMhSLSDTdGdRomdoqczuaYBHzXNqlykBma65hMwqKRw1QzR9RTnq/jAxl6/4v3Muj7MWhiiMyzDq5kbVmFbC0Jkmd4SZqeJajM0hhZICs70/WmLW1jK2hdz7jtpMmrK2D4maKmrSh7MirNTCZNaEqUmhMkXWRuVtJGUYqNRUOzivMjGIN2HWVEMZ0lN9ZOza437Btvp0t4MuEQkfou4Xlplfrer1PHSL8WzMAYKSKGQXaOMlesRipsLrYkKSVCTNwhViQRp6MaP08V3WPJKKrtTddJTiMQmO4kZys3Wi6BYjGN5cJuX9K9TJFHKlAmmah/CEIausTAixmZZLUjCOe80fVIvWJICxd4Cuk9nEPS2IqiOPq4erd6RbNWoK5hVvFGk0dGkchaCN51V2llgPYlKq4VC6pOi5ejDowkpJYA5Dw0kbRwyNmGYzySXTq0b0SaQUVGRFcMIxMrE9gQgABxVWUVUFTPNSxwVEdXoeVJqiSBo544lhEiulNUB4pIYoVZdKGMxlh1nWFU2ca18zVVLRxStTCaKpqJZkWNptFM1Mghi61JI1aR6tGZ2RiqRsFAZw6cWd53Pl8BMssNXqqNEU1XUQ0SpG0esLUypCVeQOrxp6PTFnDQhkFpJcCVIa9Fs4WP4yJ362GonbtorTHO4czqerXS+mPSskcQZifUDHdnxnU9FMjtXo0ka08sU8dBp6wyQemyek1xlsyXV6tYyqxOpEaaQym1lDhXpMkqfR+qplJkSoeYmpASNaWqNLMIm6n49mlBaLWIFdAS7Qmynf0Z9JXpkrRFacHquuVqatWsULrCPFMyQxxx1CFlusbzJ61nNgSJEkcM9HU6v1sixRyGajJH4QzCvcxUzTOfj67cXaobRCkSBe2WkfrLJgnR9WJSZbTq8bClpBT1CJWVVGGkCU4WojqKSL0mRI+qkHo5MKy9YCzAouFPO+NalpE6mJVpBXx0jz9cplZlmEc49HaEqsBcPTiUTdbrFxGEPWDzo+6WkrJo0VYuqnR3p2Sp62fSo1D0mDqU9GMkd3QCSYbFXMbaVIjqlDh/hepgyqnpI3hNVTwQREuC1NL1OgSRNeMusdRGrRFwjNHr1BX06Wa9L0Yy2laKCjy06qOWOmgkeSkknpalKjrZQKaBImfR6OHiiZ9JEjayiRq5Heerq6qNKqajho2ihtAlOZJZpKeOoaV2qYJwY0SeJERFUallLmQFVRrZb0hVB0da5usy0svUrGgklTPo8tacdbHKUjmTtPGDdVd1RlcJKp1QlWp4OrWkWstS+mLWCoFOJ5RTdWKCahEZqfRDKZdM5n6z0UDZYtNl60pkHBGYosMRSiqqeOerq5I2q6inWSomzCoqoC4GX1OuGnSWOQRkj48AksIUZl+p6V41hErRNdYqyWoQMLwtQyiCaLUwVWdqlhDGWMasNT3CqccmQ9JpnYRlUSVZqUM1JUpU0zJUNKoTrpKWM60MDdbGIkYK0RWQ69kSrjzXgSulqklYwmNKyGoB9OrAvVK6FoBRJAlJ1sY1aaiQyPKUFxCZNUUsVq3VgOZUgfRiLh0ry+i0tSYKWBaxRJB6TmSwJoaON0Rj6K8hqpDI2iGGKZdCFnkjJCF88LcQLVUcNUgZFnp0mVTbUokjDhSRtqW9rjwwqFVDo36NJvwVRPGvWSyRawikAHrWMivLI5CxIkbKtlVibbAsTczLoILW9Pr+23aWkpIXmO2wMa3WRwCbF2hsLm7Ab4fvwVuLUqsthgJIlp4I4306kOgXVCrK2q4RVuylTdh33xLVBQpECEVUBN2sN2PznPrO35TEk+OPM8zjWRiZEjHbHUaoCyL23N0PYfNWY8OKTzj+d187Mtl6qKWe9mVeqi5g9bOrqG25GOBaiVTcaZEiPkWBTyX28/wBmHfx7PohghvvoNRJv8uoC9UtuVhTRQyjvBnfxwyaI7/v4jHpEY6qBxW6p20nwNse18mDMBt7x9lv2Y4Q2JEiyhO+Nzykez9mNAXvx7M30YEKXMg4skgtLGQDpAIYB4pI3sJIZY27MkMinSyHmNxZgrB2U2RxzoKmhB+LPWTUZYtPTlSGZoiRqqqQHlIo6yNSBIuxcxLRS6qfzVbH3WP2C3tBwpcB5u8R1ozIyMrKykq6nftKQbg+YxTlg2Japg5TtxchMERG/VVSG/hHNHJGeXcX0eVyPbhBzAho279j9X6wRbyOJByqsir6UrIywTyxgtOBphd4nDGSRUHxUitGW1xgKd9Sr62IyzmOop30VEZXs7kkaSDe0iuLxuGGwKtpa2x54vcCzWNYYXmjaxeKRnxNXooxzKLSTttc+4jux2cGzSRt1iFlsmtwrW1LqBVT4gmx0nmBjpZlbXf1WRiNwLMEJU3OwIYW99twcY0MTLSSyAECSSGMH8hBISb+GpQh8/di1KwX6KvHITHXW6XtOplniVSb9Ylz8oFnF2PmANWFLN68PPMSVF5mUXYKNKkRoSSQoBVQSTYDfCN0fz2qkubDrFJJ7gBcn6Afrwk1JKswbZgSGuflA2b674pHH1N0gqbTTt+ye2Qwy00hkiqaaMc2Q1KMj/kOiM1/AEAkb6SOWHR0nx01XTtMsidfEC+kOtz2dxY2JDBVuwuLR3+TiJ4gQe1cXFxe+/wBPd33x30yoTZwSpBBtfUL/ACgO/wACDzBPI2IhHDNTg/VuPT7FWRneGNNbJEgyeZoXqBE5p0Kq8trRhmbSqgm2s6iFOjVpJGq1xfPJM3ZCATePkQfk78x4Wvc+O/laacl48p0opcvrmJBi0QOoLB4nBEbggHSYmUNcg2K3tzVYU4uytIZSkconisCkoBAcWGq1wL6H1JcbG1xzGKUUkkkkkM0ekg0OoI23B7+ivjSA17DzCVoE0zMy2KGxP5r3uw8QGUH2H34sjTdINPFRRVE0igmO3Vh0M8jxsY30IXBa7oxubADckWvipS1rgAAkWuARsbHz5+XsAxpa5778/r5+82F/Gwxfx3PhcSO1f5VTOxGZTQHdDf8AhSD0g9I8lVOZFHVxBdEaNZiF5ksQN2c3YgXA2FzbUcMn6TJ0kEjLFM67XeJNdvKQqZAQNgQeVxyJwyYk+nDs4PyAy6nYrFBGAZpmQFUBvpUAWaSaQgiOFWDOQSSqLJIiPfVuPNTRYzGNDQOSk+j6YY3S8lMwta5UIyi5sCS1gtzyvzOEqvrDWMqQQSyJq1FFDamYWuF6jWVbSLawLja1rYb0vHrxC1GXo4FI7CsOsluGHXVLgfHSE37BvDFqVY1WzM69m/HNU8UReeYodOpesZYyGXe6KQhuT4YrOfIeStMhaeikLhXKzGqLLlnoyiwbV6PENIO5L1LRvuOZbUb7m+JQoaHJ5SoT0aGS/ZNPVQk7jdAIZW1RfK0FOrBGoAEasVir88aoSGPRGiQrpBjTSzbntSG5DP2m7QC3ub32t05ZRskgUudQAeN9wLA+AJ9Ru7wZe82xFrmB2NIMDN7CtPmVL6LpjX1gg+NPrk2sxUXIi7QJsCWF9mA2wrcLceLFAEdHaVNlCjZ++5YmwNzuT7Re9sc2aK0tFFI28iAaiDcEE6CfPUAj3/bhoasdBiwMkjt256lcVn5EuHkHwzQI27brrz+uaocySbE7KoOyKOSjx8Se8+7CVPSW3W+30+0Ed4x1asea8aYOgV0WI7VK4k7nnalDo04x65erlYCVbWJIHWDx3+UO+3t77A6ED/F6j/rLNf8AxSqxFcsCnmMPj4LaWoprf9IZh/3yQYzcuFrDqb16LquDZ75h4b+g591K+DBgxSW8jBgwYEJKosljWomqAT100UEUnauuiBp2isLXBvUSXN99vDDebo0gCoEkqICsRgZ4ZjHJLCXZ+qkYL8h5HKSIEli1vodNbX48pzKKPNa/XJHGTTZf67qpI1V3ziLjDTy3jGrata8wBFfNTCjaalCGGPWI9MQg9NE0kCLXh2lKlXO3VlbCFIlRwHAwmHbtPUUtS/b/AM5SClENiQTp/iUOoG5btbjVti/R7TGNImVpIklq5tDkFXat9J9IRxbtRsKyYBdrArubbxJWdINQlGZ4q/0mokymsq5Y+rpyKOohjRlKKkIaJYp3ekMFUZWJQAnXFMXXuKOIqimlqqZqyd9ssaKXq6IVAermrEmhV5EhpIo3WiGl5ldkMjhA7tCgLRScK9GAMsgklmmgemgiVpJ2aojkp6lpoWRtIAMLCN1kOp2dbyFyLlRTo5hGo9bUCoecVXpHXDrxMKZaZnHY6so0ChDEYzFysg0ppZHAfE1TVPDEKx9Aqa1HkjNJLJIlOtM8cbSrTdSSjTMjNFGt1FiWPbLeTi2YrT1Yq/Sa45bmFQ1G0cLClnVIS8SxwRpUBaeXVTNDMzyOYgAVcSF0QptquD4mo5aNmleGaOaORnlZ5mE+vrW6x9R1EyMRtpXYKqqoUaYuE9162pqKjTJHIglMAAeMki3VU8ZN77gk7Da2+ETo5zotUywpWnM4BTU8/Wn0djHJK0o09ZSRRQsk8aLMqaSybm+iSNVY9ZnjS1VC71XWVK5pXqMv+IAjFPRZnHEAgiFSG6rqXZ5JGV/SA4AV4QqoUoZZwDBHHFEpk0Q0TUCXYX6hhECT2ReS0KdqwHPbfCdP0Y07FrSTqhlp53iWUCJp6ZacQzONGosFpYLoW6s6A2jV2sNboT4tqZ5YesqFqFmo2nlQyUzPHKHhCmKOnhR4IQZJoWjqXdwY4wCXSdmcfAGZRpV5srSIjHMI7KzqrG+VZaBYEgm52Hng6oTl4t4bWfQ3WSU80JZop4WQSx6l0uLSxyQvG6+sksboSqtbUiMqDD0cwCzCeo9K65qhqrrY/SZJTAKdi4EXUaPRwkQjWFUQBSqqe1iLKvi+qlin6ydWE0GZrPStLSkQrDTz9mKKKAVSSU8qRQuZ5WU9a5btGEDqObSRS1nVSxU7fx1xLIqaY3TLMm0Ss7QytGiaiWOlk5M6SBAuAdUKVMh6OKaFCg610MdTEwklZyyVdQ1ROGc/GMzSO1nLFrHck7494U4QihmEqzzTyrAacdbMrqItaMqhFRVBQpbWBqYHtl7IV4OhfPHlSoSSSWWSGYKTI9LLpDQxyBEmo1SOZO3rBeOOVQ4DLbSzRPw1kmmDKXeGkp4HqIGNbFBesSRZNcMUj9jqlrWX0R5ryA9b1RQ9eHQQpoquAIWmEuudQJ1qupWUin9IW3xui1+3a7R36ssTJpEh6zG7hngiKnZTHJP1causMDTMaeFXILLGgAuq2CoJDIIU7EfVqdOI4pOMp9UTitDzzT18U1IyQmOkSnhqmV+rijFWDSyQU6SPJIwl68iwMsARJyvjmpMMkfphMymgcyPUUDQSR1JnXTS1kNJ1SNUtAdKVdJGwPZQr10bRIhS5n3CAkmM0VRUUc7oqStTtD8bGhbQHSognjDIXYCVESWxC67BQOb/JrShY0UOixdSVAckloa2OtDuzhnkkkqYg8jsxZ9TknU2rDe6I+JmqJIiWeT+LVCs00dMJy8Fe8DapKW8TgGMrqhYRSW1hVJ0rKeFCE026PqYvWOys3p6KlQpkfRpVGS0QUgwFtRdmjKsXs99QBxlR8DoCGeaoncSQyBpZtVuoLmJAoVY1UGRixVA8lxrZtK2dWDAhMuLo4hVKVI5J4TSU5pYnjlCyGnYQ64nYofXNPE2tAkilLqy3N1vIskjpqVKeO4hghEUeo6mCImlQTteygC/M2ws40Zj6j/mt9hwIXza6KOkH8GSQTI/WKYk66Jd9aMBrjvewZWBIJIKstiLE3uzw3xclQkc0WmWmlQusitY31ABNBBNwRIGuwKMmkgknT82MngvY9wAPv7hiVOCOmapoKdqeJY3UuXUyBm6rUO2qqpUdtu32iQGLmx1bZeXwrGy7MzATyvr9U91xbRk3z9FGXH+ZiWeVxsrOdI+agOmNNtrJGqoLdy4bsL2ONlQ18aMaTW0KTSbSrVLscJWFOne6+631YTohuPbhyClBYOzb9744ZFw5ctpNYaysxHzVYgbeQsD7cZ8J5otPKZT6yiydlWN25kBgbMLWB2tc774EFduT5LKIQ5QqjqyEnbfcq1ue6sAPfjh4DppHL2UspW5PmvcL8zYk7YVZuMHqFaFR1aWDLvdi6G4ubbAjuHncnCFw7xE8TBVsF1G+24DbNbw2J5jbCUlU79DtRrVoSSLaxcesFlicEgcyQ2o+8Y5OFeNalUWNwmYUZ7QDlY235vCwAankPMizqeTq++E7ocrtFbCDykbqz7Wtp/2gF9+IyqOJHglk07/GSEqb6SdZBJ7wfZvigcdrnuBHYpJWaqtS1x9kR6l54kjelYX6zqgKmnc8oKkJMVSQnZZkV4pLXBG6KyuG2lRSURqlWu0sCK7aVCsvWtoSRQlnsWPsI9U4VOAelZUa7KVupSRGAkgljbZ4ZUIGuNxsVI8CCGAYTdwZWwBvSqONZopbxEJGgnikJVzTSPFEJGa4Voy9+tSxtqDaXskdD5endVvAZX5foqv00BeVgg0D1irttGg9ZpG0qRGt7XtqNwoDMQC7UnnqKkpESgZmZSjdXFo3aSeUgjR8uVy19NyovZQbZ5jUQurrVwoY1Ul0kemk1MouE0M7SEhuyB1d+8W1YbmXvlUV+oghhL2DFYm1kKyuqk9WNKh1VwFsNSqTcotlGU4mmqUsi0b81DPDvBkbs6LFUSHk0zxlopG73CCNZ6dQwJU6p5GBuyxm6jHNuiidG7MVQFPJliM0R8N17aX56JArjvAxZihzCIoLMR2bxqEa7KO9V030DuIFjZreo1jLeJKcHU8hVfJHJbyIVSdPjtviLxnA2HUVAcVruqqLxvwHUxw3dGIU9gmKRLFiARcoQqtsSC1r788S1S9FscUEcE0bVVKUjM3xqpNDMisXq4C2hY4u0IGh6zVKO1a6s7uPp56SaMwimheQSTOup0hchFR0cromeG5kNlupOlb7XsCzIlkSj61nkqIBadwpOllNTJGbRSiJ40Kpq6xgyOS4UhUJOZxbIynBpY4bc+hPaiK3VzFjbGNO5/b/AAmvnfwc63+UpBHVQuNSASLHIFO6gdey6xo0nUzKx3BUEHDa6TuCoaIRU6v6RV6C9Ww/k42OnRBFsNlGoux7RNvVHZxdDhbiNZE1KCiFiqbqVdQLq8ZRmDI62IGzAgggEYiXirhRcwnlnjjUNKUQXhhaZGAEZkYvKqyKHRrrG4Okald7acZ/DeL5E8miXbTz9T0v5dvmr82M1osFVm4YyQysxLCKCJQ80zglYULBRdRYySSN2I4lOqRtrqod0Wc9z5XAijUxU8RPVxEgvdgA08xWyyVMwA1OOyqhY0siAY6+PKaaNVjWmkhpIbtY2kR5LMktTNNEvVNOSGjte1OFaIBbOGauWZVLKfiEkmYfJiR5HAPyCkYZyp5jbb2Y6kEO3KqLhoW7VjyPZPsaxv8A0TZvauHxU/8AMwd7rGp5fNt+zCZH0b1oF3p2hBG/pEkNMfaRVSxEe+2HtBw0Wp9Lz0kd42ufTIZtyGJ2o2qGsDfkp5bXwSSNFJzDSanR9U31A/vvh818ZsrqO0huo+cD2Wj9jg28m0n5OGvwFS0ccljVvOSDdYaRlS+xt1tVLC491O1/qxI7ZrAEOmn1WN7zTu/Ijf4gU1vGxv7TiCR5B5JHOTy6LuM9KejuT6PKLIx30a+RtzA5EW5HYixBRyzwMrFSLFSQfaP1Yh9853ICxopOwWNbC9ibFtTc2v6xO5xKHCmaNLEGdi7m+ok7khiN/O1saXDZpLLfmuX+Iom6GydbpdhGE2eotKo8R+3DkNEdOrSdHztJt4bHv92GdmdvS4lvYHSNzYC7Ebk8h5+GNWSTbfusLCiJedv6Sl2+Hz8F7/mU3/WOY/8AfJcYjgRFjLyziM91rFQfC5PaPkLYy+DHKnotSsbiRVzLMbEb7GslKEm1rslnsO5ge/EOXK142WxwTFlheTJ1A679VKuDBgxQXTIwYMGBCaWe5xSemQ0ksStUTxu8bNCjIRHqJi1tv1hVZJFS26xyH5JwZLxfSSvVOCsfokpppppAkaaltqVZGO8auTGbkDWrDuwg9J/D9S8xqKaMSTQQwyU4LookmiqWaSnuzDqzUUzy0/WEaVEpNza2GzVcD1cKaViao01FM5lQUTVbFMuaKWrplrGWkSoarY6jOB8W8xRSSmEQpgpeoYEr1TLMpcldBEq7AubbSL2gNW47Q8ccubZlTgdrqpGkUER6oi8qL2hYOyh0UPr52AN+/eKOBuFa2lNPI1M87KM3hdeupFdfT8xhqqeokKtFB1WiJutECalZhohI2GXCnR9UJRMjwAVP4OyqnHbhLdZTRFZ4wwcgCNmYXuFa50lsCFKtXmVPGZAvVtNDG0rQo0In0hB8lnQLqVEQM5RdlBYAbdCVNOJtAMIqWGspdBOVtbXpvrIsLarW2xD3EnB1S1LWUooElmcZxJHWmWAK5rUq/R0i1SekCpYVENNIJkjhREe0hCRqVeq4SnMpi9GGo5nHXjMNcNhEkySGIjWKsVHo6tlgURmIwEXksTHhUJ551xJHTTR00dPLLLLHLOEp0hA0RvEkjsZJYlvrnjGxJN/I4zyjiakkjWo1Rxa5GhvMFilE8MkkLwHrLN1sciyR2BN99JKsCUrinhWSbMaeYNLFBHRVcTSxSKjCSSoonjjI3cqyQytcLpBQXIJALR466OZI5IvRUqJKcU88LJC+XyTdZPN100kzZxDMJI6tmJmeN+t1It0mBHVohSytVAjldUUcr6nZbosjBQCzkXDMFDKSx5BhfnhPM1LIY5VSGoErlRMnUOoKRu+suWuQvVaOxrYEjYKGZY0q+jCY0lXEFtM8eWosrSRSyzx0cVP10LSywaZOt0TwFp6cRuZWZoijsp05dwRVFklMdRc1jSyCpky0TaRk9fSrM0eXRRUys0s8EG0k8jIsZbq1QhRClSfOKNdTtLTLrYws5kiGtwLGFmLdpgDYoSSPDG3P8whpwjun8pNFApVFJ11DpEt+XZJ0Bj4Ac7DET8T8GVIgpYYaVgFy8UzNTjLg6SFI1kgneuEirRuETU1LFLISh+bHd0cQZTUfg2hVYJJaiB8uklgV4BL/ABeSF5kDyTpAzqEYfy2liNmNwcKEJ81VTDTx3ZoqaIG12KRRgsdhclVBY93eca62sp0Cxu0KK5VURmjAYsSUVVJsxYoxUAb6TbkcMfPOvlmpqs5dUOtOtVCaSSSgM5acUxjrI7Vz0pEaxTU51zJIFnksCCQ8e03C8qSVFKaGOqmkyelgAWSHqaQT1mblIS0xjc0cIKRloEd7U8emJttIhT6aqBZtGqJamRb6boJ3Re/TfWyrbnuBbGqgkpn6yKMwPpa00aGNtLHmJEW9mNvlC5tiI6vo+qvTWv6VLHJWUlV10cmWpT2p0pwWmaWlkzQTL6OyCOEtHIjIhkiWSXQsdFvDFRDVktA0NMkM8fxzUc2hnnjdEoqiBVrWpXCvJKtcqtqWDSCQ1hClWKnUWsoFhpFgBZRyUW5L5csbcGDAhGDBgwIRjGaO4IPIgj6cZYMCF8g6SpAUWFthe5Jvtv4czc27vO1yST322A8ALDDlzPg4xU8ExPZljRhZr21KDYgr+s4brUvffDCkBvdIJx5gwYelXVSybEeRxqpR2hjBWx7DJY3wIUp5Jx0sNKI0T44KV3AK7i2rmNzzN97+PfHOZm+/juf39+NPpp8vr/bjXJUEi22BKs6GoKOGHNT9PiPeNsbmlBlJ5qXJHdsW/YccUj3x4pwJFL3Dk6/FyBtDqVdSVuodCCLkHVbUvcpw3uO+FJHmqZ4AKiAyyyEwsHeJGd2BmiHx8KqPlyRqn5WGzRcQug0gKRvzB7zf5w78amzyTrBKrGORTdXQsrKQbgqwbUpHiCMQtjIdaeXWFopNicPDgjPZIjIElaAyR6Nau6Ddhs5RgdBTWlyG0lw21iw7OGumKog1nqaSaSQATPLAxaYKSUMgSRI2ddTfGaQ76u2zkAhv5nxeHlEvolLEL3aKIVEcL7EWKrU3Qcj8UY7kC997uc20wHdPaiymoVQ4kMMKnTI8sjRwo4GrRtqaRmUghIVkkKkMFKm+OvPuIliVZIC1Q3Iyy3EaEXGqOnJIYm/r1WtdlvAjYY1d0hTSKVlSKVRYRAoyCBR/m4VheNY4v9XYre7W1MzNwU/FJUMBDD2hYkiVx7dMkzRkjuupxA2FwN/yk4kJwZ3ncpJfr5pCbMWeVyxY7i51Dcbey3kMSx0C9LMLzCOvio9IDNJWTiQTsFuQDo1RzTtcAEojOFu7u4GuudTmTNe9t/AW8AAALAAAWsBjmimI5YkfAHNootXdq+m7JpgmqkjcxMHj65YlCFT2XF1uDv6uk35b40v0jZfVzdWpbVNIhERdGgLBREnZCqQgJvvub2J0gRilhqz5Y6Muzd42DLba+xvp3FuQIxF+CjIAIuu6LN819DpOhyndTYqga5bqrgElSpbSrKgfSSNfrDxw5ujjg0UMIi6wzKEWMExWOhQdIfQzqSATdgFXfkL4oDk3TZmMP8nOVF72a8g9gEpew9lsLsvwks0IsZI/P4vn7i2nflsBh7sdjiCW8uSQkjkVNXwmM9y/sPSVnU1yOO3TNIwvq0GOV4TZQlmc3LOgSwRtQRobzXNqlk0vmNOIrEdXHLKsBvz+IoqXq9Xn1e/j4xnnfFUsxZpNJZ3LswBBLEkk7GwuSdgLYTfwg3l9f7cOEW26dqpPynoaVSL1Oo33FNSO497VclE1z5I1vPDziyQNEstNIZ41VjLGyCKqiQOeslaISSI8Ca+1JDJJpDapFiGIUpszIIJVXA+SdQB9pR1Ye5hhZj47qFaNo2EBibVF1V10Pe+sdonVva5vcXBvc3Y+AnknB9LRlVWY5A3eNx+/1YlmizS8ZI5Ebe8Yh/POITLK8vVRRa2LmOJWWJSd2CKXYqpa7aQdK3soVQqjsoONZUXSFjI81a/1OMOcx3ZIHBSDmucs5jVDaVnCqCOyWIKLc223e/Pu9mLQ8NrRWeOCc9bGzlke+liLAojMqkMSt17T6u0PArRT/lRJrV7JqRg42a1wbj5XK+FbL+kiojHZEd7WvZ7+24kG+JYdUZsKpmY7ciMsIV28/wAxMcZb1gCBYnle/t7+4YYFfW6qyD8oRHbzGIMzn4QFbKmho6YC4N1SUHb2zkb38MIkvS3Ul0cpBdAqgaZLWW9r/G37/EYvSTNcQsLD4VLEw6qsgjn9FbbpJz5YIAS173GkHx5J+SXO5/JG/hiPOhnpcnyyGr000NSsk8tSxNTJEygguRpFLIPVHPV3eG+II4n6VaioUK6QrpNwUWQHly7UrC3fsB9Qsmrx3MI5I9MdpRZms+q1gLD4ywAtflck73FgEkexz/SvurvD8J+PF/8AV7+3/itbJ8MqUf8A6dF/fX+547eHvhbVNRKkMWWRvLIbIorWux52H8TsLC7EmwABJIAxSr8LN4D6/wBuFngnjuejqI6mHSJIzsCG0Mp2aNwrqSjjYgEHzGKo57rYdVbc1ffi3porqYqr0dE8hUsUjzCoZlUC5Lf/APMAA2PInkTyF8YZ/wBNldAut6Oi0gDWRmE9o2YXETk5aF6/cfEqWk/JsCRUD+ELX9cJ9NOJLgtaN+2Rb1j1uoWAAUIyiP5Gi5u1OOOkyrrZetqJOsYXCLuI41PyI0DBUXYchdrXYsd8SSuaANAs+pTMRhc//fdTfQWV9AabpMzBollFLlwiZFcMcyqAulgCCT+CrDn488RlxB8LWeFyrZfC3gVrZNJ3ttqoVP0gYqRk3SPUxJ1YYPECWVHLlEJ9YoOsAXUdzbvueZN9OdccSSjtRxbd4VwfrkOI/wCn1TxTZT1b06FWkT4a8hNvwan99b7pjf8Awzpf+jY/7633PFOGzNrk2W557H6efM49/CreC/X+3CC+qWSr8vJXU4f+F7LNKkXoNPFrLDXLXusS6UZ+0woWIB06QbHcjkLkSlwv0jZpUrrjy+k6snsO9fUIHF7a1DZYG03B9YKTzAsQcfP7o26R5qCoWpijgkkTVYTI7J2lZSbLKhuAxsQRviXx8M3NPxeg/san75h1ilEQ6+auH+Gs2/Fct/SVX+rJ8YtnubfimXH2ZjV/4PioA+Ghmn4vl/8AY1P33A3w0M0/mKD+xqfvmETlcNM3zc/6Jl36Sq/8Ixn+E83/ABTLf0lV/wCD4pjJ8MfNT/m6IeyKf71jlqPhc5o3NKX+pUfesIjdXY/Ceb/imW/pKr/wfHjZtmw/0XLf0lV/4PijcnwpsyPOOlP9Cf71jmk+EzmHdHSr7I5f11BOFSEu7K9JzvNfxbLP0nV/4RjFc0zW+oUmWXIALfhGquQCSBf8D3IBZrDuufE4oa/wi68/Jg+ioP21Jt7sKGUfCjzOIWUU5F72ZJmH0GosOXdbAmgv7K71fxDm6KW9Dy97fJXMqnUd+7VlKjbnzw0M66Xs0h9fKo7fOWqqmX6RllsVqX4YeZ/zFB/Y1H3zAfhh5n/MUH9lUffMCUh3RTbP8J6oX1qKlX87MJR9tAMaG+FVKP8ARKP9IyfcMVx4t+EPVVQIkpaEajdikdQpJ8T/ABsi/na+I2m4lkLEgIt+5QdI9gLHAfRDdXVXOqPhaygf8yp28lrpif8Aw+304Tz8Mef/AKLX++P9zxUD/lLJ4L9Df+rGQ4mf5qfQ3/rwwl3ZSABW+X4Yk/8A0WnvrSPtpMeR/DHnJt+DEJva3prb+y9JviocPEzgglI3HerB7HyOmRWt7CDh5cMdMj0/8nQ5fqvcO0VQzg92/pfdhRq6pD6J1ZZ0K51VQxn0ZurjVY0WSaKI2Cgagkkga3dcgeQ2wofwYs3744B//cg29yq2MqX4YOZqLCCgt/8ASqfvmNx+GTmn8xQf2NT98wtJu6rdgwYMKnIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQv/2Q==\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"960\"\n",
       "            height=\"540\"\n",
       "            src=\"https://www.youtube.com/embed/_XtuvAax4fs\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x23ad7c69438>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "YouTubeVideo('uSZE9gnheSI', width=960, height=540)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa70af2c",
   "metadata": {},
   "source": [
    "# Discussion\n",
    "\n",
    "\n",
    "Differently from the works found in the literature, which train and validate their models only against a RandomPlayer, our methods were trained against a MaxDamagePlayer and validated against a MaxDamagePlayer and against a RandomPlayer, showing significant improvements. All of our implemented methods showed better results than a RandomPlayer with the same battle configurations.\n",
    "\n",
    "Our Monte-Carlo Control 1st Visit agent, in a stochastic environment, averaged $34.91\\%$ wins against a MaxDamagePlayer and $92.7\\%$ wins against a RandomPlayer. In a deterministic environment, the agent averaged $39.4\\%$ wins against a MaxDamagePlayer and $90.57\\%$ wins against a RandomPlayer. In a visual analysis of the battles, we noticed that the agent promoted switches to better type Pokémon, to not suffer super-effective attacks, and to Pokémon with super-effective attacks against the opponent. Compared to the other implemented methods, the agent with Monte-Carlo was the one with the worst performance against MaxDamagePlayer. We hypothesize that this occurs because Monte-Carlo methods occur through sampling, acquiring the rewards and updating their tables only at the end of each episode. In a battle against MaxDamagePlayer, this can demonstrate a negative performance, as MaxDamagePlayer always selects attacks with the highest Base Power, which can cause the end of a battle very quickly, making it difficult for the agent to define the best strategy. This lack of bootstrapping, however, allows the method to have a good convergence, performing well against a RandomPlayer.\n",
    "\n",
    "\n",
    "Our Monte-Carlo Control 1st Visit with Function Approximation agent, in a stochastic environment, averaged $83.46\\%$ wins against a MaxDamagePlayer (improving by $48.55\\%$ compared to the agent without Function Approximation) and $92.7\\%$ wins against a RandomPlayer (improving by $6.52\\%$ compared to the agent without Function Approximation). In a deterministic environment, the agent averaged $60.05\\%$ (improving by $20.64\\%$ compared to the agent without Function Approximation) wins against a MaxDamagePlayer and $99.29\\%$ wins against a RandomPlayer (improving by $8.72\\%$ compared to the agent without Function Approximation). We noticed that our Feature Vector performed very well in generalizing our states (out of $1,016,064$ possible states). As in the method without Function Approximation, in a visual analysis of the battles, we noticed that the agent promoted switches to better type Pokémon, to not suffer super-effective attacks, and to Pokémon with super-effective attacks against the opponent. \n",
    "\n",
    "\n",
    "Our Q-Learning agent, in a stochastic environment, averaged $41.69\\%$ wins against a MaxDamagePlayer and $90.04\\%$ wins against a RandomPlayer. In a deterministic environment, the agent averaged $39.84\\%$ wins against a MaxDamagePlayer and $85.36\\%$ wins against a RandomPlayer. In a visual analysis of the battles, as previous methods, we noticed that the agent promoted switches to better type Pokémon, to not suffer super-effective attacks, and to Pokémon with super-effective attacks against the opponent. In addition, the agent made good use of Pokémon abilities to lower the opponent's attack stat. The agent also used Roost (recover Health Points) and Sleep Powder (cause status to the opponent) several times. This is because our rewards have parameters in these cases (HP value and opponent's status). As Q-Learning looks for the best reward in each state, the agent took advantage of the execution of these moves to maximize its rewards.\n",
    "\n",
    "Our Q-Learning with Function Approximation agent, in a stochastic environment, averaged $83.84\\%$ wins against a MaxDamagePlayer (improving by $42.15\\%$ compared to the agent without Function Approximation) and $99.22\\%$ wins against a RandomPlayer (improving by $9.18\\%$ compared to the agent without Function Approximation). In a deterministic environment, the agent averaged $58.73\\%$ wins against a MaxDamagePlayer (improving by $18.89\\%$ compared to the agent without Function Approximation) and $99.34\\%$ wins against a RandomPlayer (improving by $13.98\\%$ compared to the agent without Function Approximation), being the agent that showed the best performance in an average of all executed battles. This demonstrates again that our Feature Vector performed very well in generalizing our states. Furthermore, we hypothesize that, as Q-Learning has bootstrapping, it was able to generalize well the seen states in order to maximize their rewards. In a visual analysis of the battles, as previous methods, we noticed that the agent promoted switches to Pokémon with super-effective attacks against the opponent. Also, the agent made a lot of use of the Giga Drain damage move (promotes damage and recovers HP), even when this move wasn't very effective. We hypothesize that this occurs because the Player realized that, even if the move was not very effective, it promoted a decrease in the opponent's Health Points and an increase in ours, promoting greater rewards.\n",
    "\n",
    "Our SARSA($\\lambda$) agent, in a stochastic environment, averaged $54.79\\%$ wins against a MaxDamagePlayer and $90.94\\%$ wins against a RandomPlayer. In a deterministic environment, the agent averaged $52.93\\%$ wins against a MaxDamagePlayer and $99.34\\%$ wins against a RandomPlayer, being the best performing tabular method against MaxDamagePlayer. In a visual analysis of the battles, the agent made a lot of use of non-damage move Leech Seed, which removes the opponent's Health Points in each battle and adds them to ours. This is because our rewards have parameters in these cases (+ our HP and - opponent's HP). The agent also used the non-damage move Sunny Day when the opponent was with a Water-type Pokémon, causing the attacks of this opposing Pokémon to decrease its Base Power. As requested in the project specification, we use the same step-size and exploration schedules as the Monte-Carlo method. For this, the same base class was used in both agents.\n",
    "\n",
    "Our SARSA($\\lambda$) with Function Approximation agent, in a stochastic environment, averaged $83.46\\%$ wins against a MaxDamagePlayer (improving by $28.67\\%$ compared to the agent without Function Approximation) and  $61.21\\%$ wins against a RandomPlayer (decreasing by $33.63\\%$ compared to the agent without Function Approximation, being the only case where a method with Function Approximation performed worse than the method without Function Approximation). We hypothesized that this happened because, as SARSA($\\lambda$) is bootstrapped, the method may have become biased due to the training being done against a MaxDamagePlayer, and thus, our Feature Vector was not sufficient to generalize well with a RandomPlayer. Also, as the environment is stochastic, it's possible that our Player missed the attacks. In a deterministic environment, the agent averaged $61.21\\%$ wins against a MaxDamagePlayer (improving by $8.28\\%$ compared to the agent without Function Approximation) and $99.13\\%$ wins against a RandomPlayer (improving by $9.65\\%$ compared to the agent without Function Approximation). The good performance of the agent in a deterministic environment in relation to the method without Function Approximation corroborates our hypothesis that our agent may have missed attacks due to the randomity of the stochastic environment. In a visual analysis of the battles, as previous methods, we noticed that the agent promoted switches to Pokémon with super-effective attacks against the opponent. Also, the agent made a lot of use of the Giga Drain damage move (promotes damage and recovers HP), even when this move wasn't very effective."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "597160ed",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Given the complexity of a Pokémon battle (as shown, $1,016,064$ states and $9,144,576$ possibilities), the implemented methods were able to generalize the states and explore the best reward acquisitions, with all methods performing better than a RandomPlayer. The methods showed increasing learning, in view of the increase in rewards and victory rate as battles went on.\n",
    "\n",
    "Particularly, the methods that used Function Approximation found it easier to generalize the states, given the influence of our Features Vector, since the tabular methods find it difficult to generalize the millions of possibilities in a few battles ($10k$).\n",
    "\n",
    "In addition to the difference in training and validating using a MaxDamagePlayer (getting good results), most of our methods presented better performances than related works.\n",
    "\n",
    "As future work, we propose the exploration of specific parameters for each method. Following the guidelines of the project specifications, the same parameters defined for Monte-Carlo Control were used. However, some methods can take better advantage of other parameters, such as SARSA($\\lambda$), which is very dependent on the value set for $\\lambda$. We also propose to consider in the observations the types of Pokémon, their abilities and possible side effects of moves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d95c90",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "acb23db6",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "90f9584a",
   "metadata": {},
   "source": [
    "# Contributions\n",
    "\n",
    "## Methods\n",
    "\n",
    "### Monte Carlo Control\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### Monte Carlo Control Function Approximation\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### Q-Learning\n",
    "* **Code development**: Henrique;\n",
    "* **Code review**: Bruno, Leonardo, Maurício;\n",
    "* **Experiments**: Henrique, Leonardo.\n",
    "\n",
    "### Q-Learning Function Approximation\n",
    "* **Code development**: Henrique;\n",
    "* **Code review**: Bruno, Leonardo, Maurício;\n",
    "* **Experiments**: Henrique.\n",
    "\n",
    "### SARSA($\\lambda$)\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Leonardo, Maurício;\n",
    "* **Experiments**: Bruno, Leonardo, Maurício.\n",
    "\n",
    "### SARSA($\\lambda$) Function Approximation\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Leonardo, Maurício;\n",
    "* **Experiments**: Bruno, Leonardo, Maurício.\n",
    "\n",
    "## Video\n",
    "* **Editing**: Leonardo;\n",
    "* **Review**: Bruno, Henrique, Maurício.\n",
    "\n",
    "## Report\n",
    "* **Writting**: Bruno, Henrique, Leonardo, Maurício.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79867a85",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}