Reinforcement Learning with Frozen Lake Game.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Frozen Lake: Reinforcement Learning Model based on stochastic Q-learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Abstract\" data-toc-modified-id=\"Abstract-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Abstract</a></span><ul class=\"toc-item\"><li><span><a href=\"#Problem-Statement\" data-toc-modified-id=\"Problem-Statement-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Problem Statement</a></span></li><li><span><a href=\"#Game-Introduction\" data-toc-modified-id=\"Game-Introduction-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Game Introduction</a></span></li><li><span><a href=\"#Actions-and-states-in-the-game\" data-toc-modified-id=\"Actions-and-states-in-the-game-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Actions and states in the game</a></span></li><li><span><a href=\"#Rewards\" data-toc-modified-id=\"Rewards-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Rewards</a></span></li><li><span><a href=\"#Solution-approach\" data-toc-modified-id=\"Solution-approach-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Solution approach</a></span></li></ul></li><li><span><a href=\"#Building-Reinforcement-Learning-Model\" data-toc-modified-id=\"Building-Reinforcement-Learning-Model-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Building Reinforcement Learning Model</a></span><ul class=\"toc-item\"><li><span><a href=\"#Importing-Libraries\" data-toc-modified-id=\"Importing-Libraries-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Importing Libraries</a></span></li><li><span><a href=\"#Obtaining-Frozen-Lake-environment-from-OpenGym-AI\" data-toc-modified-id=\"Obtaining-Frozen-Lake-environment-from-OpenGym-AI-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Obtaining Frozen Lake environment from OpenGym AI</a></span></li><li><span><a href=\"#Defining-a-baseline-performance\" data-toc-modified-id=\"Defining-a-baseline-performance-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Defining a baseline performance</a></span><ul class=\"toc-item\"><li><span><a href=\"#Step-1:-Initializing-the-parameters\" data-toc-modified-id=\"Step-1:-Initializing-the-parameters-2.3.1\"><span class=\"toc-item-num\">2.3.1&nbsp;&nbsp;</span>Step 1: Initializing the parameters</a></span></li><li><span><a href=\"#Step-2:-Training-the-agent\" data-toc-modified-id=\"Step-2:-Training-the-agent-2.3.2\"><span class=\"toc-item-num\">2.3.2&nbsp;&nbsp;</span>Step 2: Training the agent</a></span></li><li><span><a href=\"#Step-3:-Making-the-agent-play-the-game\" data-toc-modified-id=\"Step-3:-Making-the-agent-play-the-game-2.3.3\"><span class=\"toc-item-num\">2.3.3&nbsp;&nbsp;</span>Step 3: Making the agent play the game</a></span></li></ul></li></ul></li><li><span><a href=\"#Tuning-with-hyperparameters\" data-toc-modified-id=\"Tuning-with-hyperparameters-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Tuning with hyperparameters</a></span><ul class=\"toc-item\"><li><span><a href=\"#Testing-with-different-hyperparameters-(set-1)\" data-toc-modified-id=\"Testing-with-different-hyperparameters-(set-1)-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Testing with different hyperparameters (set 1)</a></span></li><li><span><a href=\"#Testing-with-different-hyperparameters-(set-2)\" data-toc-modified-id=\"Testing-with-different-hyperparameters-(set-2)-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Testing with different hyperparameters (set 2)</a></span></li><li><span><a href=\"#Testing-with-different-hyperparameters-(set-3)\" data-toc-modified-id=\"Testing-with-different-hyperparameters-(set-3)-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Testing with different hyperparameters (set 3)</a></span></li><li><span><a href=\"#Testing-with-different-hyperparameters-(set-4)\" data-toc-modified-id=\"Testing-with-different-hyperparameters-(set-4)-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Testing with different hyperparameters (set 4)</a></span></li></ul></li><li><span><a href=\"#Changing-policy-to-'Mean'-of-the-action-values-at-that-state-and-observing-the-performance\" data-toc-modified-id=\"Changing-policy-to-'Mean'-of-the-action-values-at-that-state-and-observing-the-performance-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Changing policy to 'Mean' of the action values at that state and observing the performance</a></span></li><li><span><a href=\"#Changing-policy-to-'Min'-of-the-action-values-at-that-state-and-observing-the-performance\" data-toc-modified-id=\"Changing-policy-to-'Min'-of-the-action-values-at-that-state-and-observing-the-performance-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Changing policy to 'Min' of the action values at that state and observing the performance</a></span></li><li><span><a href=\"#Improved-performance-results\" data-toc-modified-id=\"Improved-performance-results-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Improved performance results</a></span></li><li><span><a href=\"#Questions-&amp;-Answers\" data-toc-modified-id=\"Questions-&amp;-Answers-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Questions &amp; Answers</a></span></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Conclusion</a></span></li><li><span><a href=\"#Author\" data-toc-modified-id=\"Author-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>Author</a></span></li><li><span><a href=\"#Citation\" data-toc-modified-id=\"Citation-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>Citation</a></span></li><li><span><a href=\"#Licensing\" data-toc-modified-id=\"Licensing-11\"><span class=\"toc-item-num\">11&nbsp;&nbsp;</span>Licensing</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Abstract\n",
    "   The goal of this assignment is to implement Q-learning using a Q-table. Q-learning is an off policy learning method, which learns from the environment and comes up with a policy, that helps the agent to reach the goal. The first phase of the assignment  gives a baseline implementation of reinforcement learning model , with a clear and step by step implementation of Q learning. The final phase of the report involves an implementation of Q learning using both methods. The environments namely FrozenLake-v0 is generated with the Openaigym, which is a toolkit for developing and comparing reinforcement learning algorithms.\n",
    "\n",
    "### Problem Statement\n",
    "   Developing a reinforcement learning agent model that automatically learns to play Frozen Lake game and acheive the best possible score over time.\n",
    "   \n",
    "### Game Introduction\n",
    "   The agent should start from 'S' and move across to reach the frisbee(goal 'G'). While taking steps, there are holes 'H' which lead to end of the game or Frozen ground 'F' which are safe to pass. The agent should choose path intelligently from start 'S' to goal 'G' to gain a reward.\n",
    "   <center>\n",
    "    <img src=\"game_table.png\" width=300 height=300 />\n",
    "</center>\n",
    "<center>\n",
    "    Fig. Frozen Lake game view\n",
    "</center>\n",
    "   \n",
    "### Actions and states in the game\n",
    "   There are 4 actions and 16 states in the game.\n",
    "<br><br>\n",
    "**States (4 in each stage):**\n",
    "* SFFF\n",
    "<br>\n",
    "* FHFH\n",
    "<br>\n",
    "* FFFH\n",
    "<br>\n",
    "* HFFG\n",
    "<br>\n",
    "\n",
    "S - start state\n",
    "<br>\n",
    "F - frozen \n",
    "<br>\n",
    "H - Hole\n",
    "<br>\n",
    "G - Goal, Frisbee\n",
    "<br><br>\n",
    "**Actions:**\n",
    "Left, Right, Up and Down\n",
    "### Rewards\n",
    "1. S - start state (safe, reward = 0)\n",
    "2. F - frozen (safe, reward = 0)\n",
    "3. H - Hole (reward = -1, terminate the episode)\n",
    "4. G - Goal (reward = 1, terminate the episode)\n",
    "   \n",
    "   \n",
    "### Solution approach\n",
    "   First the agent is trained over 10,000 episodes in which we update the Q-table (containing the action values of the steps taken) using the Bellman's equation and use it as a reference to play the game. The maximum number of steps taken during an episode will remain fixed.<br><br> \n",
    "   The terms in the Bellman's equation are pre-defined which are discounting rate, learning rate and the reward, state, action values are obtained from the environment during the episode based on the action chosen. The Q-table is updated with the result of Bellman's equation.<br><br>\n",
    "   An exploration variable 'epsilon' is also chosen to define the exploration ratio and how it should reduce over time is defined using the exploration equation.<br><br>\n",
    "   Finally the results are evaluated with respect to average steps taken in all the episodes(when the agent plays the game) and the score over time(when agent was trained to play the game).\n",
    "   \n",
    "**Size of Q-table:** (4,16) 4 actions and 16 states.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Reinforcement Learning Model\n",
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtaining Frozen Lake environment from OpenGym AI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "env = gym.make(\"FrozenLake-v0\")\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining action and state sizes as per the Frozen Lake environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environemnt properties \n",
      "\n",
      "Action size:  4 \n",
      " State size:  16\n"
     ]
    }
   ],
   "source": [
    "action_size = env.action_space.n\n",
    "state_size = env.observation_space.n\n",
    "print (\"Environemnt properties\",\"\\n\")\n",
    "print(\"Action size: \", action_size, \"\\n\", \"State size: \", state_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a baseline performance\n",
    "The parameters for the Bellman's equation for setting the baseline performance are listed below.\n",
    "#### Step 1: Initializing the parameters\n",
    "Initially the Q-table is started with zero matrix of size (4,16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q-table:  \n",
      "\n",
      "[[0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "print(\"Q-table: \",\"\\n\")\n",
    "print(qtable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "later the values in the Q-table are updated using the Bellman's equation which is\n",
    "<center>\n",
    "    <img src=\"bellman.png\" width=1400 height=700 />\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_episodes = 5000        # Total episodes\n",
    "alpha = 0.7           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.8                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Training the agent\n",
    "The agent is trained over 5000 episodes initially to benchmark as a baseline performance. Below are the steps run for training the agent.\n",
    "1. For each episode, and in each step of the episode, generate a random number from a uniform distribution and if the number is greater than epsilon(exploration rate) then the agent choses the action that has maximum action value (future rewards).\n",
    "2. Based on the action chosen, the agent receives a rewards as a value and the value is fed into the Bellman's equation and the Q-table is updated repeatedly based on the reward.\n",
    "3. After each episode, the epsilon is reduced at an exponential rate as per the equation with the decay-rate defined.\n",
    "4. Finally, at the end of all episodes the Q-table will have all the action values required to play the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****** Baseline performance of Frozen Lake agent ******\n",
      "\n",
      "Total episodes:  5000\n",
      "Learning rate:  0.7\n",
      "Discounting rate:  0.8 \n",
      "\n",
      "Score over time: 0.321\n",
      "\n",
      "[[2.01868798e-03 1.83881566e-03 2.77875168e-03 1.95429521e-03]\n",
      " [3.58470115e-04 8.88737740e-05 8.40307187e-04 1.94744987e-03]\n",
      " [1.37296322e-04 7.19128296e-04 8.42765935e-04 1.52908203e-04]\n",
      " [3.26880176e-05 2.01969030e-04 6.66864346e-05 5.54878723e-04]\n",
      " [1.08662775e-02 1.64942211e-03 1.37605798e-03 8.26887789e-04]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [8.95707868e-05 4.53410148e-04 1.98593111e-03 2.24638971e-06]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.75003151e-03 3.19647729e-03 2.30344232e-03 6.43107599e-02]\n",
      " [2.29726973e-02 2.09576238e-02 1.57425451e-02 6.44327905e-03]\n",
      " [3.78144999e-02 3.63525260e-02 9.87841987e-03 8.16926083e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.07557635e-02 9.82723363e-02 5.90263506e-01 9.72860703e-03]\n",
      " [5.06205387e-02 5.63086410e-02 9.13673690e-01 5.54839838e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.01\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"****** Baseline performance of Frozen Lake agent ******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score over time of 0.3062 is achieved using Q-learning. This is set as \"Baseline performance\" of the model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: Making the agent play the game\n",
    "Now, the agent is ready for playing the game. Steps executed by the below code for playing the game:\n",
    "1. For 50 episodes or chances, the agents plays the Frozen Lake game.\n",
    "2. In each episode, the specific action in the particular state is chosen in the Q-table that has the highest expected future value.\n",
    "3. In each episode, the last action chosen is printed and the last state of the rendered is printed to understand whether it is 'Hole' or 'Goal'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 46\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 19\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 40\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 59\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 69\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 40\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 42\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 38\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 25\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 39\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 45\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 20\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.78\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning with hyperparameters\n",
    "### Testing with different hyperparameters (set 1)\n",
    "\n",
    "The hyperparameters, alpha and gamma are changed to evaluate the change in baseline performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 5000        # Total episodes\n",
    "alpha = 0.75           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.85                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent at given hyperparameters******\n",
      "\n",
      "Total episodes:  5000\n",
      "Learning rate:  0.75\n",
      "Discounting rate:  0.85 \n",
      "\n",
      "Score over time: 0.3394\n",
      "\n",
      "[[1.13160078e-02 1.50824877e-03 1.38342705e-03 1.48666337e-03]\n",
      " [4.69567510e-04 1.92859777e-04 2.78123308e-04 2.78066509e-02]\n",
      " [6.82009032e-04 1.81063918e-04 2.56395844e-04 1.26152579e-02]\n",
      " [8.84992662e-04 7.44054542e-06 7.06267128e-05 7.53325585e-04]\n",
      " [3.65932089e-02 9.55403844e-04 3.78783202e-04 4.96932510e-04]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [6.06653021e-04 1.36825235e-05 2.04406503e-03 6.68053743e-08]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.77652663e-03 8.35450179e-04 5.92122799e-05 6.07809552e-02]\n",
      " [5.32533838e-03 1.78055707e-01 7.25641491e-03 1.10527660e-02]\n",
      " [5.37302806e-01 6.86925497e-04 2.56658627e-03 1.00407497e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [4.71663245e-04 4.17741634e-03 6.62523036e-01 1.26652604e-02]\n",
      " [6.30652167e-02 4.17437410e-02 9.61276811e-01 3.22237680e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.01\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"******Performance of Frozen Lake agent at given hyperparameters******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 62\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 60\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 19\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 90\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 57\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 32\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 25\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 64\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 16\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 54\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 66\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 74\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 61\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 16\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 33\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 47\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 37\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 39\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 48\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 98\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 17\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "38.88\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with different hyperparameters (set 2)\n",
    "\n",
    "The hyperparameters, alpha and gamma are changed to evaluate the change in baseline performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 5000        # Total episodes\n",
    "alpha = 0.8           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.9                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent at given hyperparameters******\n",
      "\n",
      "Total episodes:  5000\n",
      "Learning rate:  0.8\n",
      "Discounting rate:  0.9 \n",
      "\n",
      "Score over time: 0.412\n",
      "\n",
      "[[4.19579971e-03 4.79686919e-02 4.17395556e-03 3.45251385e-02]\n",
      " [1.15564406e-03 9.90869412e-04 2.60558183e-05 6.49700166e-02]\n",
      " [5.25989338e-03 9.04894061e-02 1.51259971e-03 6.46026658e-04]\n",
      " [2.29185933e-02 5.26476265e-06 2.21085249e-03 2.15189542e-02]\n",
      " [4.87802653e-02 1.74782518e-02 7.74458436e-03 3.84230598e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.35941697e-06 2.29432615e-10 2.38298754e-02 3.77839587e-09]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.15271380e-02 1.10587738e-02 1.96082680e-03 2.15926744e-01]\n",
      " [9.33867175e-03 3.34334736e-01 5.67540697e-05 7.59446817e-03]\n",
      " [5.31205024e-01 4.42216269e-03 4.34607292e-04 9.39520516e-05]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [8.15731494e-03 1.04999196e-02 7.20868124e-01 1.89721660e-02]\n",
      " [1.65532966e-01 1.72397048e-01 1.69949941e-01 9.78180994e-01]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.01\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "    \n",
    "print(\"******Performance of Frozen Lake agent at given hyperparameters******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 4\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 43\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 25\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 42\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 20\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 48\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 53\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 32\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 16\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 40\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 80\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 35\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 55\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 89\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 47\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 44\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 64\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 16\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 64\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Left)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 16\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24.74\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with different hyperparameters (set 3)\n",
    "\n",
    "The hyperparameters, epsilon and  decay rate are changed to evaluate the change in baseline performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 5000        # Total episodes\n",
    "alpha = 0.7           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.8                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 0.9                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.0075"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent at given hyperparameters******\n",
      "\n",
      "Total episodes:  5000\n",
      "Learning rate:  0.7\n",
      "Discounting rate:  0.8 \n",
      "\n",
      "Score over time: 0.293\n",
      "\n",
      "[[2.46591893e-03 1.27173914e-02 1.43329347e-02 5.76927960e-04]\n",
      " [1.68677139e-05 1.54512132e-04 8.83079117e-04 2.15058730e-02]\n",
      " [8.02604119e-02 4.30267679e-04 4.65529422e-05 1.50596490e-02]\n",
      " [2.95549758e-05 4.70940679e-05 2.54030966e-05 1.91780046e-02]\n",
      " [2.78508895e-02 2.17250252e-04 9.46577657e-03 7.30857425e-05]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.54498518e-01 1.95619494e-07 1.39166865e-04 3.08870430e-06]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [9.55038846e-04 9.09176005e-05 2.22041974e-02 8.55500529e-02]\n",
      " [1.48457496e-01 2.50422413e-01 1.59340501e-03 6.19490169e-03]\n",
      " [1.82500570e-01 1.75719493e-01 1.76832450e-03 9.28656741e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.70989556e-03 3.79872467e-04 6.22528082e-01 1.08407775e-03]\n",
      " [1.89521200e-01 2.45786790e-01 9.93993855e-01 7.14529128e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.010000000000000052\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"******Performance of Frozen Lake agent at given hyperparameters******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 33\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 58\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 42\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 43\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 57\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 25\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 25\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 20\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 41\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 69\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 44\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 20\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 16\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 54\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 34\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 42\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 19\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 44\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 49\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 61\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 31\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30.0\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with different hyperparameters (set 4)\n",
    "\n",
    "The hyperparameters, epsilon and  decay rate are changed to evaluate the change in baseline performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 5000        # Total episodes\n",
    "alpha = 0.7           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.8                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 0.8                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.005"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent at given hyperparameters******\n",
      "\n",
      "Total episodes:  5000\n",
      "Learning rate:  0.7\n",
      "Discounting rate:  0.8 \n",
      "\n",
      "Score over time: 0.2738\n",
      "\n",
      "[[3.32596938e-04 2.76879939e-04 3.75150322e-03 3.10437671e-04]\n",
      " [1.28398189e-03 1.95585622e-04 1.66492183e-04 3.85425278e-03]\n",
      " [3.78565798e-03 3.21847329e-03 5.72972227e-04 6.85931289e-05]\n",
      " [1.20497562e-05 2.16542260e-05 1.84136772e-05 3.13883956e-03]\n",
      " [9.21004581e-02 1.75528957e-05 2.78671261e-04 1.31928576e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.47381805e-02 1.02361982e-06 1.55233130e-05 1.76147344e-05]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [9.01712278e-04 1.34987038e-03 1.02822133e-03 1.16595749e-01]\n",
      " [3.45396178e-03 3.96192216e-01 3.45368609e-04 1.21009108e-02]\n",
      " [1.96460108e-01 4.96482487e-03 2.42965123e-03 9.72948467e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [7.88153671e-03 1.15052760e-02 7.47503370e-01 9.70988274e-03]\n",
      " [2.34856443e-02 3.34143416e-02 4.25134287e-02 8.01332492e-01]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.010000000013817982\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"******Performance of Frozen Lake agent at given hyperparameters******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 56\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 58\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 70\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 59\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 48\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 23\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 45\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 63\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 31\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 53\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 94\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 40\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 90\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 21\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 69\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 49\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 75\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 62\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 50\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34.32\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing policy to 'Mean' of the action values at that state and observing the performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 10000        # Total episodes\n",
    "alpha = 0.7           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.8                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.01  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent with Mean of action values as the policy******\n",
      "\n",
      "Total episodes:  10000\n",
      "Learning rate:  0.7\n",
      "Discounting rate:  0.8\n",
      "Decay rate:  0.01 \n",
      "\n",
      "Score over time: 0.096\n",
      "\n",
      "[[4.98243206e-11 6.88566746e-08 2.97705443e-10 1.58714057e-09]\n",
      " [1.38760635e-12 4.96834519e-10 1.86815630e-11 1.04298665e-08]\n",
      " [4.19965186e-11 5.67008113e-11 5.05512767e-08 3.89020911e-11]\n",
      " [3.93373830e-14 1.04225875e-12 2.00110277e-13 2.37802592e-08]\n",
      " [5.41031511e-08 2.12188990e-11 2.19384442e-08 7.28067814e-08]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [6.47383329e-10 1.27563565e-12 6.31201289e-08 1.10587017e-12]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [4.80034388e-11 1.48157899e-11 6.42321977e-10 1.34441053e-05]\n",
      " [9.56334382e-09 1.37933971e-02 4.04121202e-08 4.91458630e-08]\n",
      " [2.95085268e-02 3.37155174e-07 7.76470948e-09 3.34344676e-07]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.08201289e-07 1.09707624e-07 1.17157260e-07 8.58117495e-03]\n",
      " [4.52642396e-05 4.03550455e-05 5.31297201e-05 7.87358599e-01]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.01\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.mean(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"******Performance of Frozen Lake agent with Mean of action values as the policy******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma)\n",
    "print(\"Decay rate: \", decay_rate,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 1\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 4\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 1\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 4\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 1\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 1\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 19\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 4\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 1\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 30\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Right)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 45\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 2\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing policy to 'Min' of the action values at that state and observing the performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 10000        # Total episodes\n",
    "alpha = 0.7           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.8                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.01  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******Performance of Frozen Lake agent with 'Min of action values' as the policy******\n",
      "\n",
      "Total episodes:  10000\n",
      "Learning rate:  0.7\n",
      "Discounting rate:  0.8\n",
      "Decay rate:  0.01 \n",
      "\n",
      "Score over time: 0.1121\n",
      "\n",
      "[[1.11193686e-09 4.31928739e-09 3.68317764e-06 1.21978958e-09]\n",
      " [6.20996383e-11 3.54435590e-10 6.38903376e-11 4.17952057e-09]\n",
      " [5.39188343e-10 2.83724787e-04 8.99206882e-10 8.74871799e-10]\n",
      " [8.62342040e-13 8.73076680e-10 4.88486680e-14 5.09512959e-13]\n",
      " [2.97106227e-06 9.63996341e-10 2.67057056e-10 1.78026435e-10]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [2.72191984e-09 1.82393683e-04 1.30551725e-09 1.09391411e-09]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [5.87698329e-10 8.94037566e-08 1.34721467e-07 4.85730625e-06]\n",
      " [1.05884050e-06 1.67823779e-03 7.63705122e-06 8.56135672e-07]\n",
      " [3.57126218e-03 3.77257833e-07 5.05958549e-06 3.30531907e-04]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [5.33792132e-05 2.59274358e-07 3.79703642e-05 9.07215835e-07]\n",
      " [3.39952244e-04 3.81391577e-03 4.42339566e-04 4.95685659e-05]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.01\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.mean(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"******Performance of Frozen Lake agent with 'Min of action values' as the policy******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma)\n",
    "print(\"Decay rate: \", decay_rate,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 48\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 9\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 13\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 10\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 30\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 4\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 22\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 82\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 29\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 19\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 36\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 38\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 66\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 51\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 28\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 3\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 38\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 8\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 17\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 27\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Down)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 5\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "\u001b[41mH\u001b[0mFFG\n",
      "Number of steps 10\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improved performance results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "qtable = np.zeros((state_size, action_size))\n",
    "total_episodes = 40000        # Total episodes\n",
    "alpha = 0.85           # Learning rate\n",
    "max_steps = 99                # Max steps per episode\n",
    "gamma = 0.95                  # Discounting rate\n",
    "\n",
    "# Exploration parameters\n",
    "epsilon = 1.0                 # Exploration rate\n",
    "max_epsilon = 1.0             # Exploration probability at start\n",
    "min_epsilon = 0.01            # Minimum exploration probability \n",
    "decay_rate = 0.001  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****** Improved performance of Frozen Lake agent ******\n",
      "\n",
      "Total episodes:  40000\n",
      "Learning rate:  0.85\n",
      "Discounting rate:  0.95\n",
      "Decay rate:  0.001 \n",
      "\n",
      "Score over time: 0.450725\n",
      "\n",
      "[[3.84899447e-02 4.48261812e-01 3.74964246e-02 3.82613384e-02]\n",
      " [3.91016382e-04 4.56372086e-03 6.24860963e-03 5.61429834e-02]\n",
      " [2.31659308e-03 6.72197764e-03 2.86539947e-03 2.90275529e-02]\n",
      " [1.88859043e-03 3.37481095e-03 1.62519184e-03 2.41124548e-02]\n",
      " [5.49727654e-01 3.00330164e-03 3.42980271e-02 1.17360806e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [4.55684262e-04 7.01484813e-10 2.11124168e-05 1.06732982e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.19181272e-04 7.97120704e-03 7.11577540e-03 5.94555563e-01]\n",
      " [1.36466029e-06 1.09477649e-01 1.49040318e-02 1.28162595e-03]\n",
      " [8.08142483e-01 4.85343311e-05 1.99804114e-03 3.24233988e-03]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [8.16671850e-02 1.28646901e-03 8.57887104e-01 3.08036930e-03]\n",
      " [8.81893765e-02 9.78922591e-01 1.34920677e-01 2.15466792e-01]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] \n",
      "\n",
      "Epsilon at maximum steps:  0.010000000000000004\n"
     ]
    }
   ],
   "source": [
    "# List of rewards\n",
    "rewards = []\n",
    "\n",
    "# 2 For life or until learning is stopped\n",
    "for episode in range(total_episodes):\n",
    "    # Reset the environment\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    total_rewards = 0\n",
    "    \n",
    "    for step in range(max_steps):\n",
    "        # 3. Choose an action a in the current world state (s)\n",
    "        ## First we randomize a number\n",
    "        exp_exp_tradeoff = random.uniform(0, 1)\n",
    "        \n",
    "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
    "        if exp_exp_tradeoff > epsilon:\n",
    "            action = np.argmax(qtable[state,:])\n",
    "\n",
    "        # Else doing a random choice --> exploration\n",
    "        else:\n",
    "            action = env.action_space.sample()\n",
    "\n",
    "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
    "        new_state, reward, done, info = env.step(action)\n",
    "\n",
    "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
    "        # qtable[new_state,:] : all the actions we can take from new state\n",
    "        qtable[state, action] = qtable[state, action] + alpha * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
    "        \n",
    "        total_rewards += reward\n",
    "        \n",
    "        # Our new state is state\n",
    "        state = new_state\n",
    "        \n",
    "        # If done (if we're dead) : finish episode\n",
    "        if done == True: \n",
    "            break\n",
    "        \n",
    "    # Reduce epsilon (because we need less and less exploration)\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
    "    rewards.append(total_rewards)\n",
    "\n",
    "print(\"****** Improved performance of Frozen Lake agent ******\\n\")\n",
    "print(\"Total episodes: \", total_episodes )\n",
    "print(\"Learning rate: \", alpha)\n",
    "print(\"Discounting rate: \", gamma)\n",
    "print(\"Decay rate: \", decay_rate,\"\\n\")\n",
    "print (\"Score over time: \" +  str(sum(rewards)/total_episodes)+\"\\n\")\n",
    "print(qtable,\"\\n\")\n",
    "print(\"Epsilon at maximum steps: \", epsilon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************************\n",
      "EPISODE  0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 74\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 18\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "****************************************************\n",
      "EPISODE  10\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 35\n",
      "****************************************************\n",
      "EPISODE  11\n",
      "****************************************************\n",
      "EPISODE  12\n",
      "****************************************************\n",
      "EPISODE  13\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 95\n",
      "****************************************************\n",
      "EPISODE  14\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 84\n",
      "****************************************************\n",
      "EPISODE  15\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 26\n",
      "****************************************************\n",
      "EPISODE  16\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 39\n",
      "****************************************************\n",
      "EPISODE  17\n",
      "****************************************************\n",
      "EPISODE  18\n",
      "****************************************************\n",
      "EPISODE  19\n",
      "****************************************************\n",
      "EPISODE  20\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 33\n",
      "****************************************************\n",
      "EPISODE  21\n",
      "  (Up)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 15\n",
      "****************************************************\n",
      "EPISODE  22\n",
      "****************************************************\n",
      "EPISODE  23\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 34\n",
      "****************************************************\n",
      "EPISODE  24\n",
      "****************************************************\n",
      "EPISODE  25\n",
      "****************************************************\n",
      "EPISODE  26\n",
      "  (Up)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 31\n",
      "****************************************************\n",
      "EPISODE  27\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  28\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 20\n",
      "****************************************************\n",
      "EPISODE  29\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 70\n",
      "****************************************************\n",
      "EPISODE  30\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  31\n",
      "  (Up)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 6\n",
      "****************************************************\n",
      "EPISODE  32\n",
      "****************************************************\n",
      "EPISODE  33\n",
      "****************************************************\n",
      "EPISODE  34\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  35\n",
      "****************************************************\n",
      "EPISODE  36\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 11\n",
      "****************************************************\n",
      "EPISODE  37\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 38\n",
      "****************************************************\n",
      "EPISODE  38\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 64\n",
      "****************************************************\n",
      "EPISODE  39\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 35\n",
      "****************************************************\n",
      "EPISODE  40\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 53\n",
      "****************************************************\n",
      "EPISODE  41\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 12\n",
      "****************************************************\n",
      "EPISODE  42\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 24\n",
      "****************************************************\n",
      "EPISODE  43\n",
      "  (Up)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "Number of steps 7\n",
      "****************************************************\n",
      "EPISODE  44\n",
      "****************************************************\n",
      "EPISODE  45\n",
      "****************************************************\n",
      "EPISODE  46\n",
      "****************************************************\n",
      "EPISODE  47\n",
      "****************************************************\n",
      "EPISODE  48\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 14\n",
      "****************************************************\n",
      "EPISODE  49\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Number of steps 9\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "episode_steps = []\n",
    "for episode in range(50):\n",
    "    state = env.reset()\n",
    "    step = 0\n",
    "    done = False\n",
    "    print(\"****************************************************\")\n",
    "    print(\"EPISODE \", episode)\n",
    "\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Take the action (index) that have the maximum expected future reward given that state\n",
    "        action = np.argmax(qtable[state,:])\n",
    "        \n",
    "        new_state, reward, done, info = env.step(action)\n",
    "        \n",
    "        if done:\n",
    "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
    "            env.render()\n",
    "            \n",
    "            # We print the number of step it took.\n",
    "            print(\"Number of steps\", step)\n",
    "            break\n",
    "        state = new_state\n",
    "    episode_steps.append(step)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "59.78\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(episode_steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions & Answers\n",
    "1. Establish a baseline performance. How well did your RL Q-learning do on your problem?\n",
    "        Baseline performance - Score over time: 0.3412\n",
    "\n",
    "2. What are the states, the actions and the size of the Q-table?<br>\n",
    "        There are 16 states and 4 actions.\n",
    "    \n",
    "        States (4 in each stage):\n",
    "        1. SFFF\n",
    "        2. FHFH\n",
    "        3. FFFH\n",
    "        4. HFFG\n",
    "        \n",
    "        S - start state\n",
    "        F - frozen \n",
    "        H - Hole\n",
    "        G - Goal, Frisbee\n",
    "    \n",
    "       Actions:\n",
    "        Left\n",
    "        Right\n",
    "        Up\n",
    "        Down\n",
    "        \n",
    "       Size of Q-table:\n",
    "        (4,16) - 4 actions and 16 states\n",
    "        \n",
    "* What are the rewards? Why did you choose them?\n",
    "        S - start state (safe, reward = 0)\n",
    "        F - frozen (safe, reward = 0)\n",
    "        H - Hole (reward = -1, terminate the episode)\n",
    "        G - Goal (reward = 1, terminate the episode)\n",
    "* How did you choose alpha and gamma in the following equation?\n",
    "        1. Alpha and gamma are chosen based on the weightage given to prior knowledge and future rewards respectively. \n",
    "        2. For baseline performance alpha is given as 0.7 which means 70% weightage given to prior knowledge and gamma is chosen as 0.8 considering 80% weightage to the future rewards at the state.\n",
    "    \n",
    "* Try at least one additional value for alpha and gamma. How did it change the baseline performance?\n",
    "        1. When an increase of 0.05 to alpha and gamma from (0.7,0.8) to (0.75,0.85), the baseline performance has increased by around 0.04 points from 0.321 to 0.3394\n",
    "        2. When an increase of 0.05 to alpha and gamma from (0.7,0.8) to (0.8,0.90), the baseline performance has increased by around 0.09 points from 0.321 to 0.412\n",
    "\n",
    "* Try a policy other than maxQ(s', a'). How did it change the baseline performance?\n",
    "        1. For different policy which is meanQ(s',a') the performance is greatly reduced from 0.321 to 0.096.\n",
    "        2. For different policy which is minQ(s',a') the performance is greatly reduced from 0.321 to 0.1121.\n",
    "* How did you choose your decay rate and starting epsilon? Try at least one additional value for epsilon and the decay rate. How did it change the baseline performance? What is the value of epsilon when if you reach the max steps per episode?\n",
    "        1. The decay rate and starting epsilon are considered to 0.01 and 1.0 respectively. Intially the epsilon should be high to increase the possibility of exploration and later it should reduce to increase the possibility of exploitation.\n",
    "        2. For a starting epsilon of 0.9, decay rate of 0.0075 the score over time reduced from 0.321 to 0.293.\n",
    "        3. For a starting epsilon of 0.8, decay rate of 0.005 the score over time reduced from 0.321 to 0.2738.\n",
    "        4. The value of epsilon at max steps is around 0.01.\n",
    "* What is the average number of steps taken per episode?\n",
    "        1. For the baseline performance of score over time 0.321, the average steps taken are 18.86 or 19 approx.\n",
    "        2. For the improved performance of score over time 0.450725, the average steps taken are 37.44 or 37 approx.\n",
    "* Does Q-learning use value-based or policy-based iteration?\n",
    "        Value-based because the Q-table is updated each time with the future value of the action at that state. \n",
    "* What is meant by expected lifetime value in the Bellman equation?\n",
    "        Expected lifetime value in the Bellman's equation derived from the concept of converging the future reward value while calculating the Q-value function.\n",
    "        The discounting rate is introduced to add weightage to the future reward (usually between 0 and 1) and get the value using the discounting rate. The last term in the Bellman's equation refers to the expected lifetime value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "        1. Initially the baseline performance is set with certain hyperparameters and the reinforcement learning model is trained using the Q-table. The Q-table is updated using the Bellman's equation by giving weightage to the future reward and prior knowledge.\n",
    "        2. With hyperparameter tuning, the model is improved by changing number of training episodes, alpha(learning rate), gamma(discounting rate), starting epsilon and the decay rate.\n",
    "        3. It has been observed that the model performed poorly with 5000 episodes and later the model showed a bit improved when the alpha and gamma are increased. However, the model required a big number of training episodes to show significant. Hence, a set of 40000 episodes, (learning rate, discounting rate) as (0.85,0.95), starting epsilon as 1.0, decay rate as low as 0.001 resulted in an increased performance of 0.450725 which is an increase of 40% from the baseline performance.\n",
    "        4. Meanwhile, it is also concluded that using different policy functions like meanQ() and minQ() resulted in the decreased performance than the baseline.\n",
    "        3. Later, the model is tested on a 50 episode test-bed and the wins and loses are observed.\n",
    "        4. Finally, the model will result the maximum possible score over time of '0.450725' that it has gained during the Q-learning and plays the game intelligently on it's own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Author\n",
    "    Bhagyashri Rangnath Gundal (NUID: 001081806)\n",
    "    Master of Science in Information Systems\n",
    "    Northeastern University, Boston, MA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Citation\n",
    "**References:**<br>\n",
    "[1] ADL (3 September 2018), \"*An introduction to Q-Learning: reinforcement learning*\" retrieved from https://www.freecodecamp.org/news/an-introduction-to-q-learning-reinforcement-learning-14ac0b4493cc/\n",
    "\n",
    "[2] Adesh Gautam (9 July 2018),\"*Introduction to Reinforcement Learning (Coding Q-Learning) — Part 3*\" retrieved from https://medium.com/swlh/introduction-to-reinforcement-learning-coding-q-learning-part-3-9778366a41c0\n",
    "\n",
    "[3] Richard S. Sutton and Andrew G. Barto (2018), \"*Reinforcement Learning - An Introduction*\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Licensing\n",
    "Copyright 2020 Bhagyashri Rangnath Gundal\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "378px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}