EECS 395 - Course Project - Submission Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EECS 395 - Deep Reinforcement Learning - Final Project - Submission Notebook\n",
    "\n",
    "\n",
    "This notebook has been abbreviated for easier reading. \n",
    "\n",
    "Full Notebook available at https://github.com/Lukas-Justen/Deep-Reinforcement_Learning.\n",
    "\n",
    "## Notebook Structure\n",
    "\n",
    "- Each core Deep Q Learning algorithm is reviewed in depth\n",
    "- Results on both Cartpole and Lunar Landar are displayed\n",
    "- Conclusion states our major learnings from the project\n",
    "- Appendix reviews deployed architectures and shows example code to run each algorithm\n",
    "\n",
    "_by Rhett Dsouza, Keith Pallo and Lukas Justen_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "import copy\n",
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import shutil\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.models import Sequential\n",
    "from keras.models import model_from_json\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Dropout\n",
    "from keras.layers import Activation, Input\n",
    "from keras.layers.core import Dense, Lambda\n",
    "from keras.layers.merge import Add\n",
    "from keras.losses import *\n",
    "from keras.optimizers import Adam\n",
    "from keras import optimizers\n",
    "from keras import layers\n",
    "from keras import backend as K\n",
    "from keras import utils as np_utils\n",
    "\n",
    "from IPython.display import HTML\n",
    "from agent import EnvironmentAgent\n",
    "from functools import partial\n",
    "from collections import deque\n",
    "from tensorflow.python.framework import ops\n",
    "from matplotlib import gridspec\n",
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Basic Deep Q-Learning\n",
    "\n",
    "Here we present our core implementation of \"Basic\" Deep Q-Learning. As discussed in class, the goal of Deep Q Learning is to use a neural network to alleviate the main drawback of standard Q Learning, which is representation of a non-trivial Q matrix. In other words, instead of explicitly storing the reward for every state and action pair, one instead uses a non-linear function approximator ( Neural Network ) to accomplish the underlying goal of directing the agent optimally given it's current state.\n",
    "\n",
    "In order to train our Neural Network, we let our agent take actions repeatedly until an end state, or the max number of steps per episode is reached. This loop is called an episode. Each action in the episode is either taken randomly or optimally from the current network, where the probability of each is determined by a set parameter, epsilon. Over time, epison is decayed so that actions taken from the network are more likely - i.e. we begin to trust the network over time. All of the state, action, and rewards pairs for a given episode are stored in a memory which is implemented as a queue ( previous info from old episodes will eventually exit this queue). After each episode, we update our neural network via training by randomly sampling from the memory, which at any point contains the values for several episodes. This random sampling assists with overfitting to a single episode.\n",
    "\n",
    "After completing this process several times, our Neural Network essentially performs the purpose of the standard Q matrix - given an observation state, it will direct the agent optimally.\n",
    "\n",
    "The core functions to accomplish this are: initialization, build_model, take_action, take_step, remember, and train_model. \n",
    "\n",
    "<br/>\n",
    "\n",
    "#### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DeepQLearning:\n",
    "  \n",
    "    def __init__(self, action_size, observation_size, environment):\n",
    "        '''\n",
    "        Initialize all the parameters that are required for basic deep Q-Learning.\n",
    "\n",
    "        :param action_size: The number of different actions the agent can take.\n",
    "        :param observatoion_size: The number of input observations the agent gets.\n",
    "        :param environment: The environment, in which the agent is living in.\n",
    "        '''\n",
    "        self.epsilon = 1.0\n",
    "        self.epsilon_min = 0.05\n",
    "        self.epsilon_decay = 0.9997\n",
    "        self.gamma = 1.0\n",
    "        self.learning_rate = 0.001\n",
    "        self.batch_size = 512\n",
    "        self.action_size = action_size\n",
    "        self.observation_size = observation_size\n",
    "        self.target_model = self.build_model()\n",
    "        self.memory = deque(maxlen=2000)\n",
    "        self.environment = environment\n",
    "    \n",
    "    def take_action(self, observation, train):\n",
    "        '''\n",
    "        Takes one action based on the given observation and the environment.\n",
    "\n",
    "        :param observation: The current observation which the agent should act on.\n",
    "        '''\n",
    "        if train and np.random.rand(1) < self.epsilon:\n",
    "            return self.environment.action_space.sample()\n",
    "        return np.argmax(self.target_model.predict(observation)[0])\n",
    "  \n",
    "    def remember(self, observation, action, reward, next_observation, done):\n",
    "        '''\n",
    "        Takes all the parameters of the environment and stores them in a memory.\n",
    "\n",
    "        :param observation: The current observation.\n",
    "        :param action: The action the agent took based on the observation.\n",
    "        :param reward: The reward the agent got for his action.\n",
    "        :param next_observation: The resulting observation based on the last action.\n",
    "        :param done: Is the agent done with the simulation.\n",
    "\n",
    "        :return: The new next observation\n",
    "        '''\n",
    "        self.memory.append((observation, action, reward, next_observation, done))\n",
    "        return next_observation\n",
    "    \n",
    "    def train_model(self, episode, done):\n",
    "        '''\n",
    "        Trains all the models that are required for this Q-Learning implementation.\n",
    "\n",
    "        :param episode: Takes in the current episode of training.\n",
    "        '''\n",
    "        if len(self.memory) >= self.batch_size:\n",
    "            minibatch = random.sample(self.memory, self.batch_size)\n",
    "            minibatch = np.array(minibatch)\n",
    "            observations = np.vstack(minibatch[:, 0])\n",
    "            next_observations = np.vstack(minibatch[:,3])\n",
    "            target = np.copy(minibatch[:,2])\n",
    "            done_states = np.where(minibatch[:, 4] == False)\n",
    "            if len(done_states[0]) > 0:\n",
    "                predictions = self.target_model.predict(next_observations)\n",
    "                predictions = np.amax(predictions, axis=1)\n",
    "                target[done_states] += np.multiply(self.gamma, predictions[done_states])\n",
    "            actions = np.array(minibatch[:, 1], dtype=int)\n",
    "            target_f = self.target_model.predict(observations)\n",
    "            target_f[range(self.batch_size), actions] = target\n",
    "            self.target_model.fit(observations, target_f, epochs=1, verbose=0)\n",
    "\n",
    "        if done and self.epsilon > self.epsilon_min:\n",
    "            self.epsilon = self.epsilon * self.epsilon_decay\n",
    "           \n",
    "    def build_model(self):\n",
    "        '''\n",
    "        Builds the initial architecture for a basi deep Q-Learning algorithm\n",
    "\n",
    "        :return: A new keras model for the current environment.\n",
    "        '''\n",
    "        model = Sequential()\n",
    "        model.add(Dense(24, input_dim=self.observation_size, activation='relu'))\n",
    "        model.add(Dense(24, activation='relu'))\n",
    "        model.add(Dense(self.action_size, activation='linear'))\n",
    "        model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))\n",
    "        return model  \n",
    "  \n",
    "    def reset(self):\n",
    "        '''\n",
    "        Resets the environment to a new observation and returns the observation.\n",
    "\n",
    "        :return: An initial observation state as well as other reset values.\n",
    "        '''\n",
    "        observation = self.environment.reset()\n",
    "        observation = np.reshape(observation, (1, self.observation_size))\n",
    "        return observation, False, 0\n",
    "  \n",
    "    def take_step(self, action):\n",
    "        '''\n",
    "        Takes step based on the specified action. This will simulate a new step. \n",
    "\n",
    "        :return: The results of the step the agent took\n",
    "        '''\n",
    "        next_observation, reward, done, info = self.environment.step(action)\n",
    "        next_observation = np.reshape(next_observation, (1,self.observation_size))\n",
    "        return next_observation, reward, done, info\n",
    "  \n",
    "    def save_models(self, prefix, folder_id):\n",
    "        '''\n",
    "        This function saves the model on disk and uploads all files to Google drive.\n",
    "\n",
    "        :param prefix: A prefix for the files to be stored.\n",
    "        '''\n",
    "        self.target_model.save_weights(prefix + '_weights.h5')\n",
    "    \n",
    "    def load_weights(self, weights):\n",
    "        '''\n",
    "        Loads the weights for the target network based on the given input weights.\n",
    "        '''\n",
    "        self.target_model.load_weights(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Double Deep Q-Learning\n",
    "\n",
    "Compared to the basic deep Q-Learning implementation, the only modification needed to implement Double Deep Q-Learning is to change the way how the models is trained. A big drawback of basic deep Q-Learning is that the algorithm overestimates action values under certain conditions. To mitigate that problem, one can use two networks with the exact same network architecture (proposed here: https://arxiv.org/abs/1509.06461). However, instead of updating the target network after each episode we train the second network which is called q network for several episodes. After a fixed number of episodes the target network will then be updated by copying the parameters of the q network.\n",
    "\n",
    "One of our major findings for this implementation was that this architecture leads to more stable results. Moreover, if the algorithm stagnates at a certain reward level this implementation will likely stay at that level as opposed to dropping off. Nonetheless, this implementation also introduces the parameter for updating the target network after n episodes. This new hyperparameter increases the complexity compared to our basic q-learning implementation. Thus, for simple problems like the cart-pole environment we would recommend to use a simple q learner because tuning all the parameters can take quite a lot of time.\n",
    "\n",
    "#### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DoubleDeepQLearning(DeepQLearning):\n",
    "  \n",
    "    def __init__(self, action_size, observation_size, environment):\n",
    "        '''\n",
    "        Initialize all the parameters that are required for double deep Q-Learning.\n",
    "        Double deep q learning needs a second network and the number of epsidoes\n",
    "        after the algoruthm should update the target network based on the q model.\n",
    "\n",
    "        :param action_size: The number of different actions the agent can take.\n",
    "        :param observatoion_size: The number of input observations the agent gets.\n",
    "        :param environment: The environment, in which the agent is living in.\n",
    "        '''\n",
    "        DeepQLearning.__init__(self, action_size, observation_size, environment)\n",
    "        self.q_model = self.build_model()\n",
    "        self.update_target_every_episodes = 2\n",
    "  \n",
    "    def train_model(self, episode, done):\n",
    "        '''\n",
    "        Trains all the models that are required for this Q-Learning implementation.\n",
    "        Compared to the basic implementation we now have two networks that interact\n",
    "        with each other. The target network is only updated after n episdoes.\n",
    "        '''\n",
    "        if len(self.memory) >= self.batch_size:\n",
    "            minibatch = random.sample(self.memory, self.batch_size)\n",
    "            minibatch = np.array(minibatch)\n",
    "            observations = np.vstack(minibatch[:, 0])\n",
    "            next_observations = np.vstack(minibatch[:,3])\n",
    "            target = np.copy(minibatch[:,2])\n",
    "            done_states = np.where(minibatch[:, 4] == False)\n",
    "            if len(done_states[0]) > 0:\n",
    "                q_values = self.q_model.predict(next_observations)\n",
    "                best_actions = np.argmax(q_values, axis=1)\n",
    "                q_targets = self.target_model.predict(next_observations)\n",
    "                target[done_states] += np.multiply(self.gamma, q_targets[done_states, best_actions[done_states]][0])\n",
    "            actions = np.array(minibatch[:, 1], dtype=int)\n",
    "            target_f = self.target_model.predict(observations)\n",
    "            target_f[range(self.batch_size), actions] = target\n",
    "            self.q_model.fit(observations, target_f, epochs=1, verbose=0)\n",
    "\n",
    "        if done and self.epsilon > self.epsilon_min:\n",
    "            self.epsilon = self.epsilon * self.epsilon_decay\n",
    "\n",
    "        if done and (episode + 1) % self.update_target_every_episodes == 0:\n",
    "            self.target_model.set_weights(self.q_model.get_weights())\n",
    "\n",
    "    def load_weights(self, weights):\n",
    "        '''\n",
    "        Loads the weights for the target/q network based on the given input weights.\n",
    "        '''\n",
    "        self.target_model.load_weights(weights)\n",
    "        self.q_model.load_weights(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Dueling Deep Q-Learning\n",
    "\n",
    "The next step after building a double deep Q-Learning algorithm was extending the architecture to a more recent optimization, which is a dueling network. Our implementation first inherits all the functions of the double deep q learner.\n",
    "\n",
    "The only thing that we need to change is the “build_model” function. In Dueling Deep Q-learning (as shown in the image below) two streams or branches perform different purposes - one is called the advantage branch and the other is called the Value branch. Each branch takes the observations of the environment as an input. The advantage branch then calculates the q values for all the actions while the value branch tries to model the reward landscape. The value branch by contract models the state value, V. By separating these two calculations each network can fit the desired landscape much better.\n",
    "\n",
    "Since this algorithm inherits from our double deep Q-Learning implementation we also have two copies of the network.\n",
    "\n",
    "#### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DuelingDeepQLearning(DoubleDeepQLearning):\n",
    "  \n",
    "    def __init__(self, action_size, observation_size, environment):\n",
    "        '''\n",
    "        Initialize all the parameters that are required for dueling deep Q-Learning.\n",
    "\n",
    "        :param action_size: The number of different actions the agent can take.\n",
    "        :param observatoion_size: The number of input observations the agent gets.\n",
    "        :param environment: The environment, in which the agent is living in.\n",
    "        '''\n",
    "        DoubleDeepQLearning.__init__(self,action_size,observation_size, environment)\n",
    "  \n",
    "    def build_model(self):\n",
    "        '''\n",
    "        Builds the initial architecture for the dueling deep q learning algorithm. \n",
    "        The actual training does not change but only the way how we compose the \n",
    "        network. Changing this function is the only change compared to double Q-L.\n",
    "\n",
    "        :return: A new keras model for the current environment.\n",
    "        '''\n",
    "        inputs = Input(shape=(self.observation_size,))\n",
    "\n",
    "        advt = Dense(64, activation='relu')(inputs)\n",
    "        advt = Dense(64, activation='relu')(advt)\n",
    "        advt = Dense(64, activation='relu')(advt)\n",
    "        advt = Dense(self.action_size)(advt)\n",
    "\n",
    "        value = Dense(16, activation='relu')(inputs)\n",
    "        value = Dense(16, activation='relu')(value)\n",
    "        value = Dense(1)(value)\n",
    "\n",
    "        advt = Lambda(lambda advt: advt - tf.reduce_mean(advt, axis=1, keep_dims=True))(advt)\n",
    "        value = Lambda(lambda value: tf.tile(value, [1, self.action_size]))(value)\n",
    "\n",
    "        final = Add()([value, advt])\n",
    "        model = Model(inputs = inputs, outputs = final)\n",
    "\n",
    "        model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))\n",
    "\n",
    "        return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 4. Policy Gradient Learning\n",
    "\n",
    "Policy Gradient applies a different mindset when deciding what actions to take. Rather than learning (regressing) the optimal Q-values associated with a next best action (like in the other DQN algorithms), Policy Gradient attempts to directly generate the next action as a classifiction task following the Markov property, using a softmax operation in the last layer of the network. For this case, to encorporate the reward structure, each step of the optimizer is weighted by the cumulative reward of the future actions in the episode. \n",
    "\n",
    "This was much harder to implement with the Keras API, as it involved changing the fundamental loss function supplied to the model compilation of Keras. Instead, we opted to use a Tensorflow implementation, that very easily configures the loss function based on how you wish to supply the weighting.\n",
    "\n",
    "Judging from the plots, it appears that Policy Gradient in this context is very erratic in its training. It is able to achieve extremely high average rewards compared to the other Deep Q-learning algorithms, but has no stability to its training curve. This results in a lower reliability on the reward capability of Policy Gradient. Random dropping of the average reward result value, even after the network appears to be performing optimally, requires the weights to be cherry-picked (we opted not to do so, hence the videos represent sub-optimal models) hinting at the idea that the network may not be easily generalisable to newer situations of similar environments. \n",
    "\n",
    "\n",
    "#### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PolicyGradient:\n",
    "  \n",
    "    def __init__(self, n_y, n_x, environment, learning_rate=0.01, reward_decay=0.95, load_path=None, save_path=None): \n",
    "        '''\n",
    "        Initialize all the parameters that are required for policy gradient learning.\n",
    "\n",
    "        :param n_y: The number of different actions the agent can take.\n",
    "        :param n_x: The number of input observations the agent gets.\n",
    "        :param environment: The environment, in which the agent is living in.\n",
    "        '''\n",
    "        self.n_x = n_x\n",
    "        self.n_y = n_y\n",
    "        self.environment = environment\n",
    "        self.lr = learning_rate\n",
    "        self.gamma = reward_decay\n",
    "\n",
    "        self.save_path = None\n",
    "        if save_path is not None:\n",
    "            self.save_path = save_path\n",
    "\n",
    "        self.episode_observations, self.episode_actions, self.episode_rewards = [], [], []\n",
    "        self.build_model()\n",
    "        self.cost_history = []\n",
    "        self.sess = tf.Session()\n",
    "        tf.summary.FileWriter(\"logs/\", self.sess.graph)\n",
    "        self.sess.run(tf.global_variables_initializer())\n",
    "        self.saver = tf.train.Saver()\n",
    "\n",
    "        if load_path is not None:\n",
    "            self.load_path = load_path\n",
    "            self.saver.restore(self.sess, self.load_path)\n",
    "    \n",
    "    def reset(self):\n",
    "        '''\n",
    "        Resets the environment to a new observation and returns the observation.\n",
    "\n",
    "        :return: An initial observation state as well as other reset values.\n",
    "        '''\n",
    "        observation = self.environment.reset()\n",
    "        observation = np.reshape(observation, (1, self.n_x))\n",
    "        return observation, False, 0\n",
    "  \n",
    "    def remember(self, s, a, r, ns, done):\n",
    "        \"\"\"\n",
    "        Store play memory for training.\n",
    "        \n",
    "        :param s: observation\n",
    "        :param a: action taken\n",
    "        :param r: reward after action\n",
    "        \"\"\"\n",
    "        self.episode_observations.append(s[0])\n",
    "        self.episode_rewards.append(r)\n",
    "\n",
    "        action = np.zeros(self.n_y)\n",
    "        action[a] = 1\n",
    "        self.episode_actions.append(action)\n",
    "        return ns\n",
    "\n",
    "    def take_action(self, observation, train):\n",
    "        \"\"\"\n",
    "        Choose action based on observation.\n",
    "        \n",
    "        :param observation: array of state, has shape (num_features)\n",
    "            :return: index of action we want to choose\n",
    "        \"\"\"\n",
    "\n",
    "        observation = np.asarray(np.asarray(observation).T)\n",
    "        # Run forward propagation to get softmax probabilities\n",
    "        prob_weights = self.sess.run(self.outputs_softmax, feed_dict = {self.X: observation})\n",
    "\n",
    "        # Select action using a biased sample this will return the index of the action we've sampled\n",
    "        action = np.random.choice(range(len(prob_weights.ravel())), p=prob_weights.ravel())\n",
    "        return action\n",
    "           \n",
    "    def take_step(self, action):\n",
    "        '''\n",
    "        Takes a step based on the specified action. In fact, this will simulate a new step. \n",
    "\n",
    "        :return: The results of the step the agent took\n",
    "        '''\n",
    "        next_observation, reward, done, info = self.environment.step(action)\n",
    "        next_observation = np.reshape(next_observation, (1,self.n_x))\n",
    "        return next_observation, reward, done, info\n",
    "       \n",
    "    def train_model(self,e, done):\n",
    "        # Discount and normalize episode reward\n",
    "        discounted_episode_rewards_norm = self.discount_and_norm_rewards()\n",
    "\n",
    "        # Train on episode\n",
    "        self.sess.run(self.train_op, feed_dict={\n",
    "             self.X: np.vstack(self.episode_observations).T,\n",
    "             self.Y: np.vstack(np.array(self.episode_actions)).T,\n",
    "             self.discounted_episode_rewards_norm: discounted_episode_rewards_norm,\n",
    "        })\n",
    "\n",
    "        # Reset the episode data\n",
    "        self.episode_observations, self.episode_actions, self.episode_rewards  = [], [], []\n",
    "\n",
    "        return discounted_episode_rewards_norm\n",
    "      \n",
    "    def discount_and_norm_rewards(self):\n",
    "        discounted_episode_rewards = np.zeros_like(self.episode_rewards)\n",
    "        cumulative = 0\n",
    "        for t in reversed(range(len(self.episode_rewards))):\n",
    "            cumulative = cumulative * self.gamma + self.episode_rewards[t]\n",
    "            discounted_episode_rewards[t] = cumulative\n",
    "\n",
    "        discounted_episode_rewards -= np.mean(discounted_episode_rewards)\n",
    "        discounted_episode_rewards /= np.std(discounted_episode_rewards)\n",
    "        return discounted_episode_rewards\n",
    "\n",
    "    def build_model(self):\n",
    "        # Create placeholders\n",
    "        tf.reset_default_graph()\n",
    "        with tf.name_scope('inputs'):\n",
    "            self.X = tf.placeholder(tf.float32, shape=(self.n_x, None), name=\"X\")\n",
    "            self.Y = tf.placeholder(tf.float32, shape=(self.n_y, None), name=\"Y\")\n",
    "            self.discounted_episode_rewards_norm = tf.placeholder(tf.float32, [None, ], name=\"actions_value\")\n",
    "\n",
    "        # Initialize parameters\n",
    "        units_layer_1 = 24\n",
    "        units_layer_2 = 24\n",
    "        units_output_layer = self.n_y\n",
    "        with tf.name_scope('parameters'):\n",
    "            W1 = tf.get_variable(\"W1\", [units_layer_1, self.n_x], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "            b1 = tf.get_variable(\"b1\", [units_layer_1, 1], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "            W2 = tf.get_variable(\"W2\", [units_layer_2, units_layer_1], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "            b2 = tf.get_variable(\"b2\", [units_layer_2, 1], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "            W3 = tf.get_variable(\"W3\", [self.n_y, units_layer_2], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "            b3 = tf.get_variable(\"b3\", [self.n_y, 1], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n",
    "\n",
    "        # Forward prop\n",
    "        with tf.name_scope('layer_1'):\n",
    "            Z1 = tf.add(tf.matmul(W1,self.X), b1)\n",
    "            A1 = tf.nn.relu(Z1)\n",
    "        with tf.name_scope('layer_2'):\n",
    "            Z2 = tf.add(tf.matmul(W2, A1), b2)\n",
    "            A2 = tf.nn.relu(Z2)\n",
    "        with tf.name_scope('layer_3'):\n",
    "            Z3 = tf.add(tf.matmul(W3, A2), b3)\n",
    "            A3 = tf.nn.softmax(Z3)\n",
    "\n",
    "        # Softmax outputs, we need to transpose as tensorflow nn functions expects them in this shape\n",
    "        logits = tf.transpose(Z3)\n",
    "        labels = tf.transpose(self.Y)\n",
    "        self.outputs_softmax = tf.nn.softmax(logits, name='A3')\n",
    "\n",
    "        with tf.name_scope('loss'):\n",
    "            neg_log_prob = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels)\n",
    "            loss = tf.reduce_mean(neg_log_prob * self.discounted_episode_rewards_norm)  # reward guided loss\n",
    "\n",
    "        with tf.name_scope('train'):\n",
    "            self.train_op = tf.train.AdamOptimizer(self.lr).minimize(loss)\n",
    "    \n",
    "    def close_policy(self):\n",
    "        self.sess.close()\n",
    "      \n",
    "    def save_models(self, prefix, folder_id):\n",
    "        self.save_path = prefix + '/model_weights'\n",
    "        save_path = self.saver.save(self.sess, self.save_path)\n",
    "        print(\"Model saved in file: %s\" % save_path)\n",
    "        shutil.make_archive(prefix, 'zip', prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cart Pole Results\n",
    "\n",
    "### Y Axis - Average reward of last 100 episodes. \n",
    "![alt text](./cart_pole/results/cart-pole_policy.png \"Title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lunar Lander Results\n",
    "\n",
    "### Y Axis - Average reward of last 100 episodes. \n",
    "\n",
    "\n",
    "![alt text](./lunar_lander/results.png \"Title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Conclusion\n",
    "- It is difficult to tell which version of Deep Q Learning will work best on a given problem\n",
    "\n",
    "- Learned policies can change drastically over time, and behavior can deviate from settings that were previously superior in performance\n",
    "\n",
    "- There are a significant number of tuning parameters which are highly sensitive and there is often less intuition than standard deep learning parameter tuning\n",
    "\n",
    "- State of the art Reinforcement algorithms present an excellent way to solve extremely challenging problems, but it is critical that they are one of the last solutions considered at it is a daunting task to implement them correctly. The algorithms shown here are simply a tool, where the real work is often in creating the environment and more importantly the reward structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (A) Appendix\n",
    "## Network Architectures\n",
    "\n",
    "\n",
    "Cart Pole:\n",
    "- Basic DQN:       4 x 64 x 64 x 2\n",
    "- Double DQN:      4 x 24 x 24 x 2\n",
    "- Dueling DQN:     4 x 24 x 24 x 2 + 4 x 24 x 24 x 2\n",
    "- Policy Gradient: 4 x 24 x 24 x 2\n",
    "\n",
    "Lunar Lander:\n",
    "\n",
    "- Basic DQN:   8 x 24 x 24 x 4\n",
    "- Double DQN:  8 x 512 x 512 x 512 x 4\n",
    "- Dueling DQN: 8 x 64 x 64 x 64 x 4 + 8 x 16 x 16 x 4\n",
    "- Policy Gradient: 4 x 24 x 24 x 2\n",
    "\n",
    "## Simulating Results\n",
    "\n",
    "__Note:__ You might need to change the network architecture in the implementation above. As of now only the Policy Gradient, Dueling, and basic implementation will work.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EnvironmentAgent allows code to run. \n",
    "\n",
    "Provided for reference. In github repo the code is included in the agent.py file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnvironmentAgent:\n",
    "  \n",
    "    def __init__(self, environment, qlearner, verbose = True, weights=None):\n",
    "        '''\n",
    "        Initialize the Framework for the Training Environment. We can set training\n",
    "        relevant parameters right here and take the environment/agent as parameters.\n",
    "        '''\n",
    "        self.environment = environment\n",
    "        self.episodes = 10000\n",
    "        self.steps = 5000\n",
    "        self.average_steps = 0\n",
    "        self.average_reward = 0\n",
    "        self.logging_steps = 100\n",
    "        self.qlearner = qlearner\n",
    "        self.history_episodes = []\n",
    "        self.history_steps = []\n",
    "        self.history_rewards = []\n",
    "        self.verbose = verbose\n",
    "        if weights:\n",
    "            self.qlearner.load_weights(weights)\n",
    "\n",
    "    \n",
    "    def train_agent(self, train=True):\n",
    "        '''\n",
    "        Sets up the training loop for multiple epsidoes & steps in each episode. \n",
    "        Here we never need to change anything to test our different agents.\n",
    "        '''\n",
    "        for e in range(self.episodes):\n",
    "            observation, done, t = self.qlearner.reset()\n",
    "\n",
    "            while t < self.steps and not done:\n",
    "                action = self.qlearner.take_action(observation, train)\n",
    "                if not train:\n",
    "                    self.environment.render()\n",
    "                next_observation, reward, done, info = self.qlearner.take_step(action)\n",
    "                self.average_reward += reward\n",
    "                observation = self.qlearner.remember(observation, action, reward, next_observation, done)\n",
    "                t += 1\n",
    "            self.record_training(t, e)\n",
    "            if train:\n",
    "                self.qlearner.train_model(e, done)\n",
    "        \n",
    "        \n",
    "    def record_training(self, steps, episode):\n",
    "        '''\n",
    "        Prints training relevant information on the command line & stores the values\n",
    "        in lists so that we can save them on disk to create visualizations.\n",
    "        '''\n",
    "        self.average_steps += steps\n",
    "        if (episode + 1) % self.logging_steps == 0:\n",
    "            a_steps = self.average_steps/self.logging_steps\n",
    "            a_reward = self.average_reward/self.logging_steps\n",
    "            if self.verbose:\n",
    "                print(episode + 1, self.episodes, a_steps, a_reward)\n",
    "            self.history_episodes.append(episode + 1)\n",
    "            self.history_steps.append(a_steps)\n",
    "            self.history_rewards.append(a_reward)\n",
    "            self.average_steps = 0\n",
    "            self.average_reward = 0      \n",
    "    \n",
    "    \n",
    "    def close_environment(self):\n",
    "        '''\n",
    "        Closes the environment that we are using for training.\n",
    "        '''\n",
    "        if type(self.qlearner).__name__ == 'PolicyGradient':\n",
    "            self.qlearner.close_policy()\n",
    "        self.environment.close()\n",
    "    \n",
    "    \n",
    "    def save_models(self, prefix, folder_id):\n",
    "        '''\n",
    "        Saves given qlearner model on disk by appending given prefix to file name.\n",
    "        '''\n",
    "        self.qlearner.save_models(prefix, folder_id)\n",
    "    \n",
    "    \n",
    "    def get_training_history(self):\n",
    "        '''\n",
    "        Returns the average steps and average reward per episode step.\n",
    "        '''\n",
    "        history_df = pd.DataFrame()\n",
    "        history_df[\"Episodes\"] = self.history_episodes\n",
    "        history_df[\"Steps\"] = self.history_steps\n",
    "        history_df[\"Rewards\"] = self.history_rewards\n",
    "        return history_df\n",
    "  \n",
    "  \n",
    "    def save_history(self, prefix, folder_id):\n",
    "        '''\n",
    "        Loads the history of the steps and rewards to the Google drive folder.\n",
    "        :param prefix: The prefix for the file that will be saved on disk.\n",
    "        '''\n",
    "        history_df = self.get_training_history()\n",
    "        history_df.to_csv(prefix + \"_history.csv\", index = False, header = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code below deploys LunarLandar, but can be easily modified for any environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment = gym.make(\"LunarLander-v2\")\n",
    "basicDeepQLearner = DeepQLearning(4, 8, environment)\n",
    "agent = EnvironmentAgent(environment, basicDeepQLearner, weights='./lunar_lander/basic_deep_q_learner_weights.h5')\n",
    "agent.train_agent(train=False)\n",
    "agent.close_environment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment = gym.make(\"LunarLander-v2\")\n",
    "doubleDeepQLearner = DoubleDeepQLearning(4, 8, environment)\n",
    "agent = EnvironmentAgent(environment, doubleDeepQLearner, weights='./lunar_lander/double_deep_q_learner_weights.h5')\n",
    "agent.train_agent(train=False)\n",
    "agent.close_environment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment = gym.make(\"LunarLander-v2\")\n",
    "duelingDeepQLearner = DuelingDeepQLearning(4, 8, environment)\n",
    "agent = EnvironmentAgent(environment, duelingDeepQLearner, weights='./lunar_lander/dueling_deep_q_learner_weights.h5')\n",
    "agent.train_agent(train=False)\n",
    "agent.close_environment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment = gym.make(\"LunarLander-v2\")\n",
    "policyGradientLearner = PolicyGradient(4, 8, environment, load_path='./lunar_lander/policy_gradient_weights/model_weights')\n",
    "agent = EnvironmentAgent(environment, policyGradientLearner)\n",
    "agent.train_agent(train=False)\n",
    "agent.close_environment()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}