DL4CV-Final-Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accelerated Reinforcement Learning via Imitation Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reinforcement learning (RL), sparse rewards\n",
    "are a natural way to specify the task to be learned. However,\n",
    "most RL algorithms struggle to learn in this setting since the\n",
    "learning signal is mostly zeros."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"plots/old_plots/rl_only_reward.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the example, an RL agent struggles to learn in 200,000 timesteps in a sparse-reward setting, receiving negative average rewards throughout the process. In environments where positive signals are extremely sparse, such as the Game of Go or Starcraft, where the only positive signal is winning the game, it would take forever for the agent to randomly \"stumble upon\" an policy that consistently guides the agent to receive the positive reward. Most of the time, as the agent is exploring, it receives no informative feedback on its actions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to devise a novel approach to enable efficient RL in the sparse reward setting. \n",
    "\n",
    "The method we are proposing is: RL + IL\n",
    "\n",
    "We recognize that RL and IL are already well-studied, our novelty is to combine them for more efficient learning. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"plots/Accelerated_RL_via_IL.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pitfall! ATARI 2600\n",
    "\n",
    "Reason that we choose it:\n",
    "- human can provide guidance\n",
    "- sparse reward setting\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning from demonstration (Imitation Learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn a network that maps state (inputs) to actions (outputs)\n",
    "\n",
    "State: 84 (height) * 84 (width) * 3 (RGB) image pixels\n",
    "Action: 8-dimensional boolean vector, where each element of the vector corresponds to a button on the ATARI console \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reinforcement Learning in Sparse-Reward Setting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## My contributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "openai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}