Jupyter_Notebook_AWS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS+Jupyter, Screening and Aliasing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook will serve as a tutorial to set up Jupyter notebooks on AWS.\n",
    "\n",
    "I will assume that you have already set up and AWS account and are aware of SSH keys etc.\n",
    "\n",
    "I also assume the following **configuration** of your AWS instance:\n",
    "\n",
    "   Type         |  Protocol    | Port Range    | Source\n",
    "  ------------- | ------------- -------------  | -------------\n",
    "  SSH           | TCP          | 22            | Anywhere 0.0.0.0/0\n",
    "  HTTPS         | TCP          | 443           | Anywhere 0.0.0.0/0\n",
    "  Custom TCP Rule | TCP        | 8888          | Anywhere 0.0.0.0/0\n",
    "\n",
    "This tutorial will **NOT** install jupyter notebooks via downloading **anaconda source-code.**\n",
    "\n",
    "Since we are **NOT** installing the anaconda distribution, you should install all the packaged you require via `pip`/`pip3`. If you do this ahead of time, your jupyter notebook will be ready to run all of your favourite packages at the end of this tutorial.\n",
    "\n",
    "As such, complete the following **first**:\n",
    "\n",
    "* `pip3` installed on your ubuntu machine\n",
    "    * upgrade to the latest version of pip: `pip3 install --upgrade pip`\n",
    "\n",
    "\n",
    "* install jupyter : `pip3 install jupyter`\n",
    "\n",
    "### LET'S BEGIN!\n",
    "\n",
    "##### Note: `$:` represents the terminal prompt\n",
    "\n",
    "#### 1. Create a password for your Jupyter Notebook - remember this, you will need it later!\n",
    "\n",
    "In your terminal, do the following:\n",
    "\n",
    "> `$: ipython`\n",
    "\n",
    "> `$: from IPython.lib import passwd`\n",
    "\n",
    "> `$: passwd()`\n",
    "\n",
    "At this point, you will enter a password of your choice. Twice. Be sure to remember this password.\n",
    "\n",
    "After entering your password twice, the interpreter will output a **_hashed_** version of your entered password. It will start with *`'sha1:`*\n",
    "\n",
    "**COPY THIS HASHED PASSWORD AND PASTE IN SOMEWHERE FOR EASY ACCESS. YOU WILL NEED THIS LATER.**\n",
    "\n",
    "> `$: exit`\n",
    "\n",
    "#### 2. Create a Jupyter Config File\n",
    "\n",
    "> `$: jupyter notebook --generate-config`\n",
    "\n",
    "The above command will generate the default jupyter notebook configuration file. Don't worry, this wont change anything as all the configurations in this file have been hashed out!\n",
    "\n",
    "We will now be adding some of our own configurations to this file to make our notebook integrate with AWS.\n",
    "\n",
    "But before that, we have to add some security certificates!\n",
    "\n",
    "#### 3. Security certificates!\n",
    "\n",
    "In your *ROOT* directory i.e. `~$` type the following:\n",
    "\n",
    "> `$: mkdir certificates`\n",
    "\n",
    "> `$: cd certificates`\n",
    "\n",
    "> `$: sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout notebook_certificate.pem -out notebook_certificate.pem`\n",
    "\n",
    "At this point, you will see a series of questions appear on your terminal. These questions are required for creating a _unique_ certificate for you (i.e. the user). Just answer the questions as stated on the screen:\n",
    "\n",
    "e.g.: Provide the initials (2) of your country of residence [AU]: US <-- if you lived in the US.\n",
    "\n",
    "**Note: It does not matter if you make a mistake in answering any of the questions! We jsut need to answer these questions to generate a security certificate**\n",
    "\n",
    "After you have answered the questions, you should be back in your root directory!\n",
    "\n",
    "At this point, type `pwd`. Copy the output - this is the file location of your certificate. We will need this path later.\n",
    "\n",
    "The text you copy should look something like this: \n",
    "\n",
    "> `/home/igabr/certificates/notebook_certificate.pem`\n",
    "\n",
    "#### 3. Configure Jupyter\n",
    "\n",
    "Now type the following into terminal:\n",
    "\n",
    "> `$: cd ~/.jupyter/`\n",
    "\n",
    "> `$: nano jupyter_notebook_config.py`\n",
    "\n",
    "You should now see the jupyter notebook configuration file opened in the nano text editor. There is **LOTS** of text that is hashed out in this file. No need to be intimidated! We are just going to add a few lines of **UNHASHED** text to this file.\n",
    "\n",
    "**Enter the following in jupyter_notebook_config.py:**\n",
    "\n",
    "\n",
    ">`c = get_config()`\n",
    "\n",
    ">`c.IPKernelApp.pylab = 'inline'  # This enables %matplotlib inline by default`\n",
    "\n",
    "> `c.NotebookApp.certfile = u'PATH TO CERTIFICATE HERE'`\n",
    "\n",
    "> `c.NotebookApp.ip = '*'`\n",
    "\n",
    "> `c.NotebookApp.open_browser = False`\n",
    "\n",
    "> `c.NotebookApp.password = u'HASHED PASSWORD HERE'`  \n",
    "\n",
    "> `c.NotebookApp.port = 8888 #JUST LIKE WE SET IN AWS`\n",
    "\n",
    "After you have typed in the above lines, we need to save and exit the nano editor. Do the following:\n",
    "\n",
    "`control + o`\n",
    "\n",
    "`press enter`\n",
    "\n",
    "`control + x`\n",
    "\n",
    "#### CONGRATS - LETS LAUNCH JUPYTER IN AWS!\n",
    "\n",
    "When you set up your EC 2 instance on AWS, you should have a url that resembles something like this:\n",
    "\n",
    "> `ec2-34-226-197-203.compute-1.amazonaws.com`\n",
    "\n",
    "Copy your distinct url and add `:8888/` to the end and `https://` to the beginning.\n",
    "\n",
    "You should have something like this:\n",
    "\n",
    "> `https://ec2-34-226-197-203.compute-1.amazonaws.com:8888/`\n",
    "\n",
    "Type the above into your browser of choice on your local machine!\n",
    "\n",
    "You should then be directed to a page that gives you are **warning about the safety** of the page you are visiting.\n",
    "\n",
    "**Ignore this warning and proceed to the webpage**\n",
    "\n",
    "You will now see a jupyter webpage with a password box.\n",
    "\n",
    "**Enter the NON-HASHED version of the password you created earlier on!** You will only have to do this once per session!\n",
    "\n",
    "After you enter the password, you should now see the jupyter dashboard. Awesome!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aliasing in AWS\n",
    "\n",
    "For those user's who wish to have a more familiar command line experience in AWS, it is possible to transfer over local terminal aliases.\n",
    "\n",
    "AWS does **NOT** have a `.bash_profile` file. You can verify this by typing `$: ls -a`.\n",
    "\n",
    "However, AWS does have a file called `.profile` which is the same as `.bash_profile`. In fact, if you create a `.bash_profile` file, it would supercede `.profile`\n",
    "\n",
    "You can type all of your aliases into `.profile` or simply cpy them over from your local `.bash_profile`\n",
    "\n",
    "After you have saved the changes to `.profile` and exited the file, be sure to type `source .profile`\n",
    "\n",
    "You must then exit the AWS terminal and relogin. This can be done by typing `exit` into the terminal and the ssh'ing back into AWS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Screening Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say you want to SSH into AWS, and you have a very long script that needs to run. Wouldn't it be awesome to not have to keep that connection open and active in order for the job the finish? What if you could 'screen' in and 'screen' out to check on the jobs progress?! What if I told you this was possible, and that your terminal window would be exactly how you left it the last time you logged in?!\n",
    "\n",
    "With 'screening' - if you ever accidently close terminal or lose connection (internet outage), your terminal will be unaffected and most importantly, your jobs will keep running! yes - I know, it's amazing. You're welcome.\n",
    "\n",
    "To start a 'screen' simply type:\n",
    "\n",
    "> `$: screen`\n",
    "\n",
    "You will see a welcome screen appear. Press enter to get past this. I will show you how to disable this later!\n",
    "\n",
    "To see a list of all possible commands press the following keys on your keyboard:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `?`\n",
    "\n",
    "To see a list of open 'screen' terminal do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `\"`\n",
    "\n",
    "You should currently only see one terminal in the list: \n",
    "\n",
    "> `0 -bash`\n",
    "\n",
    "You can navigate between open screens using the arrow keys and then hitting enter to choose a screen.\n",
    "\n",
    "If you want to rename this terminal window do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `A`\n",
    "\n",
    "You can then type the name you want and hit enter.\n",
    "\n",
    "To open another 'screen' terminal do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    ">`c`\n",
    "\n",
    "To open a new screen and name it at the same time do type the following:\n",
    "\n",
    "> `$: screen -t \"name of terminal here\"`\n",
    "\n",
    "To switch between screen's do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `n #for next screen`\n",
    "\n",
    "> `p # for previous screen`\n",
    "\n",
    "\n",
    "Let's say you wanted to split your terminal window between 2 'screens' - do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `S`\n",
    "\n",
    "To navigate between the two 'screens' - do the following:\n",
    "\n",
    ">`control-a`\n",
    "\n",
    "> `[TAB]`\n",
    "\n",
    "\n",
    "To exit split-screen mode - do the following:\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `Q`\n",
    "\n",
    "If it doesnt immeadiately exit split screen mode, try: `control-L`\n",
    "\n",
    "\n",
    "# The really cool stuff:\n",
    "\n",
    "Let's say you have started a job on one of the 'screens' - now you want to return to your local terminal instance.\n",
    "\n",
    "This is known as **detaching** the screen.\n",
    "\n",
    "Type the following to **detach** - Don't worry, this doesnt stop _any_ of the active 'screens':\n",
    "\n",
    "> `control-a`\n",
    "\n",
    "> `d`\n",
    "\n",
    "You should now see the local instance of your terminal!\n",
    "\n",
    "TO see a list of all active screens from your home terminal, type the following:\n",
    "\n",
    "> `$: screen -list`\n",
    "\n",
    "Note: The first few numbers that are displayed before `.pts` for every screen instance is known as the **PID**\n",
    "\n",
    "To **re-attach** type the following:\n",
    "\n",
    "> `$: screen -r`\n",
    "\n",
    "To **re-attach to a specific screen**:\n",
    "\n",
    "> `$: screen -r PID #type the PID number`\n",
    "\n",
    "**IMPORTANT: you dont have to detach before you close terminal! If you accidently close terminal without detaching, you can just reattach when you re-open terminal. WOOHOOO!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aliasing in Screen!\n",
    "\n",
    "You will notice that your aliases from your local instance of terminal will not be available when you are 'screened'!\n",
    "\n",
    "This can be very annoying, especially if you rely on aliases for a faster workflow!\n",
    "\n",
    "We can fix this - let's also get rid of that welcome page when we start screen!\n",
    "\n",
    "In your **home** directory, do the following:\n",
    "\n",
    "> `touch .screenrc`\n",
    "\n",
    "`.screenrc` is equialent to `.bash_profile` but for 'screens'!\n",
    "\n",
    "Let's do the following:\n",
    "\n",
    "> `nano .screenrc`\n",
    "\n",
    "Then type in the following lines:\n",
    "\n",
    "> `startup_message off #turns of start message`\n",
    "\n",
    "> `defshell -bash #transfers your alises from local isntance of bash!`\n",
    "\n",
    ">`caption always \"%{= Wk}%-w%{= Bw}%n %t%{-}%+w %-=\" #this will add a helpful navigation bar to the bottom of terminal!`\n",
    "\n",
    "Now save the file with:\n",
    "\n",
    "> `control-o`\n",
    "\n",
    "> `[enter]`\n",
    "\n",
    "> `control-x`\n",
    "\n",
    "Then type the following:\n",
    "> `$: source .screenrc`\n",
    "\n",
    "Exit AWS and then re-ssh in and you will be good to go!\n",
    "\n",
    "\n",
    "I hope you found this tutorial helpful - let me know if you have questions!\n",
    "\n",
    "Email: igabr@uchicago.edu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}