hello_world_pge_create.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PGE Creation\n",
    "This notebook is the driver by which the *hello_world_notebook* will be processed in order to be available as an SDS PGE. The notebook *hello_world_submit_job* is then used to submit it to the SDS for execution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Set-up\n",
    "To start, we; enter the directory containing the notebook to be executed (this step assumes the repository has already been cloned locally), ensure that we are in the proper branch, then execute notebook-pge-wrapper to generate the files necessary to build and run the *papermill*'ed notebook as a PGE.\n",
    "\n",
    "Make note of the branch being used (e.g. *develop* in this example), it will become part of the job specification name and used when submitting the job in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "M\thello_world_pge_create.ipynb\n",
      "M\thello_world_submit_job.ipynb\n",
      "M\tnotebook_pges/hello_world_notebook.ipynb\n",
      "Your branch is up to date with 'origin/develop'.\n",
      ".ipynb_checkpoints is not a notebook, skipping...\n",
      "inspecting notebook: hello_world_notebook.ipynb\n",
      "generated docker/hysds-io.json.hello_world_notebook\n",
      "generated docker/job-spec.json.hello_world_notebook\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Already on 'develop'\n",
      "2021-02-03 16:16:27 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n",
      "2021-02-03 16:16:27 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n",
      "2021-02-03 16:16:27 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ~/hello_world\n",
    "git checkout develop\n",
    "notebook-pge-wrapper specs all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manual edits.\n",
    "\n",
    "##### 1. Edit docker/job-spec.json.hello_world_notebook\n",
    "A manual edit to the job-spec file is necessary. In the line:\n",
    "\n",
    ">`    \"command\": \"notebook-pge-wrapper execute /home/ops/hello_world/notebook_pges/hello_world_notebook.ipynb\",`\n",
    " \n",
    "Change *'ops'* to *'jovyan'*:\n",
    "\n",
    ">`    \"command\": \"notebook-pge-wrapper execute /home/jovyan/hello_world/notebook_pges/hello_world_notebook.ipynb\",`\n",
    " \n",
    "##### 2. Update the docker/Dockerfile\n",
    "The contents of the Dockerfile should be as follows:\n",
    "\n",
    "```\n",
    "FROM cae-artifactory.jpl.nasa.gov:16001/gov/nasa/jpl/nisar/sds/pcm/nisar_ade:r1.3\n",
    "\n",
    "ENV HOME=/home/jovyan \\\n",
    "    CONDA_DIR=/opt/conda\n",
    "\n",
    "USER root\n",
    "RUN $CONDA_DIR/bin/conda install -y nb_conda_kernels\n",
    "RUN $CONDA_DIR/bin/pip install 'ipykernel<5.0.0'\n",
    "\n",
    "RUN cd $HOME \\\n",
    "    && git clone https://github.com/hysds/notebook_pge_wrapper.git \\\n",
    "    && cd notebook_pge_wrapper \\\n",
    "    && $CONDA_DIR/bin/pip install -e .\n",
    "\n",
    "USER jovyan\n",
    "\n",
    "# copy your repo into the docker container\n",
    "################################################\n",
    "COPY . $HOME/hello_world\n",
    "################################################\n",
    "\n",
    "WORKDIR $HOME\n",
    "\n",
    "CMD [\"/bin/bash\", \"--login\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Commit changes\n",
    "The notebook as a PGE is obtained by the system via a git checkout, in this case from github, so we need to make sure the remote repository is up to date. There may not be anything to commit, but we need to ensure consistency - there is no harm in trying to commit in the absence of changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[develop 92bacc1] Results of notebook-pge-wrapper spec generation.\n",
      " 3 files changed, 70 insertions(+), 71 deletions(-)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://github.com/jimmie/hello_world\n",
      "   ed58e85..92bacc1  develop -> develop\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ~/hello_world\n",
    "git add -A\n",
    "git commit -m\"Results of notebook-pge-wrapper spec generation.\"\n",
    "git push"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Initialize the api that will be used to specify the PGE job to HySDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "import otello\n",
    "if not os.path.exists(f\"{Path.home()}/.config/otello/config.yml\"):\n",
    "    otello.client.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Specify the repository and branch to the api\n",
    "This creates a handle to the job specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "repository = \"https://github.com/jimmie/hello_world.git\"\n",
    "branch = \"develop\"\n",
    "\n",
    "ci = otello.CI(repo=repository, branch=branch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Ensure the most current version of the job is registered\n",
    "If the job exists previously, unregister it. Regardless, register the job to ensure the most current version. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job exists, unregistering\n",
      "[2021-02-03 17:11:54,921: DEBUG/sdscli/ci] sds_type: hysds\n",
      "[2021-02-03 17:11:54,922: DEBUG/sdscli/__init__] file: /export/home/hysdsops/.sds/config\n",
      "[2021-02-03 17:11:55,134: DEBUG/sdscli/get_func] mod: <module 'sdscli.adapters.hysds.ci' from '/export/home/hysdsops/mozart/ops/sdscli-1.0.6/sdscli/adapters/hysds/ci.py'>\n",
      "[2021-02-03 17:11:55,134: DEBUG/sdscli/get_func] func_name: add_job\n",
      "[2021-02-03 17:11:55,134: DEBUG/sdscli/ci] func: <function add_job at 0x7fcae001f5e0>\n",
      "[2021-02-03 17:11:55,134: DEBUG/sdscli/__init__] file: /export/home/hysdsops/.sds/config\n",
      "[100.64.122.201] Executing task 'add_ci_job'\n",
      "[100.64.122.201] run: mkdir -p /var/lib/jenkins/jobs/nisar-jdyoung-21_container-builder_jimmie_hello_world_develop\n",
      "[100.64.122.201] run: chmod -R 777 /var/lib/jenkins/jobs/nisar-jdyoung-21_container-builder_jimmie_hello_world_develop\n",
      "[100.64.122.201] put: <file obj> -> tmp-jenkins-upload\n",
      "[100.64.122.201] run: cp -rp tmp-jenkins-upload /var/lib/jenkins/jobs/nisar-jdyoung-21_container-builder_jimmie_hello_world_develop/config.xml\n",
      "[100.64.122.201] run: rm tmp-jenkins-upload\n",
      "[100.64.122.201] Executing task 'reload_configuration'\n",
      "[100.64.122.201] run: java -jar /var/lib/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080 -http -auth jdyoung:110473865ddbff91cd3db65a4204f1ba0d reload-configuration\n",
      "\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    if (ci.check_job_exists()):\n",
    "        print(\"Job exists, unregistering\")\n",
    "        ci.unregister()\n",
    "except Exception as e:\n",
    "    print(\"Exception : {}, trying to register anyway\".format(e))\n",
    "\n",
    "ci.register()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Build the PGE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'success': True,\n",
       " 'message': 'job successfully submitted to build nisar-jdyoung-21_container-builder_jimmie_hello_world_develop (1)',\n",
       " 'queue': 50056,\n",
       " 'build_number': 1}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ci.submit_build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Wait for the build to complete, then sample the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for build to start\n",
      "Build in progress.........................................................................\n",
      "Build completed successfully\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "move_on = False\n",
    "while not move_on:\n",
    "    try:\n",
    "        ci.get_build_status()\n",
    "        move_on = True\n",
    "    except Exception as e:\n",
    "        print(\"Waiting for build to start\")\n",
    "        time.sleep(10)\n",
    "    \n",
    "build_stat = ci.get_build_status()\n",
    "print(\"Build in progress.\", end=\"\")\n",
    "\n",
    "while (build_stat['building']):\n",
    "    time.sleep(10)\n",
    "    print(\".\", end=\"\", flush=True)\n",
    "    build_stat = ci.get_build_status()\n",
    "\n",
    "if (build_stat['result'] == 'FAILURE'):\n",
    "    print(\"\\nBuild Failed.\")\n",
    "else:\n",
    "    print(\"\\nBuild completed successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Get the final details on the build completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'job_name': 'nisar-jdyoung-21_container-builder_jimmie_hello_world_develop', 'build_number': 1, 'result': 'SUCCESS', 'timestamp': 1612372332629, 'url': 'https://nisar-pcm-ci.jpl.nasa.gov/job/nisar-jdyoung-21_container-builder_jimmie_hello_world_develop/1/', 'duration': 938114, 'building': False}\n",
      "Build completed successfully\n"
     ]
    }
   ],
   "source": [
    "build_stat = ci.get_build_status()\n",
    "print(build_stat)\n",
    "if (build_stat['result'] == 'FAILURE'):\n",
    "    print(\"Build Faild.\")\n",
    "else:\n",
    "    print(\"Build completed successfully\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}