hello_world_pge_create.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PGE Creation\n",
    "This notebook is the driver by which the *hello_world_notebook* will be processed in order to be available as an SDS PGE. The notebook *hello_world_submit_job* is then used to submit it to the SDS for execution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Set-up\n",
    "To start, we; enter the directory containing the notebook to be executed (this step assumes the repository has already been cloned locally), ensure that we are in the proper branch, then execute notebook-pge-wrapper to generate the files necessary to build and run the *papermill*'ed notebook as a PGE.\n",
    "\n",
    "Make note of the branch being used (e.g. *develop* in this example), it will become part of the job specification name and used when submitting the job in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello_world_pge_create.ipynb: needs merge\n",
      ".ipynb_checkpoints is not a notebook, skipping...\n",
      "inspecting notebook: hello_world_notebook.ipynb\n",
      "DEPRECATION WARNING: please prefix parameter (time_limit) with '_' instead of 'hysds_'\n",
      "DEPRECATION WARNING: please prefix parameter (soft_time_limit) with '_' instead of 'hysds_'\n",
      "DEPRECATION WARNING: please prefix parameter (disk_usage) with '_' instead of 'hysds_'\n",
      "DEPRECATION WARNING: please prefix parameter (submission_type) with '_' instead of 'hysds_'\n",
      "DEPRECATION WARNING: please prefix parameter (label) with '_' instead of 'hysds_'\n",
      "generated docker/hysds-io.json.hello_world_notebook\n",
      "generated docker/job-spec.json.hello_world_notebook\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "error: you need to resolve your current index first\n",
      "2021-10-29 21:26:19 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n",
      "2021-10-29 21:26:19 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n",
      "2021-10-29 21:26:19 [INFO] Input Notebook:  notebook_pges/hello_world_notebook.ipynb\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ~/hello_world2\n",
    "git checkout develop\n",
    "notebook-pge-wrapper specs all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manual edits.\n",
    "\n",
    "##### 1. Edit docker/job-spec.json.hello_world_notebook\n",
    "A manual edit to the job-spec file is necessary. In the line:\n",
    "\n",
    ">`    \"command\": \"notebook-pge-wrapper execute /home/ops/hello_world/notebook_pges/hello_world_notebook.ipynb\",`\n",
    " \n",
    "Change *'ops'* to *'jovyan'*:\n",
    "\n",
    ">`    \"command\": \"notebook-pge-wrapper execute /home/jovyan/hello_world/notebook_pges/hello_world_notebook.ipynb\",`\n",
    " \n",
    "##### 2. Update the docker/Dockerfile\n",
    "The contents of the Dockerfile should be as follows:\n",
    "\n",
    "```\n",
    "FROM cae-artifactory.jpl.nasa.gov:16001/gov/nasa/jpl/nisar/sds/pcm/nisar_ade:r1.3\n",
    "\n",
    "ENV HOME=/home/jovyan \\\n",
    "    CONDA_DIR=/opt/conda\n",
    "\n",
    "USER root\n",
    "RUN $CONDA_DIR/bin/conda install -y nb_conda_kernels\n",
    "RUN $CONDA_DIR/bin/pip install 'ipykernel<5.0.0'\n",
    "\n",
    "RUN cd $HOME \\\n",
    "    && git clone https://github.com/hysds/notebook_pge_wrapper.git \\\n",
    "    && cd notebook_pge_wrapper \\\n",
    "    && $CONDA_DIR/bin/pip install -e .\n",
    "\n",
    "USER jovyan\n",
    "\n",
    "# copy your repo into the docker container\n",
    "################################################\n",
    "COPY . $HOME/hello_world\n",
    "################################################\n",
    "\n",
    "WORKDIR $HOME\n",
    "\n",
    "CMD [\"/bin/bash\", \"--login\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Commit changes\n",
    "The notebook as a PGE is obtained by the system via a git checkout, in this case from github, so we need to make sure the remote repository is up to date. There may not be anything to commit, but we need to ensure consistency - there is no harm in trying to commit in the absence of changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ~/hello_world\n",
    "git add -A\n",
    "git commit -m\"Results of notebook-pge-wrapper spec generation.\"\n",
    "git push"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Initialize the api that will be used to specify the PGE job to HySDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "import otello\n",
    "if not os.path.exists(f\"{Path.home()}/.config/otello/config.yml\"):\n",
    "    otello.client.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Specify the repository and branch to the api\n",
    "This creates a handle to the job specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "repository = \"https://github.com/jimmie/hello_world.git\"\n",
    "branch = \"develop\"\n",
    "\n",
    "ci = otello.CI(repo=repository, branch=branch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Ensure the current version of the job is registered\n",
    "If the job exists previously, unregister it. Regardless, register the job to ensure the most current version. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    if (ci.check_job_exists()):\n",
    "        print(\"Job exists, unregistering\")\n",
    "        ci.unregister()\n",
    "except Exception as e:\n",
    "    print(\"Exception : {}, trying to register anyway\".format(e))\n",
    "\n",
    "ci.register()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Build the PGE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ci.submit_build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Wait for the build to complete, then sample the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "move_on = False\n",
    "while not move_on:\n",
    "    try:\n",
    "        ci.get_build_status()\n",
    "        move_on = True\n",
    "    except Exception as e:\n",
    "        print(\"Waiting for build to start\")\n",
    "        time.sleep(10)\n",
    "    \n",
    "build_stat = ci.get_build_status()\n",
    "print(\"Build in progress.\", end=\"\")\n",
    "\n",
    "while (build_stat['building']):\n",
    "    time.sleep(10)\n",
    "    print(\".\", end=\"\", flush=True)\n",
    "    build_stat = ci.get_build_status()\n",
    "\n",
    "if (build_stat['result'] == 'FAILURE'):\n",
    "    print(\"\\nBuild Failed.\")\n",
    "else:\n",
    "    print(\"\\nBuild completed successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Get the final details on the build completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_stat = ci.get_build_status()\n",
    "print(build_stat)\n",
    "if (build_stat['result'] == 'FAILURE'):\n",
    "    print(\"Build Faild.\")\n",
    "else:\n",
    "    print(\"Build completed successfully\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "isce [conda env:.local-isce]",
   "language": "python",
   "name": "conda-env-.local-isce-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}