1_metsim_command_line_usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to MetSim: Command line usage\n",
    "\n",
    "Welcome to the first interactive component to our tutorial!\n",
    "Now that we've got a handle on all things MetSim, let's actually _run_ it.\n",
    "Predominantly, MetSim was designed to be a command line tool run through a shell interface.\n",
    "Because we are in a Jupyter notebook we have to first discuss some technology related things to get set up.\n",
    "\n",
    "Jupyter can run multiple _kernels_ which are the core programming languages or environments.\n",
    "In fact, Jupyter actually stands for [**Ju**lia](https://julialang.org/), [**Pyt**hon](https://www.python.org/), and [**R**](https://www.r-project.org/).\n",
    "Beyond this there are many community maintained kernels for almost any imaginable programming language.\n",
    "This includes some shells, such as [bash](https://www.gnu.org/software/bash/).\n",
    "However, we will not be taking the route of installing a bash kernel, but will instead rely on _cell magic_.\n",
    "\n",
    "Cell magics are invoked with the `%` character. \n",
    "Some examples include `time`, `load`, and `who`. \n",
    "We can also use `%%bash` to invoke bash, rather than python, in a cell.\n",
    "Using the cell magic to invoke bash will allow us to switch cleanly bewtween bash and python.\n",
    "We will use bash to examine the inputs, as well as run MetSim itself.\n",
    "Then, we will use python do do some simple processing and visualization of the output.\n",
    "Following that, we will run MetSim from python itself and do some more advanced output processing.\n",
    "\n",
    "Let's try using the `%%bash` magick real quick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "for i in `seq 3`; do\n",
    "    echo \"hello ${i}!\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this in mind, let's talk about how we do things with MetSim via the command line. \n",
    "As you already know from the background, MetSim uses a configuration file to specify input data and various options.\n",
    "Let's start by looking at it.\n",
    "We have named it `test.conf`, and it's sitting right here in the same directory as this notebook.\n",
    "In bash, you can view the contents of a file by invoking a program called `cat`.\n",
    "So, to view the contents of the configuration, do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cat test.conf\n",
    "echo \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Great! \n",
    "As you can see, the configuration file is relatively short, and mostly consists of setting variables.\n",
    "Notice however, that there are several sections delineated by `[section_name]` headers.\n",
    "This sort of file format is known as INI, and is relatively common for simple configurations.\n",
    "Also note that comments in the configuration file begin with `#` just like python.\n",
    "Let's dig into each of these sections separately.\n",
    "\n",
    "### `[MetSim]`\n",
    "This is the base header, which ensures that the config file is, in fact, for MetSim.\n",
    "It controls general top level variables of interest.\n",
    "Let's walk through them. We will examine just the variable names, and leave the values to description only if necessary.\n",
    "\n",
    "* `out_vars` - this list contains all of the variables which we wish to write to the output data. For a list of the available options see [this page of the documentation](https://metsim.readthedocs.io/en/develop/data.html#output-specifications)\n",
    "* `time_step` - this is the time step of the output (in minutes). There are some limiataions on what this is allowed to be. Of course, it can't be greater than 1440 (number of minutes in a day). But it also must be less than 360 (or 6 hours) for numerical reasons. We also should note that, while you can go down to output on the minute, it really doesn't make sense to set this lower than 30.\n",
    "* `start` - The start date of the simulation, given as `YYYY/MM/DD`\n",
    "* `stop` - The stop date of the simulation, given as `YYYY/MM/DD`\n",
    "* `forcing` - The path to the forcing file(s). Multiple files can be given.\n",
    "* `domain` - The path to the domain file\n",
    "* `state` - The path to the state file\n",
    "* `out_dir` - The location to write the output to. This must exist before running.\n",
    "* `output_prefix` - The prefix that the output file(s) will get.\n",
    "* `forcing_fmt` - The format of the input forcings. This can be `netcdf`, `binary`, or `ascii`. We will discuss each of these later.\n",
    "\n",
    "### `[chunks]`\n",
    "This section specifies how the domain will be broken up spatially. \n",
    "The variable names in this section are not fixed.\n",
    "In this case we see\n",
    "```\n",
    "lat = 3\n",
    "lon = 1\n",
    "```\n",
    "specifies that our domain is on a latitude longitude grid, and each 1 by 9 chunk of the domain will be run as a separate job.\n",
    "Later in the tutorial we will see a configuration which will be \n",
    "```\n",
    "hru = 3\n",
    "```\n",
    "whose domain is on an unstructured mesh.\n",
    "\n",
    "### `[*_vars]` sections\n",
    "These sections allow you to specify the mapping between variable names that the input datasets have to the ones that MetSim expects.\n",
    "The format here is `metsim_varname = data_varname`, generally. \n",
    "The only exception to that rule will be explored later, such as when `forcing_fmt = binary`.\n",
    "\n",
    "--- \n",
    "\n",
    "# Moving on\n",
    "\n",
    "With the configuration in mind, let's look at how we run MetSim from the command line.\n",
    "Because we are running using the cell magic we will have to activate our conda environment every time we wish to interact with MetSim.\n",
    "Installing MetSim will add the `ms` command to your path.\n",
    "Let's see it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "source activate metsim_whw\n",
    "which ms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's talk options\n",
    "\n",
    "Normally you could just run `ms --help` to view the options.\n",
    "But, because we are running through the notebook interface we can't do this. \n",
    "Instead here's what you would see if you ran `ms --help`:\n",
    "\n",
    "```\n",
    "usage: ms [-h] [-n NUM_WORKERS] [-s SCHEDULER] [-v] config\n",
    "\n",
    "positional arguments:\n",
    "  config                Input configuration file\n",
    "\n",
    "optional arguments:\n",
    "  -h, --help            show this help message and exit\n",
    "  -n NUM_WORKERS, --num_workers NUM_WORKERS\n",
    "                        Parallel mode: number of processes to use\n",
    "  -s SCHEDULER, --scheduler SCHEDULER\n",
    "                        Dask scheduler to use\n",
    "  -v, --verbose         Increase the verbosity of MetSim\n",
    "```\n",
    "\n",
    "Alright, so we can just run with `ms config`. Let's do it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time to run\n",
    "Now, we can just run MetSim by invoking the `ms` command with options as shown above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "source activate metsim_whw\n",
    "ms test.conf -v -n 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We did it!\n",
    "Alright, so despite some warnings we were able to complete our run!\n",
    "Now, let's look at our output.\n",
    "We can open the output as an xarray dataset, and do some plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import xarray as xr\n",
    "output = xr.open_dataset('./output/test_19500101-19500131.nc')\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(12, 8))\n",
    "axes = axes.flatten()\n",
    "unit_str = ['($ ^o C$)', '($mm$)', '($w/m^2$)','($w/m^2$)', '($kPa$)', '($\\%$)',]\n",
    "\n",
    "\n",
    "variables = list(output.variables.keys())\n",
    "dims = list(output.dims.keys())\n",
    "[variables.remove(d) for d in dims]\n",
    "\n",
    "start = 100\n",
    "stop = 148\n",
    "for idx, var in enumerate(variables):\n",
    "    output[var].isel(lat=1, lon=1, time=slice(start, stop)).plot(ax=axes[idx])\n",
    "    axes[idx].set_title('') \n",
    "    axes[idx].set_ylabel('{} {}'.format(output[var].long_name.capitalize(), unit_str[idx]))\n",
    "    axes[idx].set_xlabel('Date')\n",
    "plt.tight_layout()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "metsim_whw",
   "language": "python",
   "name": "metsim_whw"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}