tests/benchmarks/benchmarks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmarks for Exdir #\n",
    "\n",
    "This notebook contains a number of benchmarks for Exdir.\n",
    "They compare the performance of Exdir with h5py."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: Please make sure the files are not created in a folder managed by Syncthing, Dropbox or any other file synchronization system. \n",
    "We will be making a large number of changes to the files and a file synchronization system will reduce performance and possibly become out of sync in the process.\n",
    "\n",
    "**Note**: You may experience unreliable results on some systems, where the numbers vary greatly between each run. \n",
    "This can be caused by the large number of I/O operations performed by the benchmarks. \n",
    "We have tried to improve the reliability by adding a call to `time.sleep` between setting up the benchmark and running the benchmark.\n",
    "This should allow the system to completely flush to disk the changes made while setting up and have the benchmark run unaffected.\n",
    "However, if you still experience unreliable results, you may want to try to set up a RAM disk and change the below paths to read `/tmp/ramdis/test.exdir` and `/tmp/ramdisk/test.h5`:\n",
    "\n",
    "    mkdir /tmp/ramdisk/\n",
    "    sudo mount -t tmpfs -o size=2048M tmpfs /tmp/ramdisk/\n",
    "    \n",
    "## Helper functions ##\n",
    "\n",
    "The following functions are used to set up an exdir or hdf5 file for benchmarking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import exdir\n",
    "import os\n",
    "import shutil\n",
    "import h5py\n",
    "\n",
    "def setup_exdir():\n",
    "    testpath = \"test.exdir\"\n",
    "    if os.path.exists(testpath):\n",
    "        shutil.rmtree(testpath)\n",
    "    f = exdir.File(testpath)\n",
    "    return f, testpath\n",
    "\n",
    "def setup_exdir_no_validation():\n",
    "    testpath = \"test.exdir\"\n",
    "    if os.path.exists(testpath):\n",
    "        shutil.rmtree(testpath)\n",
    "    f = exdir.File(testpath, name_validation=exdir.validation.minimal)\n",
    "    return f, testpath\n",
    "\n",
    "def teardown_exdir(f, testpath):\n",
    "    f.close()\n",
    "    shutil.rmtree(testpath)\n",
    "\n",
    "def setup_h5py():\n",
    "    testpath = \"test.h5\"\n",
    "    if os.path.exists(testpath):\n",
    "        os.remove(testpath)\n",
    "    f = h5py.File(testpath)\n",
    "    return f, testpath\n",
    "\n",
    "    \n",
    "def teardown_h5py(f, testpath):\n",
    "    f.close()\n",
    "    os.remove(testpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function is used to run the different benchmarks.\n",
    "It takes a target function to test, a setup function to create the file and the number of iterations the function should be run to get a decent average:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def benchmark(target, setup=None, teardown=None, iterations=10):\n",
    "    total_time = 0\n",
    "    setup_teardown_start = time.time()\n",
    "    for i in range(iterations):\n",
    "        data = tuple()\n",
    "        if setup is not None:\n",
    "            data = setup()\n",
    "        time.sleep(1) # allow changes to be flushed to disk\n",
    "        start_time = time.time()\n",
    "        target(*data)\n",
    "        end_time = time.time()\n",
    "        total_time += end_time - start_time\n",
    "        if teardown is not None:\n",
    "            teardown(*data)\n",
    "    setup_teardown_end = time.time()\n",
    "    total_setup_teardown = setup_teardown_end - setup_teardown_start\n",
    "    \n",
    "    mean = total_time / iterations\n",
    "    \n",
    "    return mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions are used as wrappers to make it easy to run a benchmark of Exdir or h5py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "all_results = []\n",
    "\n",
    "def benchmark_both(function, iterations=10, name_validation=True):\n",
    "    if name_validation:\n",
    "        setup_exdir_ = setup_exdir\n",
    "        name = function.__name__\n",
    "    else:\n",
    "        setup_exdir_ = setup_exdir_no_validation\n",
    "        name = function.__name__ + \" (minimal name validation)\"\n",
    "    \n",
    "    exdir_mean = benchmark(\n",
    "        target=lambda f, path: function(f),\n",
    "        setup=setup_exdir_,\n",
    "        teardown=teardown_exdir,\n",
    "        iterations=iterations\n",
    "    )\n",
    "    hdf5_mean = benchmark(\n",
    "        target=lambda f, path: function(f),\n",
    "        setup=setup_h5py,\n",
    "        teardown=teardown_h5py,\n",
    "        iterations=iterations\n",
    "    )\n",
    "    \n",
    "    result = pd.DataFrame(\n",
    "        [(name, hdf5_mean, exdir_mean, hdf5_mean/exdir_mean)],\n",
    "        columns=[\"Test\", \"h5py\", \"Exdir\", \"Ratio\"]\n",
    "    )\n",
    "    all_results.append(result)\n",
    "    return result\n",
    "\n",
    "def benchmark_exdir(function, iterations=10):\n",
    "    exdir_mean = benchmark(\n",
    "        target=lambda f, path: function(f),\n",
    "        setup=setup_exdir,\n",
    "        teardown=teardown_exdir,\n",
    "        iterations=iterations\n",
    "    )\n",
    "    result = pd.DataFrame(\n",
    "        [(function.__name__, np.nan, exdir_mean, np.nan)],\n",
    "        columns=[\"Test\", \"h5py\", \"Exdir\", \"Ratio\"]\n",
    "    )\n",
    "    all_results.append(result)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to start running the different benchmarks.\n",
    "\n",
    "## Benchmark functions ##\n",
    "\n",
    "The following benchmark creates a small number of attributes.\n",
    "This should be very fast with both h5py and Exdir:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_few_attributes(obj):\n",
    "    for i in range(5):\n",
    "        obj.attrs[\"hello\" + str(i)] = \"world\"\n",
    "\n",
    "benchmark_both(add_few_attributes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following benchmark adds a larger number of attributes one-by-one.\n",
    "Because Exdir needs to read back and rewrite the entire file in case someone changed it between each write, this is significantly slower with Exdir than h5py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_many_attributes(obj):\n",
    "    for i in range(200):\n",
    "        obj.attrs[\"hello\" + str(i)] = \"world\"\n",
    "\n",
    "benchmark_both(add_many_attributes, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, Exdir is capable of writing all attributes in one operation.\n",
    "This makes writing the same attributes about as fast (or even faster than h5py).\n",
    "Writing a large number of attributes in a single operation is not possible with h5py.\n",
    "We therefore need to run this only with Exdir:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_many_attributes_single_operation(obj):\n",
    "    attributes = {}\n",
    "    for i in range(200):\n",
    "        attributes[\"hello\" + str(i)] = \"world\"\n",
    "    obj.attrs = attributes\n",
    "    \n",
    "benchmark_exdir(add_many_attributes_single_operation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exdir also supports adding nested attributes, such as Python dictionaries, which is not supported by h5py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_attribute_tree(obj):\n",
    "    tree = {}\n",
    "    for i in range(100):\n",
    "        tree[\"hello\" + str(i)] = \"world\"\n",
    "    tree[\"intermediate\"] = {}\n",
    "    intermediate = tree[\"intermediate\"]\n",
    "    for level in range(10):\n",
    "        level_str = \"level\" + str(level)\n",
    "        intermediate[level_str] = {}\n",
    "        intermediate = intermediate[level_str]\n",
    "    intermediate = 42\n",
    "    obj.attrs[\"test\"] = tree\n",
    "    \n",
    "benchmark_exdir(add_attribute_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following benchmarks create a small, a medium, and a large dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_small_dataset(obj):\n",
    "    data = np.zeros((100, 100, 100))\n",
    "    obj.create_dataset(\"foo\", data=data)\n",
    "    obj.close()\n",
    "    \n",
    "benchmark_both(add_small_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_medium_dataset(obj):\n",
    "    data = np.zeros((1000, 100, 100))\n",
    "    obj.create_dataset(\"foo\", data=data)\n",
    "    obj.close()\n",
    "    \n",
    "benchmark_both(add_medium_dataset, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_large_dataset(obj):\n",
    "    data = np.zeros((1000, 1000, 100))\n",
    "    obj.create_dataset(\"foo\", data=data)\n",
    "    obj.close()\n",
    "    \n",
    "benchmark_both(add_large_dataset, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is some overhead in creating the objects themselves.\n",
    "This is rather small in h5py, but can be high in Exdir with name validation enabled.\n",
    "This is because the name of every created object must be checked against all the existing objects in the same group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_many_objects(obj):\n",
    "    for i in range(5000):\n",
    "        group = obj.create_group(\"group{}\".format(i))\n",
    "\n",
    "benchmark_both(create_many_objects, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without minimal validation, this is almost as fast in Exdir as it is in h5py.\n",
    "Minimal name validation only checks if file with the exact same name exist in the folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "benchmark_both(create_many_objects, 3, name_validation=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not only the number of created objects matter.\n",
    "Creating them in a tree structure can also incur a performance penalty.\n",
    "The following test creates an object tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_large_tree(obj, level=0):\n",
    "    if level > 4:\n",
    "        return\n",
    "    for i in range(3):\n",
    "        group = obj.create_group(\"group_{}_{}\".format(i, level))\n",
    "        data = np.zeros((10, 10, 10))\n",
    "        group.create_dataset(\"dataset_{}_{}\".format(i, level), data=data)\n",
    "        create_large_tree(group, level + 1)\n",
    "        \n",
    "benchmark_both(create_large_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final benchmark tests writing a \"slice\" of a dataset, which means only a part of the entire dataset is modified.\n",
    "This is typically fast in both h5py and in Exdir thanks to [memory mapping](https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_slice(dataset):\n",
    "    dataset[320:420, 0:300, 0:100] = np.ones((100, 300, 100))\n",
    "\n",
    "def create_setup_dataset(setup_function):\n",
    "    def setup():\n",
    "        f, path = setup_function()\n",
    "        data = np.zeros((1000, 500, 100))\n",
    "        dataset = f.create_dataset(\"foo\", data=data)\n",
    "        time.sleep(1) # allow changes to get flushed to disk\n",
    "        return dataset, f, path\n",
    "    return setup\n",
    "\n",
    "exdir_mean = benchmark(\n",
    "    target=lambda dataset, f, path: write_slice(dataset),\n",
    "    setup=create_setup_dataset(setup_exdir),\n",
    "    teardown=lambda dataset, f, path: teardown_exdir(f, path),\n",
    "    iterations=3\n",
    ")\n",
    "\n",
    "hdf5_mean = benchmark(\n",
    "    target=lambda dataset, f, path: write_slice(dataset),\n",
    "    setup=create_setup_dataset(setup_h5py),\n",
    "    teardown=lambda dataset, f, path: teardown_h5py(f, path),\n",
    "    iterations=3\n",
    ")\n",
    "result = pd.DataFrame(\n",
    "    [(\"write_slice\", hdf5_mean, exdir_mean, hdf5_mean/exdir_mean)],\n",
    "    columns=[\"Test\", \"h5py\", \"Exdir\", \"Ratio\"]\n",
    ")\n",
    "all_results.append(result)\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark summary ##\n",
    "\n",
    "The results are summarized in the following table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat(all_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Profiling the largest differences #\n",
    "\n",
    "While the performance of Exdir in many cases is close to h5py, there are a few cases that can be worth investigating further.\n",
    "\n",
    "For instance, it might be interesting to know what takes most time in create_large_tree, which is about 2-3 times slower in Exdir than h5py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cProfile\n",
    "\n",
    "f, path = setup_exdir()\n",
    "cProfile.run('create_large_tree(f)', sort=\"cumtime\")\n",
    "teardown_exdir(f, path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that `create_dataset` and `create_group` take up about 2/3 and 1/3 of the total run time, respectively.\n",
    "Some of the time in both of these are spent on building paths using pathlib and name validation.\n",
    "The remaining time is mostly spent on writing the array header of the NumPy files.\n",
    "Only a small amount of time is spent on actually writing files.\n",
    "Increasing performance in this case will likely mean that we need to outperform pathlib in building paths and numpy in writing files.\n",
    "While it might be possible, it is also beneficial to stick with the existing, well-tested implementations of both of these libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}