neos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# `neos`: End-to-End-Optimised Summary Statistics for High Energy Physics\n",
    "\n",
    "## Introduction\n",
    "\n",
    "**Summary statistics** are typically low-dimensional quantities\n",
    "calculated from high-dimensional data, often for the purpose of making\n",
    "any following inference stage easier to compute. They find widespread\n",
    "use in high-energy physics (HEP) data analysis for this reason, since\n",
    "HEP data, e.g. that produced at the Large Hadron Collider, is\n",
    "characterised by both its high dimensionality ($\\mathcal{O}(10^8)$\n",
    "detector read-out channels) as well as the intractability of computing\n",
    "its probability $p(x|\\theta)$ under a given physics hypothesis $\\theta$.\n",
    "A notable example of such a quantity would be the invariant mass of the\n",
    "Higgs boson, which was used in its discovery in 2012 [@2012].\n",
    "\n",
    "Thanks to the presence of high-quality physics simulators, effective\n",
    "machine-learning based summary statistics can be formed using simulated\n",
    "data, typically being trained using classification-driven objectives to\n",
    "maximally distinguish new physics from background processes. While these\n",
    "objectives can be shown to produce optimal results in simplified\n",
    "hypothesis testing settings, the presence of systematic uncertainties in\n",
    "the data may cause classification objectives to become misaligned with\n",
    "physics objectives, e.g. the expected sensitivity to a signal. This\n",
    "means that algorithms trained in this way may under-perform when\n",
    "integrated into the actual physics analysis pipeline, motivating the\n",
    "search for objectives that can better adapt to data with high systematic\n",
    "uncertainty.\n",
    "\n",
    "To combat this, we propose to optimise the summary statistic directly on\n",
    "physics objectives by formulating an *end-to-end-differentiable analysis\n",
    "pipeline*, including density estimation of the summary statistic with\n",
    "histograms, and statistical inference using standard HEP prescriptions.\n",
    "The training can then based on physics-oriented metrics, such as the\n",
    "expected sensitivity, which is based on the profile likelihood ratio: a\n",
    "quantity capable of capturing systematic effects.\n",
    "\n",
    "## Making HEP Analysis Differentiable\n",
    "\n",
    "Given a pre-filtered dataset, an analysis pipeline in HEP involves the\n",
    "following stages:\n",
    "\n",
    "1.  Construction of a learnable 1-D summary statistic from data (with\n",
    "    parameters $\\phi$)\n",
    "\n",
    "2.  Binning of the summary statistic, e.g. through a histogram\n",
    "\n",
    "3.  Statistical model building, using the summary statistic as a\n",
    "    template\n",
    "\n",
    "4.  Calculation of a test statistic, used to perform a frequentist\n",
    "    hypothesis test of signal versus background\n",
    "\n",
    "5.  A $p$-value (or $\\mathrm{CL_s}$[^1] value) resulting from that\n",
    "    hypothesis test, used to characterise the sensitivity of the\n",
    "    analysis\n",
    "\n",
    "We can express this workflow as a direct function of the input dataset\n",
    "$\\mathcal{D}$ and observable parameters $\\phi$:\n",
    "\n",
    "$$\\label{eq:decomp}\n",
    "    \\mathrm{CL_s} = f(\\mathcal{D},\\phi) = (f_{\\mathrm{sensitivity}} \\circ f_{\\mathrm{test\\,stat}} \\circ f_{\\mathrm{likelihood}}  \\circ f_{\\mathrm{histogram}}  \\circ f_{\\mathrm{observable}})(\\mathcal{D},\\phi).$$\n",
    "\n",
    "In the common case where $f_{\\mathrm{observable}}$ is a neural network,\n",
    "it seems possible to optimise this composition end-to-end, i.e. *train\n",
    "the network to directly optimise the analysis sensitivity*. This is\n",
    "exactly the task that `neos` sets out to accomplish, with a full\n",
    "workflow detailed in [\\[fig:pipe\\]](#fig:pipe){reference-type=\"autoref\"\n",
    "reference=\"fig:pipe\"}. To train this network by gradient descent, our\n",
    "choice of $\\mathrm{loss} = \\mathrm{CL_s}$ requires us to be able to\n",
    "calculate $\\partial \\,\\mathrm{CL_s} / \\partial \\phi$. However, this is a\n",
    "stronger condition than it seems, and in fact necessitates the\n",
    "differentiablility of *each individual analysis step* via the chain rule\n",
    "applied to [\\[eq:decomp\\]](#eq:decomp){reference-type=\"autoref\"\n",
    "reference=\"eq:decomp\"}.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Owing to the fact that neural networks are already differentiable, the\n",
    "last term $\\partial f_{\\mathrm{observable}}/{{\\partial \\phi}}$ isn't an\n",
    "issue, but none of the rest of the steps are differentiable by default.\n",
    "The following sections detail solutions for calculating the gradient of\n",
    "each intermediate step.\n",
    "\n",
    "### Binned density estimation (histograms)\n",
    "\n",
    "Histograms are discontinuous by nature. They are defined for 1-D data as\n",
    "a set of two quantities: intervals (or *bins*) over the domain of that\n",
    "data, and counts of the number of data points that fall into each bin.\n",
    "For small changes in the underlying data distribution, bin counts will\n",
    "either remain static, or jump in integer intervals as data migrate\n",
    "between bins, both of which result in ill-defined gradients. We address\n",
    "this inherent non-differentiability through implementing a\n",
    "differentiable surrogate: a histogram based on a *kernel density\n",
    "estimate* (KDE).\n",
    "\n",
    "A KDE is a \"non-parametric\\\" density estimate based on defining a kernel\n",
    "function $K$ centred on each data point. Then, the density at an\n",
    "evaluation point $x$ is the average of the contributions of each kernel\n",
    "function at that point.\n",
    "\n",
    "a with the full density given as $p(t) = 1/n\\sum_i K(t,t_i)$. Normally,\n",
    "a popular kernel function choice is the standard normal distribution,\n",
    "which comes with a parameter called the **bandwidth** that affects the\n",
    "smoothness of the resulting density estimate.\n",
    "\n",
    "Coming back to gradients: in our case, the data $t_i$ we construct the\n",
    "density estimate over are themselves functions of the summary statistic\n",
    "parameters, i.e. $t_i = f(x_i;\\phi)$. The resulting density estimate\n",
    "$p(t|\\phi)$ will then be differentiable as long as the kernel $K$ is\n",
    "differentiable with respect to $t_i$, and by extension with respect to\n",
    "$\\phi$. To extend this differentiability in a binned fashion, we can\n",
    "accumulate the probability mass of the KDE within the bin edges of the\n",
    "original histogram -- equivalent to evaluations of the Gaussian\n",
    "cumulative density function -- to convert $p(t|\\phi)$ to a **binned KDE\n",
    "(bKDE)**, i.e. a set of discrete per-bin probabilities $p_i(\\phi)$.\n",
    "\n",
    "In the limit of vanishing bandwidth, the bKDE recovers the standard\n",
    "histogram, but gradients become susceptible to high variance. Increasing\n",
    "the bandwidth can alleviate this, but at the cost of introducing a bias.\n",
    "There is then a trade-off between decreasing the bandwidth enough to\n",
    "minimise this bias, and increasing it enough to guarantee gradient\n",
    "stability[^2]. We can see a demonstration of this behaviour in\n",
    "[\\[fig:hist\\]](#fig:hist){reference-type=\"autoref\"\n",
    "reference=\"fig:hist\"}, where the bandwidth is tuned relative to the bin\n",
    "width.\n",
    "\n",
    "![Illustration of the bias/smoothness tradeoff when tuning the bandwidth\n",
    "of a bKDE, defined over 200 samples from a bi-modal Gaussian mixture.\n",
    "All distributions are normalised to unit area. The individual kernels\n",
    "that make up the KDE are scaled down for\n",
    "visibility.](relaxed_hist.pdf){#fig:hist width=\"\\\\textwidth\"}\n",
    "\n",
    "### Likelihood modelling\n",
    "\n",
    "A popular framework to build statistical models for binned observations\n",
    "based on \"template\\\" histogram data is HistFactory [@hifa], used widely\n",
    "used across HEP. Thankfully, the resulting log-likelihood function\n",
    "$p_\\mathbf{h}(x)$ is differentiable with respect to both the observed\n",
    "data $x$ and histogram data $\\mathbf{h}$ from which the model is\n",
    "constructed. However, these gradients only recently became readily\n",
    "accessible in software, owing to the development of `pyhf`\n",
    "[@pyhf; @pyhf2]: a Python package for building these likelihoods that\n",
    "leverages automatic differentiation.\n",
    "\n",
    "### Test statistics (profile likelihood ratio)\n",
    "\n",
    "A number of test statistics are typically used in HEP[^3], all of which\n",
    "build on the profile likelihood ratio. For likelihood function $p$,\n",
    "input data $x$, parameters of interest (POIs) $\\mu$, and nuisance\n",
    "parameters $\\psi$, we define the profile likelihood ratio as\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\lambda_\\mu(x) = \\frac{p(x|\\mu,%\n",
    "    \\settoheight{\\dhatheight}{\\ensuremath{\\hat{\\psi}}}%\n",
    "    \\addtolength{\\dhatheight}{-0.35ex}%\n",
    "    \\hat{\\vphantom{\\rule{1pt}{\\dhatheight}}%\n",
    "    \\smash{\\hat{\\psi}}}(\\mu))}{p(x|\\hat{\\mu},\\hat{\\psi})},\\end{aligned}$$\n",
    "where $\\hat{\\mu},\\hat{\\psi}$ denote the maximum-likelihood parameters,\n",
    "whilst $%\n",
    "    \\settoheight{\\dhatheight}{\\ensuremath{\\hat{\\psi}}}%\n",
    "    \\addtolength{\\dhatheight}{-0.35ex}%\n",
    "    \\hat{\\vphantom{\\rule{1pt}{\\dhatheight}}%\n",
    "    \\smash{\\hat{\\psi}}}(\\mu)$ denotes the maximum-likelihood estimate of\n",
    "$\\psi$ at fixed values of $\\mu$. A test statistic $q_\\mu$ is then built\n",
    "upon the profile likelihood, following prescriptions detailed in\n",
    "[@asymptotics]. Our physics objective, i.e. the expected\n",
    "$\\mathrm{CL}_s$, is computed from distributions of $q_\\mu$ for the null\n",
    "and alternative hypotheses, which correspond to choosing different\n",
    "values of $\\mu$. In asymptotic theory, the null is distributed as a\n",
    "$\\chi^2$-distribution, while the alternative follows a non-central\n",
    "$\\chi^2_\\mathrm{nc}(x,\\Lambda^2)$ distribution, with $\\Lambda^2$ given\n",
    "by $$\\Lambda^2 = (\\mu-\\mu')^2/\\sigma_{\\hat{\\mu}}^2,$$ where $\\mu$,\n",
    "$\\mu'$ are the POI values corresponding to the null and alternative\n",
    "hypothesis respectively, and $\\sigma_{\\hat{\\mu}}$ is the variance for\n",
    "the maximum likelihood estimate of the POIs. Our goal of minimising the\n",
    "expected $\\mathrm{CL}_s$ is then equivalent to both maximising\n",
    "$\\Lambda^2$ and minimising $\\sigma_{\\hat{\\mu}}$.\n",
    "\n",
    "While the INFERNO approach uses the Fisher Information to estimate and\n",
    "minimise $\\sigma_{\\hat{\\mu}}$, we take the route of computing\n",
    "$\\Lambda^2$ as the test statistic value for the \"Asimov dataset\" of the\n",
    "alternative hypothesis. As described in [@asymptotics], the use of\n",
    "Asimov test statistics is known to yield better approximations of the\n",
    "true sampling distributions, and therefore of the final objective\n",
    "$\\mathrm{CL_s}$. However, the drawback of this approach is the inclusion\n",
    "of optimisation routines such as the computation of $%\n",
    "    \\settoheight{\\dhatheight}{\\ensuremath{\\hat{\\psi}}}%\n",
    "    \\addtolength{\\dhatheight}{-0.35ex}%\n",
    "    \\hat{\\vphantom{\\rule{1pt}{\\dhatheight}}%\n",
    "    \\smash{\\hat{\\psi}}}$, which are not a-priori differentiable in an\n",
    "efficient manner. We thus explore methods for derivatives of\n",
    "optimisation routines that only require gradient information when\n",
    "optimisation has converged.\n",
    "\n",
    "#### Implicit differentiation of optimisation routines:\n",
    "\n",
    "Optimisation algorithms find best-fit values $\\hat{\\theta}$ for a given\n",
    "objective function $f$. In the case of the profile likelihood, the\n",
    "objective $f$ is the likelihood function, which implicitly depends on\n",
    "the summary statistic parameters $\\phi$; we denote this by $f = f_\\phi$,\n",
    "and extend this dependence to the best-fit parameters\n",
    "$\\hat{\\theta}(\\phi)$. Iterative optimisation finds $\\hat{\\theta}$ by\n",
    "starting at a initial value $\\theta_0$ and repeatedly updating it\n",
    "through a iteration step $\\hat{\\theta}_{i+1}=u(f_\\phi, \\theta_i)$ until\n",
    "a terminating condition is satisfied. An explicit expression for\n",
    "$\\hat{\\theta}(\\phi)$ capturing the dependence on $\\phi$ is not readily\n",
    "available. Despite this, we can still compute gradients of\n",
    "$\\hat{\\theta}$ by noting that at the solution $\\theta_i = \\hat{\\theta}$,\n",
    "the following relation holds for any $\\phi$:\n",
    "$$\\hat{\\theta} = u(f_\\phi,\\hat{\\theta}) ~\\Rightarrow~ g(\\phi,\\hat{\\theta}) = u(f_\\phi,\\hat{\\theta}) - \\hat{\\theta} = 0,$$\n",
    "From here, we can leverage a powerful result from the *implicit function\n",
    "theorem* that locally guarantees the existence of a continuously\n",
    "differentiable function $\\hat{\\theta}(\\phi)$, with gradients given by\n",
    "$$\\label{eq:fi}\n",
    "    \\frac{\\partial\\hat{\\theta}(\\phi)}{\\partial\\phi} = -\\left[\\frac{\\partial g}{\\partial \\theta}\\right]^{-1}  \\frac{\\partial g}{\\partial \\phi} = \\left[I - \\frac{\\partial u}{\\partial \\theta}\\right]^{-1} \\frac{\\partial u}{\\partial \\phi}$$\n",
    "This is our saving grace: by evaluating derivatives of *just the update\n",
    "step* with respect to the optimised parameters $\\theta$ and the\n",
    "parameters implicitly defining the objective $\\phi$, we can derive\n",
    "gradients of the resulting optimisation solution without unrolling all\n",
    "iterations. Implicit differentiation through optimisation procedures\n",
    "like this has been implemented in software by the authors of the\n",
    "`jaxopt` package [@jaxopt], which we use in our experiments. We refer\n",
    "the reader to Ref. [@implicit] for more detail on the connection between\n",
    "[\\[eq:fi\\]](#eq:fi){reference-type=\"autoref\" reference=\"eq:fi\"} and\n",
    "automatic differentiation.\n",
    "\n",
    "This result lets us compute the non-centrality parameter $\\Lambda^2$ of\n",
    "the hypothesis test (and as a corollary, the expected $\\mathrm{CL_s}$)\n",
    "in a fully differentiable manner, providing the final building block of\n",
    "an end-to-end differentiable analysis pipeline as described in the\n",
    "beginning of this section.\n",
    "\n",
    "## Demonstrating `neos`: End-to-end Analysis Optimisation\n",
    "\n",
    "We demonstrate the principle of end-to-end optimisation by training a\n",
    "neural network optimised directly on the physics sensitivity, expressed\n",
    "as the expected $\\mathrm{CL_s}$ value of a `pyhf` model.\n",
    "\n",
    "### Gaussian blob experiment\n",
    "\n",
    "![Analysis metrics evaluated on a test set when running the `neos`\n",
    "pipeline with a variety of loss functions, averaged over 7\n",
    "initialisations of the neural network weights. **Left**: the expected\n",
    "$\\mathrm{CL_s}$ with a null hypothesis of no signal. **Middle**: the\n",
    "uncertainty on the signal strength $\\mu$. **Right**: deviation of the\n",
    "nuisance parameter uncertainty $\\gamma$ from its nominal value of\n",
    "1.](metric_comparison_fewer.pdf){#fig:metrics width=\"\\\\textwidth\"}\n",
    "\n",
    "We consider an example where we generate toy physics data $(x, y)$ from\n",
    "2-D Gaussian blobs with covariances equal to the 2-D identity matrix\n",
    "$I_2$.\n",
    "\n",
    "The dataset $\\mathcal{D}$ consists of signal\n",
    "${s} \\sim \\mathcal{N}(m_s, I_2)$, background\n",
    "$b \\sim \\mathcal{N}(m_b, I_2)$, and two variations of the background\n",
    "data $b\\mathrm{_{up}}, {b_{\\mathrm{down}}}\\}$ that mimic simulating the\n",
    "background process after varying an imagined physical parameter $\\Psi$\n",
    "up and down by one standard deviation, which is a common situation\n",
    "encountered in HEP. In practice, $\\Psi$ is taken to influence the mean\n",
    "of the background distribution implicitly through generating blob data\n",
    "with different means, i.e.\n",
    "${b\\mathrm{_{up}}} \\sim \\mathcal{N}(m_{b{\\mathrm{up}}}, I_2)$,\n",
    "${b_{\\mathrm{down}}} \\sim \\mathcal{N}(m_{b{\\mathrm{down}}}, I_2)$.\n",
    "\n",
    "After dataset generation, as well as splitting into train and test\n",
    "datasets, the workflow then follows the steps outlined in\n",
    "[\\[fig:pipe\\]](#fig:pipe){reference-type=\"autoref\"\n",
    "reference=\"fig:pipe\"}. Results from running this experiment can be found\n",
    "in [\\[fig:metrics\\]](#fig:metrics){reference-type=\"autoref\"\n",
    "reference=\"fig:metrics\"}, which compares `neos` to three baseline loss\n",
    "functions:\n",
    "\n",
    "-   Binary cross-entropy (BCE): signal vs. nominal background\n",
    "\n",
    "-   BCE, but using up/down systematic variations as data augmentation\n",
    "    for the background sample. This is a powerful baseline that\n",
    "    maximally isolates the signal points from all background variations.\n",
    "\n",
    "-   INFERNO: using the signal strength uncertainty $\\sigma_\\mu^2$ as the\n",
    "    objective, calculated from the Hessian of the likelihood via\n",
    "    automatic differentiation.\n",
    "\n",
    "Three metrics are shown: the expected $\\mathrm{CL_s}$, the parameter of\n",
    "interest ($\\mu$) uncertainty, and the nuisance parameter ($\\gamma$)\n",
    "uncertainty, all evaluated on the test set[^4]. All metrics are averaged\n",
    "over seven different initialisations of the weights $\\phi$.\n",
    "Hyperparameters are given in [Appendix A](#app1).\n",
    "\n",
    "We can see that `neos` is able to reach sensitivities lower than any\n",
    "other loss function (left plot). Moreover, it is able to gradually\n",
    "improve the uncertainty on the parameter of interest (middle plot)\n",
    "without any additional tuning or regularisation, far outperforming\n",
    "binary cross-entropy in this regard. When mixing in the up/down\n",
    "systematic variations as part of the background class when using BCE, we\n",
    "are able to get similar performance to `neos` with lower $\\mu$\n",
    "uncertainty around epoch 4, but `neos` is able to improve to match this\n",
    "with more training, all while retaining a lower $\\mathrm{CL_s}$. INFERNO\n",
    "is able to achieve the best uncertainty on $\\mu$, but fails to reach the\n",
    "same expected sensitivity as other methods.\n",
    "\n",
    "We additionally examine the uncertainty on $\\gamma$ (right plot), also\n",
    "taken from the Fisher information matrix. It indicates whether the\n",
    "analysis is able to determine the systematic effect on the background\n",
    "shape in a way that differs from the constraint on $\\gamma$ provided by\n",
    "the up/down variations. This is typically undesirable behaviour, as it\n",
    "indicates a strong dependence of the summary statistic on the nuisance\n",
    "parameters. Here, we see that `neos` and BCE (with aug.) are able to\n",
    "keep this metric fairly low, while BCE and INFERNO show signs of over-\n",
    "or under- constraining $\\gamma$.\n",
    "\n",
    "## Discussion and Future Directions\n",
    "\n",
    "One of the more immediate concerns for using `neos` in practice is the\n",
    "issue of scaling. Training times compared to BCE in the toy problem are\n",
    "a factor of 3 or so higher, since the processing of one batch\n",
    "corresponds to running the all the downstream parts of a physics\n",
    "analysis. In addition, the batch size needs to be large enough to\n",
    "faithfully represent the analysis, hopefully avoiding situations with\n",
    "empty bins. This is an issue that is currently being investigated; a\n",
    "medium/long term goal would be to scale `neos` in application to open\n",
    "data from the major LHC experiments, such as ATLAS and CMS, allowing the\n",
    "probing of how the performance benefits highlighted in section 3 scale\n",
    "to realistic problems with many sources of uncertainty.\n",
    "\n",
    "Despite the focus on summary statistics, the work done in enabling the\n",
    "differentiability of HEP analysis opens up a variety of new approaches\n",
    "in optimising any free parameters end-to-end. For example, one could\n",
    "imagine optimising the pre-selection stage that occurs when filtering\n",
    "data prior to inference, since one can differentiably approximate a cut\n",
    "with a sigmoid function. Moreover, one could optimise the binning of an\n",
    "observable for the best sensitivity, given a fixed number of bins.\n",
    "\n",
    "An additional, exploratory direction is considering the use of a\n",
    "multi-term objective function with weighted components. The software\n",
    "released alongside this work already enables objectives of the form\n",
    "$a_0{\\mathrm{CL_s}} + a_1{\\sigma_\\mu} + a_2{(1-\\sigma_\\gamma)^2} + \\dots$,\n",
    "which could offer regularisation effects to steer away from pathologies\n",
    "that happen to satisfy one metric only. Work in this direction is also\n",
    "ongoing.\n",
    "\n",
    "A future comparison to black box methods such as Bayesian optimisation\n",
    "would also be desirable, as they do not require the use of\n",
    "approximations to enable optimisation, but still can target the physics\n",
    "goals of choice.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "The typical HEP analysis workflow has been made differentiable, and a\n",
    "use case for it in the systematic-aware optimisation of a neural network\n",
    "has been demonstrated. Comparison was made to the more standard approach\n",
    "of using binary cross-entropy, as well as more competitive baselines\n",
    "like systematic data augmentation and INFERNO. Improvements were shown\n",
    "in both the expected analysis sensitivity and the properties of the\n",
    "likelihood function. Accompanying software to run the experiments\n",
    "detailed in Section 3 can be found at [@neos], as well as a toolbox\n",
    "called `relaxed` [@relaxed] that enables the isolated or combined use of\n",
    "the differentiable components used to assemble `neos`.\n",
    "\n",
    "We would like to thank the authors of INFERNO (Pablo de Castro, Tommaso\n",
    "Dorigo) for initial productive discussions. We also extend thanks to\n",
    "Kyle Cranmer, Alexander Held, Giordon Stark, and Matthew Feickert for\n",
    "conceptual and technical support in the development of this project.\n",
    "Visualisations are produced using the `matplotlib` package [@mpl]. LH is\n",
    "supported by the Excellence Cluster ORIGINS, which is funded by the\n",
    "Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under\n",
    "Germany's Excellence Strategy - EXC-2094-390783311. NS completed this\n",
    "work with financial support from Insights, which is funded by the\n",
    "European Union's Horizon 2020 research and innovation programme, call\n",
    "H2020-MSCA-ITN-2017, under Grant Agreement n. 765710. NS extends\n",
    "gratitude to friends and colleagues that indirectly facilitated this\n",
    "work, particularly to those in the \"noonch\" lunch group at CERN.\n",
    "\n",
    "##  {#app1}\n",
    "\n",
    "Hyperparameters for the Gaussian blob study in Section 3:\n",
    "\n",
    "-   10000 data points, split evenly between all four blobs,\n",
    "\n",
    "-   3-layer neural network of size (1024, 1024, 1),\n",
    "\n",
    "-   Training with Adam optimiser, learning rate 1e-3,\n",
    "\n",
    "-   Adam optimiser also used in maximum likelihood fits with learning\n",
    "    rate 1e-2,\n",
    "\n",
    "-   $m_\\mathrm{s}=(-1, 1)$, $m_\\mathrm{b}=(2.5, 2)$,\n",
    "    $m_\\mathrm{bup}=(-2.5, -1.5)$, $m_\\mathrm{bdown}=(1, -1)$,\n",
    "\n",
    "-   Multiplicative histogram scale factors: signal scale=2, background\n",
    "    scale=10, global scale=10,\n",
    "\n",
    "-   ReLU activations, with sigmoid activation on the final layer,\n",
    "\n",
    "-   15 epochs, with a batch size of 2000.\n",
    "\n",
    "# References {#references .unnumbered}\n",
    "\n",
    "[^1]: $\\mathrm{CL_s}$ is a modification of the p-value that protects\n",
    "    against rejecting the null hypothesis when the test is not sensitive\n",
    "    to the alternative hypothesis (e.g. through largely overlapping test\n",
    "    statistic distributions).\n",
    "\n",
    "[^2]: How low is low enough when tuning? This relationship between bias\n",
    "    and gradient stability is heavily impacted by the number of data\n",
    "    points, and also by the width of the intervals, with more\n",
    "    exploration of this planned.\n",
    "\n",
    "[^3]: A full list of the different test statistics and their purposes\n",
    "    can be found in [@asymptotics].\n",
    "\n",
    "[^4]: During test evaluation and training with binary cross-entropy, no\n",
    "    differentiable approximations are used.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "651f2bad27d44a91877bf2ed58cd15f6a905a1a50516233d74eca5c893c3c6fd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}