go live

Co-authored-by: Alex Eftimiades <[email protected]>
Co-authored-by: matthewgillett <[email protected]>

DCO

@@ -0,0 +1,8 @@
+Developer's Certificate of Origin (adapted from the linux kernel)
+
+By making a contribution to this project, I certify that:
+
+The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.

README.md

@@ -1 +1,29 @@
-# model-validation-toolkit
+# Installation
+Download and `cd` into this repository.
+
+Run `pip install .`.
+
+**Windows users**: Until [Jaxlib is supported on windows
+natively](https://github.com/google/jax/issues/438) you will need to either use
+this library from you Linux subsystem or within a Docker container.
+Alternatively, you can [build jaxlib from
+source](https://jax.readthedocs.io/en/latest/developer.html#additional-notes-for-building-jaxlib-from-source-on-windows).
+
+## Developers
+Download and `cd` into this repository.
+
+Run `pip install -e ".[doc]"`.
+
+The `[doc]` is used to install dependencies for building documentation.
+
+# Submodules
+You can import:
+
+- `mvtk.credibility` for assessing credibility from sample size.
+- `mvtk.interprenet` for building interpretable neural nets.
+- `mvtk.thresholding` for adaptive thresholding.
+- `mvtk.sobol` for Sobol sensitivity analysis
+- `mvtk.supervisor` for divergence anlysis
+
+# Documentation
+You can run `make -C docs html` on a Mac or `make.bat -C docs html` on a PC to just rebuild the docs. In this case, point your browser to ```docs/_build/html/index.html``` to view the homepage. If your browser was already pointing to documentation that you changed, you can refresh the page to see the changes.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/about.rst

@@ -0,0 +1,33 @@
+.. _about:
+
+About
+========
+
+History
+-------
+
+This project was started by Alex Eftimiades in 2019 as part of an
+internal R&D effort focused on model monitoring and sensitivity
+analysis. With early usage, testing, and utility contributions from
+Dwight Gunning, Matthew Gillett, and Mona Annaparthi, this lead to the ``supervisor``
+submodule and many of the initial ideas that became the
+``thresholding``, ``sobol``, and ``credibility`` modules. Subsequent
+work on explainability lead to ``interprenet`` and the normalized
+mutual information score within ``metrics``.  
+
+Authors
+-------
+
+The following people are currently core contributors to Model Validation
+Toolkit's development and maintenance:
+
+.. include:: authors.rst
+
+Please see :doc:`contributing <contributing>` to join us!
+
+Acknowledgements
+-------
+
+We thank David Devakumar, Mohamad Ibrahim, Jonathan Bryant, and Ahmed Ibrahim
+for their support, feedback, and help allocating resources to work on this
+project. We thank Nil Weerasinghe for his help organizing R&D efforts.

docs/authors.rst

@@ -0,0 +1,8 @@
+.. raw :: html
+
+    <table width="99%" class="container">
+    <tr><td class="alert" colspan="3"><font size="5"><b>Team</b></font></td></tr>
+    <tr><td class="alert" width="10%"><img src="https://avatars.githubusercontent.com/u/4635614?v=4" title="Alex Eftimiades"/></td><td class="alert" width="40%"><b>Alex Eftimiades - Lead</b><br/><a href="https://aeftimia.github.io" target="_new" >Website</a>&nbsp;&nbsp;&nbsp;&nbsp;<a href="https://www.linkedin.com/in/alex-eftimiades-1a4836bb"target="_new" >Linkedin</a></td>
+        <td class="alert" width="10%"></td><td class="alert" width="40%"><b>Matthew Gillett - Developer</b><br/><a href="https://github.com/matthewgillett" target="_new" >Website</a>&nbsp;&nbsp;&nbsp;&nbsp;<a href="https://www.linkedin.com/in/matthew-gillett-sdet"target="_new" >Linkedin</a></td></tr>
+    </table><table style="width:100%"><tr>
+    <br>

docs/conf.py

@@ -0,0 +1,89 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../"))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "Model Validation Toolkit"
+copyright = "2021, Model Validation Toolkit Team"
+author = "Model Validation Toolkit Team"
+
+# The full version, including alpha/beta/rc tags
+release = "0.0.1"
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinxcontrib.bibtex",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+    "nbsphinx",
+    "myst_parser",
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_rtd_theme"
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["css", "images"]
+html_css_files = ["custom.css"]
+html_logo = "images/logo.svg"
+html_theme_options = {
+    "display_version": False,
+}
+html_favicon = html_logo
+
+# A fix for Sphinx error contents.rst not found
+master_doc = "index"
+
+# increase the timeout for long running notebooks
+nbsphinx_timeout = 180
+
+# Don't show full paths
+add_module_names = False
+
+# bibtex
+bibtex_bibfiles = ["refs.bib"]
+
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".txt": "markdown",
+    ".md": "markdown",
+}

docs/contributing.md

@@ -0,0 +1,139 @@
+[comment]: <> (Adapted from JAX's contribution guide)
+
+# Contributing
+
+Everyone can contribute to Model Validation Toolkit, and we value everyone's contributions. There are several
+ways to contribute, including:
+
+
+- Answering questions on Model Validation Toolkit's [discussions page](https://github.com/FINRAOS/model-validation-toolkit/discussions)
+- Improving or expanding Model Validation Toolkit's [documentation](https://finraos.github.io/model-validation-toolkit/docs/html/index.html)
+- Contributing to Model Validation Toolkit's [code-base](https://github.com/FINRAOS/model-validation-toolkit/)
+
+## Ways to contribute
+
+We welcome pull requests, in particular for those issues marked with
+[contributions welcome](https://github.com/FINRAOS/model-validation-toolkit/issues?q=is%3Aopen+is%3Aissue+label%3A%22contributions+welcome%22) or
+[good first issue](https://github.com/FINRAOS/model-validation-toolkit/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).
+
+For other proposals, we ask that you first open a GitHub
+[Issue](https://github.com/FINRAOS/model-validation-toolkit/issues/new/choose) or
+[Discussion](https://github.com/FINRAOS/model-validation-toolkit/discussions)
+to seek feedback on your planned contribution.
+
+## Contributing code using pull requests
+
+We do all of our development using git, so basic knowledge is assumed.
+
+Follow these steps to contribute code:
+
+1. Fork the Model Validation Toolkit repository by clicking the **Fork** button on the
+   [repository page](https://www.github.com/FINRAOS/model-validation-toolkit). This creates
+   a copy of the Model Validation Toolkit repository in your own account.
+
+2. Install Python >=3.6 locally in order to run tests.
+
+3. `pip` installing your fork from source. This allows you to modify the code
+   and immediately test it out:
+
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/model-validation-toolkit
+   cd model-validation-toolkit
+   pip install -e .  # Installs Model Validation Toolkit from the current directory in editable mode.
+   ```
+
+4. Add the Model Validation Toolkit repo as an upstream remote, so you can use it to sync your
+   changes.
+
+   ```bash
+   git remote add upstream http://www.github.com/FINRAOS/model-validation-toolkit
+   ```
+
+5. Create a branch where you will develop from:
+
+   ```bash
+   git checkout -b name-of-change
+   ```
+
+   And implement your changes using your favorite editor.
+
+6. Make sure the tests pass by running the following command from the top of
+   the repository:
+
+   ```bash
+   pytest tests/
+   ```
+
+   If you know the specific test file that covers your changes, you can limit the tests to that; for example:
+
+   ```bash
+   pytest tests/supervisor
+   ```
+
+   Model Validation Toolkit also offers more fine-grained control over which particular tests are run;
+   see {ref}`running-tests` for more information.
+
+7. Once you are satisfied with your change, create a commit as follows ([how to write a commit message](https://chris.beams.io/posts/git-commit/)):
+
+   ```bash
+   git add file1.py file2.py ...
+   git commit -s -m "Your commit message"
+   ```
+
+   Please be sure to sign off your work when you commit it with the `-s` or, equivalently `--sign-off` flag to agree to our [DCO](https://raw.githubusercontent.com/FINRAOS/model-validation-toolkit/main/DCO.txt).
+
+   Then sync your code with the main repo:
+
+   ```bash
+   git fetch upstream
+   git rebase upstream/main
+   ```
+
+   Finally, push your commit on your development branch and create a remote 
+   branch in your fork that you can use to create a pull request from:
+
+   ```bash
+   git push --set-upstream origin name-of-change
+   ```
+
+8. Create a pull request from the Model Validation Toolkit repository and send it for review.
+   Check the {ref}`pr-checklist` for considerations when preparing your PR, and
+   consult [GitHub Help](https://help.github.com/articles/about-pull-requests/)
+   if you need more information on using pull requests.
+
+## Model Validation Toolkit pull request checklist
+
+As you prepare a Model Validation Toolkit pull request, here are a few things to keep in mind:
+
+### DCO
+
+By contributing to this project, you agree to our [DCO](https://raw.githubusercontent.com/FINRAOS/model-validation-toolkit/main/DCO.txt).
+
+### Single-change commits and pull requests
+
+A git commit ought to be a self-contained, single change with a descriptive
+message. This helps with review and with identifying or reverting changes if
+issues are uncovered later on.
+
+Pull requests typically comprise a single git commit. In preparing a pull
+request for review, you may need to squash together multiple commits. We ask
+that you do this prior to sending the PR for review if possible. The `git
+rebase -i` command might be useful to this end.
+
+### Linting and Type-checking
+
+Model Validation Toolkit uses [mypy](https://mypy.readthedocs.io/) and [flake8](https://flake8.pycqa.org/)
+to statically test code quality; the easiest way to run these checks locally is via
+the [pre-commit](https://pre-commit.com/) framework:
+
+```bash
+pip install pre-commit
+pre-commit run --all
+```
+
+### Full GitHub test suite
+
+Your PR will automatically be run through a full test suite on GitHub CI, which
+covers a range of Python versions, dependency versions, and configuration options.
+It's normal for these tests to turn up failures that you didn't catch locally; to
+fix the issues you can push new commits to your branch.

docs/credibility.rst

@@ -0,0 +1,8 @@
+credibility
+==================
+
+.. automodule:: mvtk.credibility
+    :members:
+    :imported-members:
+    :undoc-members:
+    :special-members: __init__, __call__