✨♻️‼️ Rewrite myst-nb

[chrisjsewell]

This PR encompasses a top-to-bottom re-write of myst-nb!

It is mainly back compatible, for general users of the extension, with some more technical breaking changes listed below.

---

Highlights

• docutils API/CLI with command line tools, e.g. mystnb-docutils-html
  • Made possible by:
    1. re-working notebook output rendering to be within the initial parse (rather than a post-transform)
    2. moving to use of "native" docutils nodes with attributes
  • Including glue role/directives!
• Multi-level configuration (conf.py < notebook level metadata < cell level metadata)
  • Configuration is defined in single class, which is used to generate both docutils and sphinx configuration
  • New config options including: nb_number_source_lines, nb_remove_code_source, nb_remove_code_outputs, nb_render_error_lexer
  • See auto-generated config documentation: https://myst-nb--380.org.readthedocs.build/en/380/use/config-reference.html#auto-generated-config
• Parallel friendly (e.g. sphinx-build -j 4 can execute four notebooks in parallel)
• Page specific loading of ipywidgets JS (sphinx >= 3.5)
• Added raw cell rendering
• Added MIME render plugins
• Better log info/warnings, with type.subtype to suppress
• Reworked jupyter-cache integration to be easier to use (including parallel execution)
• New glue:md role/directive includes nested parsing of MyST Markdown (see https://myst-nb--380.org.readthedocs.build/en/380/use/glue.html#the-glue-md-role-directive)
• Removes dependencies: jupyter-sphinx, ipywidgets and nbconvert

---

New API Logic

The API is much clearer/cleaner (see also https://myst-nb--380.org.readthedocs.build/en/380/api/index.html)

The parsing of a notebook consists of a number of stages, with each stage separated into a separate module:

1. myst_nb.configuration: The configuration is set (from a file or CLI)
2. myst_nb.docutils_ / myst_nb.sphinx_: The Parser is called with an input string and source
3. myst_nb.read: The parser reads the input string to a notebook node (possibly using a custom format reader)
4. myst_nb.execute: The notebook code outputs are potentially updated, via execution or from a cache
5. myst_nb.preprocess: The notebook is "pre-processed" in-place (e.g. to coalesce output streams and extract glue outputs) (this could eventually be pluggable, akin to nbconvert.preprocessors)
6. myst_nb.parse: The notebook is converted to a Markdown-It tokens syntax tree
7. myst_nb.render: The syntax tree is transformed to a docutils document AST (calling the renderer plugin)
8. The docutils document is processed by docutils/sphinx, to create the desired output format(s)

---

Breaking changes

• A number of configuration option names have been changes, in order for all of them to share the nb_ prefix.
  • All old names are currently still allowed, and their conversion to the new name automated, with a warning emitted, except...
  • nb_render_priority config is replaced by nb_mime_priority_overrides, which have different format/semantics
• The AST structure, for code cells etc, is completely changed.
• ipywidgets needs to be specifically installed, if required for use
• The code cell output render is completely re-written, meaning existing render plugins will no longer work.
• glue directives/roles are more restricted on how they work cross-document (only working on the same page by default).
  • glue:any directive allows for a doc option, and glue:any/glue:text allow the (relative) doc path to be added before a ::.

[chrisjsewell]

Ideally, something like the class might grow the ability to register new render functions and the main method loops over the registered functions to render outputs would make for easy extensibility.

@bryanwweber see https://myst-nb--380.org.readthedocs.build/en/380/use/formatting_outputs.html#customise-the-render-process

[chrisjsewell]

glue is now available cross-document, for a small subset of roles/directives/mime types: https://myst-nb--380.org.readthedocs.build/en/380/use/glue.html#pasting-from-other-notebooks

[namurphy]

@chrisjsewell — I'm excited that this was merged! As someone following this project, thank you for doing this!

[chrisjsewell]

Cheers @namurphy!