Cell error traceback incorrectly lexed as IPython

[clarksmr]

Describe the bug

I'm using Jupyter Book to produce a book with executable OCaml cells, including demoing some OCaml code that deliberately does not compile. I tag those cells with raises-exception of course. Some (not all) OCaml compiler error messages cause Sphinx to produce a lexer warning, Could not lex literal_block as "ipythontb". I think but am not sure this is because MyST-NB is assuming that the traceback field from any error is lexable as a Python error message. I conjecture that the problem is this line of code in MyST-NB.

To Reproduce

Steps to reproduce the behavior:

1. Clone the MWE I put up at https://github.com/clarksmr/jupyter-book-ocaml-lex-warning
2. Assuming you have an OCaml installation and a Jupyter OCaml kernel (big assumption), run jupyter-book build docs. I'm happy to provide more instructions about how to do that install.

Expected behavior

I expect the notebook's error output not to be lexed as Python, hence for Jupyter Book not to produce a warning.

Actual behavior

The actual output is:

❯ jupyter-book build docs
Running Jupyter-Book v0.11.1
Source Folder: /Users/clarkson/repos/cs3110/jupyter-book-ocaml-lex-warning/docs
Config Path: /Users/clarkson/repos/cs3110/jupyter-book-ocaml-lex-warning/docs/_config.yml
Output Path: /Users/clarkson/repos/cs3110/jupyter-book-ocaml-lex-warning/docs/_build/html
Running Sphinx v3.5.4
[etoc] Changing master_doc to 'MWE'
loading pickled environment... done
myst v0.13.7: MdParserConfig(renderer='sphinx', commonmark_only=False, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, update_mathjax=True, enable_extensions=['colon_fence', 'dollarmath', 'linkify', 'substitution'], disable_syntax=[], url_schemes=['mailto', 'http', 'https'], heading_anchors=None, html_meta=[], footnote_transition=True, substitutions=[], sub_delimiters=['{', '}'])
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
Executing: MWE in: /Users/clarkson/repos/cs3110/jupyter-book-ocaml-lex-warning/docs
[NOTE] To make opam select the switch textbook in the current shell, add --set-switch or set OPAMSWITCH
2021-07-03T18:32:36  OCaml kernel main loop is exited.

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] MWE                                               
/Users/clarkson/repos/cs3110/jupyter-book-ocaml-lex-warning/docs/MWE.ipynb:30002: WARNING: Could not lex literal_block as "ipythontb". Highlighting skipped.
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.

Environment

• Python Version [e.g. 3.7.1]: 3.9.5
• Package versions or output of jupyter-book --version:

❯ jupyter-book --version
Jupyter Book      : 0.11.1
External ToC      : 0.2.2
MyST-Parser       : 0.13.7
MyST-NB           : 0.12.3
Sphinx Book Theme : 0.1.0
Jupyter-Cache     : 0.4.2
NbClient          : 0.5.3

• Operating System: MacOS v11.4

Additional context

I conjecture the problem is this line of code in MyST-NB. It installs the IPythonTracebackLexer for the traceback field of an error output. But that is making the (to me) incorrect assumption that all error outputs are Python, rather than some other language.

Alternative explanations could include:

• The OCaml Jupyter kernel is somehow producing bad output for some error cells but not others. But if so I'm not spotting it. Even without an OCaml kernel you can inspect the raw output in my MWE.
• The actual requirement of MyST-NB is that all code is Python, and I've misunderstood that. :) In which case I guess this would be a feature enhancement request.

P.S. I am migrating my OCaml textbook from GitBook 2 to Jupyter Book and am so happy with Jupyter Book! Around 700 students/year at my university use this book, and others do elsewhere too. So I'm really grateful for the excellent upgrades that Jupyter Book is going to bring to us this fall when classes resume. Thank you so much for all your work!

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[choldgraf]

I think I agree with you about the source of the bug! Maybe there is some way to check the traceback text to see if it is Python code or not, and if not then use a generic literal block instead of the ipython one?

[akhmerov]

It seems that:

• CellOutputRenderer has access to the document node.
• IIRC myst-nb keeps kernel info in the document metadata.

That suggests a possible approach, where the renderer would read the kernel metadata from the document, and apply the lexer depending on the kernel info. Inferring language from the traceback seems hard...

[clarksmr]

The error "traceback" (if it is that, depending on the language) format is dependent on the kernel, not the code block -- since there is only one kernel per notebook. So @akhmerov seems on track suggesting it be based on the kernel metadata.

[chrisjsewell]

Since #380, you can now set e.g. nb_render_error_lexer = "ipythontb" in your conf.py, for a global setting,
or add to a notebook metadata e.g.

{"mystnb": {"render_error_lexer": "ipythontb"}}

for a per-document setting

[chrisjsewell]

Note, to turn off any lexing it would be nb_render_error_lexer = "none"

I'm going to change this to a documentation issue, but let me know if this still does not fix things (obviously once #380 is released and integrated in jupyter-book)

I guess one could debate whether the default should be "ipythontb" or "none", but I think still the majority of users are using python kernels

[clarksmr]

Should this be working in Jupyter Book at this point? I'm running with v0.13.0. I tried adding that notebook metadata but the python lexer is still being used:

/Users/clarkson/repos/cs3110/textbook/src/chapters/basics/expressions.md:570: WARNING: Could not lex literal_block as "ipythontb". Highlighting skipped.

Here's how I added the metadata:

---
mystnb:
  render_error_lexer: none
---

[chrisjsewell]

Should this be working in Jupyter Book

No, jupyter book is currently not using the latest version of myst-nb

[chrisjsewell]

but if its ok, I will close this here as fixed

[clarksmr]

I am unable to test this end-to-end because Jupyter Book is not using the latest version of MyST-NB. So I will open a corresponding issue in Jupyter Book.