Build a CI tool to test jupyterlite notebooks

[ArturoAmorQ]

Related to #10. Of interest for projects implementing interactive documentation.

Back in the day @lesteve created this python file to test if the scikit-learn doc's JupyterLite is functional. It basically keyboard.press("Shift+Enter") across the notebook and waits. Not sure if it would make sense to convert it into an automated CI tool to look for (unexpected) warnings/errors being raised.

[agriyakhetarpal]

From yesterday: I previously explored this topic, but for doctests and never spent too much time on it. My idea is to port a pytest plugin, such as nbmake, nbval, or any similar ones, and run them in a pyodide venv pointing to a set of notebooks – perhaps it would work out of the box.

[lesteve]

Yeah the pyodide venv seems like potentially a simpler direction, this doesn't really test that the JupyterLite environment is functional but maybe is good enough.

[agriyakhetarpal]

If we talk about testing the JupyterLite environment with the Pyodide kernel, that is best done using a bunch of test notebooks, Jest, and Playwright in a .js script. It will not be complicated to set up, but it will take a lot more CI time to run in comparison. I would suggest that we choose a subset of the notebook from the gallery, because we might want to parametrise across browsers (Chrome and Firefox) as well (considering the platform-specific intricacies we've noticed in JupyterLite).

If there is further interest in this direction, I'd be happy to hack something together!

[lesteve]

We could also stay in pure Python with pytest-playwright as in https://github.com/lesteve/test-scikit-learn-jupyterlite, which is a PoC to test that the scikit-learn.org JupyterLite environment is functional.

To be honest I think running the notebooks inside a pyodide venv may be simpler and good enough.

[jtpio]

For reference jupyter/try-jupyter#73 added some UI tests using pytest-playwright, mostly to double check notebooks can be executed fine before the new JupyterLite site is made available to users of https://jupyter.org/try.

Detecting notebook cell execution errors can be trickier than it sounds, so these tests rely on some heuristics for now.

Sharing here in case it can be useful to this repo too.

[mcbarton]

I have developed a method using Selenium here compiler-research/xeus-cpp#415 that has been tested with the xeus-cpp kernel. It works for Firefox, Chrome and Safari.

It works by running the notebook from a Jupyter Lite website which is being served in the ci, downloading the finished notebook at the end, and then comparing this downloaded notebook to a reference notebook using nbdiff (in our case the reference notebook is a fully executed notebook in xeus-cpps notebooks folder). With the way it currently stands executing a notebook automatically is done by executing (there is also a --run-browser-gui option that exists if you want Chrome or Firefox to show a gui browser as they run locally)

python -u scripts/automated-notebook-run-script.py --driver browser --notebook notebook --kernel kernel --stdin Test_Input --timeout timeout

The script assumes you have a jupyter lite site being served in the background, as can be seen from the ci. Kernel is an option since we have multiple C++ kernels, and want to check all of them work for a particular notebook.