How can I use conda environments to build and install dependencies?

[ianhbell]

A key feature of ReadTheDocs is the ability to specify conda/mamba environment to build the docs from. Also, I need to be able to re-compile python extensions as part of the build on the development branch. That works flawlessly on RTD, and I don't see a way to do that in the current implementation. For me that is a dealbreaker.

[guyer]

[Apologies, I wasn't "watching" this repo, so missed your question last week]

I have implemented an experimental branch that lets you specify an environment.yml file. For now, to use it, put uses: usnistgov/NISTtheDocs2Death@conda in your workflow. It's all terribly slow right now, because it keeps reinstalling at every step of the overall action; figuring out the GitHub Container Registry is already on my todo list.

I've been testing with a clone of teqp. Right now, it's stalled at doxygen Doxyfile, evidently because the path isn't what it expects. My script executes conf.py in the context of its own directory because that's what Sphinx does. I'm not sure why conf.py would work for you in RtD but not with my code; I took that part directly from Sphinx.

Based on your use of READTHEDOCS, it looks like you'll want an NTD2D environment variable, or similar. I can provide that.

I need to be able to re-compile python extensions as part of the build on the development branch

Can you point me to an example of this? I'm not seeing it in teqp.

[ianhbell]

Sorry I guess I was thinking about another project where building the extension was part of the doc build. I plan to add that to teqp at some point since currently I can't update the docs until I push wheels to pypi.

On RTD you can also add apt packages, which is how I get doxygen into the RTD instance. There appears to also be a doxygen package for conda: https://anaconda.org/conda-forge/doxygen

[ianhbell]

How can I debug this on my side?

[guyer]

Not sure you need to. I think I figured out something on the plane, but not sure when I'll have a chance to test.

[guyer]

I was able to sort out the path issue. (Unsurprisingly) I was not doing exactly what Sphinx does.

[ianhbell]

Makes sense, paths and working directories are always tricky in these sorts of problems

[guyer]

There are still some gross inefficiencies on my end to sort out (re-re-installing things, mostly; those should be resolved once I understand the Container Registry). Beyond that, my code does not expect importing conf.py to be expensive; as such, it loads it a couple of different times in different sub-actions. I can resolve that for teqp by moving doxygen Doxyfile, jupyter nbconvert, and sphinx-apidoc into the pre-build-command of my action, but I don't know if I'm missing something in your workflow.

[guyer]

I've enabled conda and apt installations, and I refactored to get the runtime down to something semi-reasonable.
Things still aren't error free, but you can see what I've got at usnistgov/teqp#48.

[ianhbell]

Where do the docs appear?

[guyer]

When the action is running and, if you have appropriate permissions, they'll show up at https://pages.nist.gov/teqp. When the action runs from my fork, it doesn't have permission to push to your nist-pages branch (once you have one).

[ianhbell]

How can I set up a redirect from the existing docs to the new docs? One last hacked set of docs via RTD?

[guyer]

Kamal Choudhary provided me with a bit of JavaScript that can be added to your reST files.

.. raw:: html

    <script type="text/javascript">
    if (String(window.location).indexOf("readthedocs") !== -1) {
        window.alert('The documentation has moved. I will redirect you to the new location.');
        window.location.replace('https://pages.nist.gov/jarvis/');
    }
    </script>

I think Kamal put this in all of his reST files, but it seems simpler to me to just create a branch with stub documentation that only serves this JS and then disable all other branches at RtD.

[ianhbell]

Yes a branch with the redirect is probably best, build from that branch

[ianhbell]

There is something wrong because some notebooks didn't get run with the right teqp version, should be 0.16.0. A handful used older versions, I have no idea how that is possible.

[guyer]

It may be getting the old teqp from your environment.yml file, which includes teqp in the pip list?

If your package was pure python, you can add

import pathlib

sys.path.insert(0, pathlib.Path(__file__).parents[2].resolve().as_posix())

to your conf.py, but I don't think that'll work here. Does RtD build and install the package?

[ianhbell]

RtD just installs wheels, which should be fine since there are manylinux wheels, and those get pulled in for the doc build, which seems to be happening because some notebooks use 0.16, and others don't

[ianhbell]

It seems to be because there was already output in the notebook, and the nbconvert skipped them as a result. Is there a force option?

[guyer]

Ah, interesting. No force option, but it's possible I didn't get the logic right when I migrated Python in the conf.py to bash in the workflow.

[ianhbell]

Can you please set up your workflow to be a lot more verbose? It is impossible to debug with no output about what packages it installed and so on

[guyer]

Figuring out why there's so little output from the conda install is on my to-do list for this afternoon.
It won't help here, but, in general, you can get stupid amounts of output if you enable debug logging.

[guyer]

conda env update doesn't report what it did (and --verbose is so noisy that it's useless). I now follow with conda list.

[guyer]

I've encountered a design issue. One of the first steps I take is to copy the documentation configuration directory before modifying it. I do this because if, e.g., you use versioneer, modifying docs/ directly will cause the version to show as .dirty. This has worked fine for me, but I noticed that once I got teqp working at all with this action, while it ran doxygen, the doxygen output doesn't get linked in the C++ interface section.

I realize now that this is because your Doxygen file says OUTPUT_DIRECTORY = doc/source/_static/doxygen, which is reasonable enough, but it's not where my docs-BORGED/ configuration is looking for things.

I don't know offhand how to reconcile this.

[ianhbell]

Hmm, I don't either. That's a little bit less important to have working, but still important from a documentation standpoint.