Support meta.yaml, pyproject.toml and versions.json

[vyasr]

In addition to conda requirements files, the rapid dependency generator needs to support a few additional use cases. In order of priority, these are:

• pyproject.toml: We need to be able to encode build-time dependencies into pyproject.toml files for Python packages. This will become much more important when we start officially supporting pip installation. At present most requirements information encoded in setup.py or pyproject.toml is essentially ignored by all our conda-only build processes, but that is going to change soon.
• meta.yaml: We need to be able to write out the appropriate sections of meta.yaml. Currently the generator will produce the env files that are helpful for developers or for use in different CI stages, but the conda package definition files are still manually maintained and could go out of sync.
• versions.json: We need to be able to write out versions to some sort of dependency file that can be handled in CMake. This necessity arises because there are some dependencies (treelite for cuML is the one that comes to mind) that are needed both at C++ compile time -- which means they are brought in via CPM.cmake -- and at Python run/test time -- which means they must be handled by setuptools. I propose a versions.json file because that is already essentially supported out of the box by rapids-cmake. I'll note that at this time treelite is the only obvious example of this that I see, so supporting this is not crucial. However, I think it would be minimal extra work on top of what we would already have to do to enable pyproject.toml, so I think it would be worth it. If we go this route, we should follow the rapids-cmake spec for versions.json.

Enabling the above will require making the following changes (in roughly this order based on which tasks depend on others as well as their relative priorities):

1. Generalizing the dependencies.yml file format. Currently, the file format hardcodes the conda_and_requirements key as the way to specify requirements that are common to both types of dependency files. Instead of this, we will need to modify the file to have something like a file_types key that accepts an array of values and writes those requirements to all the associated files. That would allow one set of entries to be shared across an arbitrary subset of different files.
2. Enabling read-modify-write behavior rather than a simple overwrite behavior in the generator. Currently the generator simply reads dependencies.yml, generates complete dependency lists, and then writes out those lists to the appropriate files, overwriting any preexisting files. While that is appropriate for requirements.txt and environment.yml files, that is not appropriate for pyproject.toml files or meta.yaml files. The latter types of files could contain additional information beyond a simple list of dependencies and that information must persist.
3. Implement writing of TOML files. The TOML package is the easiest choice here.
4. Write out the meta.yaml dependencies sections into new sub yml files, then use conda-build's jinja functions to read those files in for each section.
5. Implement writing of JSON files. We can just use the built-in json module.
6. Update rapids-cmake to support reading arbitrary JSON files. That should just require extracting some of the logic from rapids_cpm_load_preset_versions into a separate function.

Some additional notes and caveats:

• The version styles used by CPM.cmake do not support the sorts of complex version string used by setuptools/conda. Instead these version strings are hardcoded to a specific number. However, I do not think this is an issue because in cases where we hardcode a dependency for CMake we should force Python to use the same exact version (no fuzzy deps), so those dependencies can always be written as $PKG=$VERSION rather than $PKG>x,<y etc.
• CPM downloads require a git tag/branch. I'm not yet sure how we would want to support this. It may require adding some keys to the appropriate section. The best outcome here is likely that we will maintain separate sections for versions.json and Python requirements for a given dependency, and the versions.json section will support the additional keys like git tag/branch, but then we'll use a YAML anchor inside dependencies.yml containing the version so that there remains only a single source of truth.

[ajschmidt8]

The PR below was opened to adjust the dependencies.yaml schema in order to support future output types.

• Update dependencies.yaml Schema #11

[vyasr]

We now support writing out pyproject.toml files. I will open a separate thread to discuss issues with supporting meta.yaml files.