PoC for Ragna meta package

[pmeier]

This is an alternative to #405. With this we have a new folder meta-package that contains a pyproject.toml and a setup.py:

• The pyproject.toml is minimal version of the one we have in our project root. We should have a CI job that makes sure both are in sync.
• The setup.py does nothing but injects the optional dependencies and ragna_base as requirements for the meta package.

Here is how you do a development install

$ # install ragna-base
$ pip install -e .
$ # either install the meta package
$ pip install -e ./meta-package
$ # or just install the optional requirements
$ pip install -r requirements-optional.txt

To release you can do

$ # build ragna-base
$ python -m build
$ # build ragna
$ python -m build --sdist --wheel --outdir ./dist ./meta-package

I don't know why I explicitly need to set --sdist --wheel for the meta-package build. If I don't, the build errors. This seems to be a bug in the build package. Will look into it.

[arjxn-py]

Thanks @pmeier, I have tested this branch thoroughly building and installing wheels a number of times in fresh environments.
Installation and everything works all good.

However I noticed one thing that the version that is being checked in setup.py looks locally something like 0.3.0.dev31+g7743725.d20240704 & say we've published ragna-base==0.3.0 on pip. Would it still be going to give error saying ERROR: Could not find a version that satisfies the requirement ragna==0.3.0.dev31+g7743725.d20240704?

[arjxn-py]

I also noticed that currently, it is essential to build ragna-base before ragna & that's just because so that pip can locate ragna-base dependency.

Would we want to do the same on the CI workflow as well? (However more experimentation is yet to be done)

[arjxn-py]

I did some workaround on a branch on my fork with this one as the base.
And also opened a PR on the fork to run workflow.

You can also download & test build artifacts from here:

• ragna-base-dist
• ragna-dist

Currently, one cannot install ragna-dist directly without installing ragna-base-dist

[arjxn-py]

I think the failure here can be used as a breaking point which should prevent us from going ahead to publish the meta-package until & unless pip has the access to the dependent ragna-base version.

[pmeier]

However I noticed one thing that the version that is being checked in setup.py looks locally something like 0.3.0.dev31+g7743725.d20240704 & say we've published ragna-base==0.3.0 on pip. Would it still be going to give error saying ERROR: Could not find a version that satisfies the requirement ragna==0.3.0.dev31+g7743725.d20240704?

This has nothing to do with ragna-base being published. If you want to install ragna locally, and there is really no need for it, you'll first have to install ragna-base. For this to work, both packages have to be build on the same commit.

I also noticed that currently, it is essential to build ragna-base before ragna & that's just because so that pip can locate ragna-base dependency.

Would we want to do the same on the CI workflow as well? (However more experimentation is yet to be done)

We'll never install ragna in CI other than in one dedicated workflow that makes sure the wheel we build is correct.

I think the failure here can be used as a breaking point

This is just a failing docker build, because I haven't updated the Dockerfile yet. Let me do that today.

which should prevent us from going ahead to publish the meta-package until & unless pip has the access to the dependent ragna-base version.

Yes, of course. We'll only ever publish them together. The first release for both of them will be v0.3.0 assuming we can get this done by then.

[arjxn-py]

This has nothing to do with ragna-base being published. If you want to install ragna locally, and there is really no need for it, you'll first have to install ragna-base. For this to work, both packages have to be build on the same commit.

Sorry to ask again, I just want to confirm that when releasing & publishing wheels, For example v0.3.0, ragna would have ragna-base==0.3.0 as a dependency instead of something like ragna==0.3.0.dev31+g7743725.d20240704 as this extra version identifier string makes it hard for pip or conda to identify the package.

We'll never install ragna in CI other than in one dedicated workflow that makes sure the wheel we build is correct.

Right, that makes sense. But just to make a note again that installation of wheels will only pass either if ragna-base is already installed in the environment or that specific version of ragna-base could be found on Pypi. It's not an issue but a checkpoint type of a thing.

This is just a failing docker build, because I haven't updated the Dockerfile yet. Let me do that today.

😬 I reffered to the another linked failure here on my fork, this is the one I refer to as checkpoint.

[pmeier]

Sorry to ask again, I just want to confirm that when releasing & publishing wheels, For example v0.3.0, ragna would have ragna-base==0.3.0 as a dependency instead of something like ragna==0.3.0.dev31+g7743725.d20240704 as this extra version identifier string makes it hard for pip or conda to identify the package.

I see where the confusion is coming from now. We are using setuptools_scm as versioning tool.

ragna/pyproject.toml

Lines 1 to 6 in 9776ec5

	[build-system]
	requires = [
	"setuptools>=45",
	"setuptools_scm[toml]>=6.2",
	]
	build-backend = "setuptools.build_meta"

ragna/pyproject.toml

Lines 67 to 70 in 9776ec5

	[tool.setuptools_scm]
	write_to = "ragna/_version.py"
	version_scheme = "release-branch-semver"
	local_scheme = "node-and-timestamp"

It looks at the current state of git and produces a version number from it. So your string ragna==0.3.0.dev31+g7743725.d20240704 translates to

• 0.3.0: estimated next release
• .dev31: 31 commits since the last release
• +g: git used as SCM
• 7743725: first 7 characters of the commit hash
• .d: project was in a dirty state when it was built / installed, i.e. there were non-commited changes
• 20240704: project was built / installed on 4th of July, 2024

You can check the version it produces with python -m setuptools_scm. If you commit everything in your local copy, the .d... part will disappear. And finally, if you add a git tag, e.g. git tag v1.2.3, you'll see that the output is 1.2.3 (make sure to delete the tag again with git tag -d v1.2.3 to avoid future clashes).

Long story short: since we'll add a git tag before each release, all the extra version information that we have during development will be gone.

But just to make a note again that installation of wheels will only pass either if ragna-base is already installed in the environment or that specific version of ragna-base could be found on Pypi.

I'm aware, but again this is not an issue. For development, we'll never install ragna and for published packages, the corresponding ragna-base will be available.

It's not an issue but a checkpoint type of a thing.

I don't understand what you mean by "checkpoint".

[arjxn-py]

Long story short: since we'll add a git tag before each release, all the extra version information that we have during development will be gone.

That makes sense, thanks a lot on confirming this.

I don't understand what you mean by "checkpoint".

I mean just a checkpoint to see corresponding ragna-base availability. Please see this error.

This looks promising, Please do let me know if I can assist you with anything related 🚀

[pmeier]

I don't know why I explicitly need to set --sdist --wheel for the meta-package build. If I don't, the build errors. This seems to be a bug in the build package. Will look into it.

The failure means that the wheel cannot be build by the sdist, i.e. the sdist is broken. This needs to be fixed before moving forward.

[pmeier]

@arjxn-py If you are still eager to help with this, could you check why we cannot build a wheel from the sdist on this branch, i.e. find out what is broken with the sdist? From the error we get when running python -m build --output-dir ./dist ./meta-package is seems the pyproject.toml file is missing. We might just need to add a MANIFEST.in file and link the pyproject.toml there.