Should we switch over to using pixi?

[jhkennedy]

Right now, we use:

1. hatchling + hatchling vcs for a build backend
2. conda/mamba or venv for environment management
3. uv for providing a locked environment
4. nox for task running

Pixi cloud consolidates all of those into one tool, and everything could be managed through the normal pyproject.toml, although we don't have to adopt it for all those parts (e.g., sticking with Hatchling as the build back end but using Pixi for environments and tasks).

Some of the earthaccess developers are already using pixi (e.g., @mfisher87 : #821 (comment) and me for projects that have annoying build/compilation steps: ASFHyP3/hyp3-autorift#361), and I'm seeing it more and more recommended around the community (e.g., conda-forge rattler build support, ICESat-2 Hackweek presentation, UR-SSI open science summer school).

Since we've (@betolink @chuckwondo @andypbarrett @jhkennedy @JessicaS11) been kicking this around a little at the ICESat-2 Hackweek, and it will likely come up in the future, I thought it'd be worth starting a discussion to record our thoughts and have a place to refer back to.

So, should we consider switching to pixi? And for what parts?

[mfisher87]

Hell yeah pixi! Though what does it mean for earthaccess to adopt it? I think we should try and make the repository conda-free (except the user-facing install instructions) and treat conda concerns as downstream. I know not everyone agrees with this. Currently our conda instructions for developers (and our environment.yml) are just using conda as a wrapper for pip, and I'm not sure what that would look like with pixi.

[jhkennedy]

what does it mean for earthaccess to adopt it?

That's what I'm hoping we'll define here 🤣

I think we should try and make the repository conda-free

🤔 can you expand on what you mean by this? Are you saying you don't want to even pull packages from conda-forge?

When we switched from poetry, we were explicitly trying to switch away from an opinionated tool that didn't work well with common practices in the community (dependency spec. and env. management, primarly), but we also didn't want to have to maintain two lists of dependencies (pyproject.toml and environment.yml). So we settled on following the PyPA's standards using only the pyproject.toml to list dependencies, being PyPI-first (only) for dependency packages (builds), and leaving a wrapper environment.yml file in the repo. Developers can use venv or conda and effectively get the same environment.

I generally agree with that sentiment still, so I'm not sure I'm in a rush to adopt an opinionated tool again. But, things in that respect are better with pixi -- we can stick with hatchling as the build backend and write a pyproject.toml in a way that it'd be pip-installable and you could use venv, the conda wrapper, pixi, etc. So, pixi doesn't preclude that pattern still.

But if we wanted to be conda-forge-first in terms of where we pull dependency packages from, we'll either need to maintain 2 lists of dependencies (we chose not to do this before), or be more opinionated on the developer environment management. As we start adding in more access functionality and supporting tools like xarray, obstore, fsspec, and anything geo, we'll likely start encountering more and more places where conda-forge packages are an easier path than using what's available on PyPI (e.g., if we ever depend on anything that pulls in GDAL) .

I would be much more comfortable being opinionated with pixi as it (so far) has been really good at working with PyPA specifications and extending on community tools and practices instead of doing its own thing, and has much better default practices around dependency specifications. I think overall it would simplify the developer workflows, tooling, and our documentation.

[mfisher87]

I think we should try and make the repository conda-free

🤔 can you expand on what you mean by this? Are you saying you don't want to even pull packages from conda-forge?

Yeah, at least not as part of this repository. Since this package has no dependencies on hard-to-install libraries like GDAL (and I hope it never does), depending on conda-forge for development is more stuff to maintain and a more complex mental model. I think we should instruct users to install earthaccess from conda-forge if they want, but that's as far as I think we should go.

I don't feel strongly that we should be absolutists about this 0-conda approach though... really I think our environment.yml wrapper which enables folks to use conda for their virtual environment (but PyPI for dependencies) is fine. It meets some contributors' workflow needs, and that's ultimately a good thing. It's just a small amount of additional complexity and maintenance. When I think about adding pixi to the mix, as much as I love it, I have similar concerns about what we're enabling, and having more than "one correct way" to do things that we have to document. I don't really think of pixi as a tool I'd expect to use as part of developing a pure python project, more as a tool I'd leverage as a user.

[jhkennedy]

That all makes sense to me, but I think I come at it from a slightly different angle.

On the development side and the tools selected for it, it doesn't really matter to me if the project is pure Python or has compiled extensions, and I don't personally see any real fundamental difference between getting packages from PyPI or conda-forge, unless packages aren't available on one or the other.

Earthacess is used as part of traditional applications (more likely to leverage PyPI) and geospatial workflows (more likely to leverage conda-forge). And I expect our contributors/developers to both normally use Python tools and normally use conda-based tools. So I think it's important to keep a light-weight package available on PyPI (and keep any heavy dependencies optional), but I also expect it to be used heavily in conda environments and interact with the conda-forge ecosystem.

So I don't really see a reason to limit it to one or the other (and therefore limit the potential tools we could use) for development. But, I also don't have a compelling reason to use one or the other! 😉

When I think about adding pixi to the mix, as much as I love it, I have similar concerns about what we're enabling, and having more than "one correct way" to do things that we have to document.

I think the question here might be more, not imposing on contributors/developers does add more work on the documentation side, and having 4 different tools (hatchling, some environment manager, uv, and nox) could also be harder than one. Now that it's been over a year since we dropped Poetry (which mostly did give us one tool), has it been challenging/frustrating to use/support these different tools? And if so, would it be better to consolidate on a (better) single tool?

Overall, in my experience, it's been fine, and I don't see a need to switch to pixi, but I don't have a strong reason not to use pixi like I had with Poetry, so figured it's worth collecting thoughts here.

[jhkennedy]

And so sorry for including so many "so"s in my response. 🤣 🆘