Rethink dependency configuration in pyproject.toml

[mfisher87]

I have a few things I'd like to address, and maybe they should each be separate issues, but figured we should start discussion here :)

• Make pyproject.toml use PEP621 conventions to specify dependencies #617
• Reconsider dependency groups #619
• Remove duplicate dependency lists in conda environment YAML files #618

[jhkennedy]

Poetry-specific configuration: e.g. [tools.poetry.dependencies] instead of a dependencies key under [project]. I think we should use PEP621-compliant configuration here regardless of whether we use Poetry or not.

+1,000,000. I'd prefer everything in the pyproject.toml to be PEP-621 compliant regardless of the build/env tools we use.

IMO, we should have a dev group which includes only dev tools like pre-commit and Nox which are capable of managing their own environments. Have two more groups: docs for mkdocs and friends. test for pytest, mypy, and friends. Use nox to handle running tasks like building docs, typechecking, and testing.

I want to be able to just run tests directly in the dev environment as this makes leveraging IDE debuggers and tooling much easier (hooking up nox to IDEs is a bunch of extra unnecessary work), so, I'd generally prefer to keep the test dependencies in the dev group as well. I feel similar but less strongly for docs.

None of that precludes using nox -- it provides a nice interface for running "sessions" which can encompass testing, linting, doc builds, etc. That way, you get one interface to learn, which you can use for everything. In that vein, I'd want a pre-commit session as well, or just to use nox for all the stuff we do with pre-commit.

I believe someone mentioned that the mindeps environment can be dynamically calculated as well.

uv and pixi can now dynamically calculate this, so we'd need to use those tools and setup our dependencies compliant with that. They can be used in conjunction with nox though.

Or, we can add some custom code to parameterize all of our dependencies at whatever granularity; here's an example implimentation:
cjolowicz/nox-poetry#289 (comment)
(Note: this does use Poetry to parse dependencies but could be easily adapted to just use tomllib/tomlkit. It also depends on an API call to PyPI)

Remove conda dependency files. I don't think this pure python project has a reason to include them. We can advise conda users to use conda to create the environment, but use pip to install into it. It's a hassle to manage duplicated dependency lists.

Depending on what you mean here, I don't necessarily agree with this one, but reducing dependency duplication would be nice. I'd very much prefer a top-level environment.yml file that sets a development env up for you, but that likely could just look like:

name: earthaccess
channels:
  - conda-forge
  - nodefaults
dependencies:
  - python>=3.9
  - pip
  - pip:
      - --editable .[dev]

[mfisher87]

I want to be able to just run tests directly in the dev environment as this makes leveraging IDE debuggers and tooling much easier (hooking up nox to IDEs is a bunch of extra unnecessary work), so, I'd generally prefer to keep the test dependencies in the dev group as well. I feel similar but less strongly for docs.

Would pip install --editable .[dev,test,docs] meet this use case while still allowing a simpler dev environment for those that don't use IDE integrations?

[jhkennedy]

Would pip install --editable .[dev,test,docs] meet this use case while still allowing a simpler dev environment for those that don't use IDE integrations?

Yeah, that's fine so long as we test the matrix of those environments, and at least [dev,test] ends up in the environment.yml (if we include it).

Another thing I'd add: we should also test managing our dependencies with conda-forge seperate from pip, which we can do with nox. Pandas, for example, has different dependency constraints across pip and conda, for the same release version 🤦.

[mfisher87]

Depending on what you mean here, I don't necessarily agree with this one, but reducing dependency duplication would be nice. I'd very much prefer a top-level environment.yml file that sets a development env up for you, but that likely could just look like:

name: earthaccess
channels:
  - conda-forge
  - nodefaults
dependencies:
  - python>=3.9
  - pip
  - pip:
      - --editable .[dev]

I could go with this. Although under the hood it's doing the same thing :) On one hand, meets people where they are; on the other hand, misses the opportunity to skill folks up in a more standard tool to use for a pure python project. 🤷 I really just care about the duplication ;)

[mfisher87]

I created new issues for each of these things :)

[itcarroll]

Perhaps you are already aware, but Poetry is not going to be PEP 621 compliant (i.e. use projects table) until 2.0 is released.

[mfisher87]

😲 I was certain it was compatible with standard PEP621 fields. I haven't tried of course 😆

Maybe it's a good time to switch to setuptools or hatchling for build? 😁

[jhkennedy]

On one hand, meets people where they are; on the other hand, misses the opportunity to skill folks up in a more standard tool to use for a pure python project

@mfisher87, I'm going to come down hard on this because, frankly, I find that an insulting assertion. Using conda for environment management is not a skill issue and has very little to do with how a package is built and dependencies are specified. Conda is the de facto standard for the Jupyter ecosystem, especially JupyterHubs, and in Earth Observation spaces, which this package is directly targeted at, conda/mamba is often the primary way to manage environments because EO tools frequently depend on non-python libraries that can be anywhere from annoying to install via pip to completely unavailable.

[mfisher87]

@jhkennedy I apologize that you heard an insult in my comment; I absolutely did not intend to imply that people who use conda lack skill. I use conda & mamba almost every day myself. I'm trying to say that when we wrap the thing we're doing in conda-language, we're obfuscating what we're doing and missing an opportunity to teach what we're doing.