Version compatibility check for optional dependencies

[toadjaune]

Hi folks,

I recently ran into an issue with the rich integration, for traceback formatting.

With structlog version 23.2.0, and rich version 13.0.1, receiving a traceback results in a crash during error handling, as follows :

[...]
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/lib64/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "<redacted>/.venv/lib/python3.10/site-packages/structlog/stdlib.py", line 1062, in format
    ed = p(logger, meth_name, ed)
  File "<redacted>/.venv/lib/python3.10/site-packages/structlog/dev.py", line 505, in __call__
    self._exception_formatter(sio, exc_info)
  File "<redacted>/.venv/lib/python3.10/site-packages/structlog/dev.py", line 218, in __call__
    Traceback.from_exception(
TypeError: Traceback.from_exception() got an unexpected keyword argument 'locals_hide_dunder'
[...]

Upgrading rich to version 13.7.0 solves the issue, stacktraces now appear normally.

I undersand that rich is entirely optional to structlog, but I was wondering if maybe we could add some kind of protection against using an incompatible version of it. On top of my head :

• specifying version constraints. I'm unsure if the dependency constraint system permits to express "This is an optional dependency, so, please don't install it automatically with me ; but if it's present, ensure it validates those constraints, because I'm gonna use it". If that's possible, it would, in my opinion, be an ideal solution
• checking, at some initialization step (I assume we're making some kind of presence check for rich during structlog init), that the installed version is indeed recent enough, and don't attempt to use it if not. Maybe log something indicating the minimum required version
• checking version when attempting to use it. This sounds expensive at runtime though, so, probably not a good idea.
• some heuristic checks might be possible too, like, checking the function signature before attempting to call it, for non-existing arguments, etc. This would be pretty brittle, could not deal with all kinds of incompatibilities or bugs, etc ; so, probably not great either.

Also, similar checks could make sense for other optional dependencies (like better-exceptions)

What do you think ? Does that seem reasonable ?

[hynek]

In my experience, trying to get – even marginally – into Python packaging only results in blood and tears. I'm open to documenting the minimal versions (and testing against them), but doing runtime checks will always backfire – the question is only when.

[toadjaune]

Documenting and testing sound like a nice first step indeed !

I don't have any experience on the packager side of the python ecosystem, so, I'll take your word for it ; however, do you think such backfires could be worse than not doing any kind of validation and exploding ?

[hynek]

The problem with Python packaging as a big picture is that it's very much in flux. In the past few years there was like 5 ways to save/check package versions (see for example my post from early 2020: https://hynek.me/articles/packaging-metadata/) and even if we started using the latest now, there's the risk of nonstandard setups that somehow work patch something here, or insert modules there and suddenly importlib.metadata.version crashes instead of returning the version. Or returns the wrong version. Or…

[toadjaune]

That's unfortunate but in line with my undersanding of the ecosystem state.

I was hoping I had missed something here :/