proposal: Move API code to a separate module/repo

[bwplotka]

Proposal

To reduce the damage radius of release (e.g. compatibility issues), increase dev velocity and reduce indirect/direct dependencies for downstream users, we could consider splitting client_golang into multiple modules.

Specifically I would move to new modules:

• All collectors https://github.com/prometheus/client_golang/tree/main/prometheus/collectors (or each collector to its own?)
• testutil, for testutil: Better registry state assertion framework/API #1639 needs.
• http Prometheus client, where we could experiment with OpenAPI spec generated code as well.

We could split more, but other stuff tended to be less problematic in past and moving slower, or not have new deps.

Risks

Personally, the main pain-point of using multi-module SDKs (e.g. otel-go) were:

• Discovery of modules. When you need a piece of code to have e.g. processor/collector working, if it's part of a single Go module, IDE auto-completion will help you a lot figuring the required type or method. This does not work across modules, and it's really painful to manually walk through docs or examples to figure this out. Counter argument: In this proposal we do have less "connections" between modules (only one core module?), plus proposed module split has different functional purpose?
• Discovery of releases and dependencies across them. It's sometimes frustrating to have 10 new go.mod deps for core functionality. It's even more frustrating if some update of e.g. collectors/processors module require anyway update of core module (little point in separate module?). Counter argument: In our case I don't think we want to change main core module out of v1 or significantly, so perhaps that connection will be more rare?
• Releasing tooling is painful? You need to create those weird releases with prefixes, but maybe community already got used to this?

Would be curious of @dashpole opinion if he would advise this (:

[dashpole]

For packages that are already v1, i'm not sure there value other than maybe having fewer dependencies. For otel, we mostly have lots of modules because:

• different pieces of the library went stable at different points in time. Separate modules allowed us to stabilize small pieces.
• We have mandatory (per spec) separation between the API and SDK, which essentially requires multiple modules.
• We have a lot of plugins (e.g. exporters, bridges, detectors) with very different sets of dependencies.

Looking at prometheus/collectors, splitting it out doesn't remove any dependencies from the core module, so probably no value there.

A library with Prom instrumentation is probably going to use testutil as well, so probably not much value splitting that out, since everyone will need to import both.

There might be value in splitting out the http Prometheus client.

OTel-go uses go.opentelemetry.io/build-tools/multimod to manage versions.

[bwplotka]

Thanks!

Looking at prometheus/collectors, splitting it out doesn't remove any dependencies from the core module, so probably no value there.

The thing is -- it's only because we forced it so, we vendor some stuff and made some trade-offs to reduce num of deps. I wished we could add some simple deps (e.g. for MacOS process things). On the other hand, everyone uses process collector, so they will need to consume this anyway... and would enjoy less deps 🙃

Generally another aspect why it's useful to split those - is to version them differently. So e.g. creating v2 or v10 of collectors, testutils or http API is bit more manageable for users, doing same for core module is painful (arguably too big adoption/module to bump major version every year with little change e.g. see what happens with yaml v3 🙈 ) and there is smaller pressure. For example doing #1639 in some testutil.v2 feels much easier to consume; no need to make whole client_golang v2 only two avoid 3 testutil methods being semi-broken.

Are we too cautious with v2? (it has this path rename issue users might be not aware of

Nevertheless I agree we should avoid small multi-module as much as possible. We can consider doing this for HTTP API one day, but only when we have a clear idea what the next thing looks like.

[dashpole]

Yeah, you can definitely split and do a v2 of just a single folder if you need to. I would wait until you are ready to release the v2 of that folder to make the new module.

But if you are planning to do a v2, you could also make other small changes (e.g. your fancy NewDesc versioning :)) while you are at it.

[bwplotka]

Changed the title to focus on API move only. We could use that for mentorship project.