feat(collector): run builder-api alongside Benthos

[eliteprox]

Third of the #57 split. #78 and #79 have landed; this now targets main. Remaining stack: this → #81 → #82.

Collector-side wiring that runs the builder-api binary beside Benthos: a multi-stage Dockerfile builds the Go service into the collector image, and entrypoint.sh supervises both.

collector.yaml is a three-way merge with #70, not #57's copy

Worth the most review attention. #57 branched before #70, and both rewrote openmeter-collector/collector.yaml. Taking #57's version verbatim would have silently reverted #70's app-owner vs M2M identity split and the exact-fractional fee mapping.

This branch carries the merged pipeline. Main's Benthos suite — which never ran against #57, because #57 predates the openmeter-collector CI job — passes in full, including #70's own cases:

- M2M compound auth_id keeps compound subject and customer key
- owner wire subject strips to bare CloudEvent subject and customer key
- owner bare id is trimmed; upstream usage_subject_type preserved for M2M
- empty owner bare id is dropped
- exact fractional fee micros (no per-ticket ceil)
- non create_signed_ticket events are dropped before mapping
Test 'collector.yaml' succeeded

Ingest credentials are resolved via env() into message metadata rather than ${OPENMETER_URL} interpolations. Benthos substitutes dollar-brace vars in the raw config text before parsing, which made them lint-required even inside comments and blocked startup in the shared-tenant configuration.

One Kong OpenMeter organization serves every tenant; customers are separated by the openmeter_customer_key contract, not per-tenant ingest credentials.

Entrypoint supervises both processes

• Requires OPENMETER_URL and OPENMETER_API_KEY
• Strips any /events suffix so builder-api hits the management API (<base>/customers), while Benthos keeps the ingest URL
• Starts builder-api only when AUTH0_MGMT_CLIENT_ID and AUTH0_MGMT_CLIENT_SECRET are set
• Reaps the first child to exit, tears down the sibling, and propagates that child's status (so kill -9 on benthos surfaces as 137, not a hardcoded 1)
• docker stop / INT/TERM exit 0 rather than falling back into the loop and looking like a crash

IDENTITY_WEBHOOK_URL dropped

Session exchange and go-livepeer both call the same /authorize contract. builder-api now reads REMOTE_SIGNER_WEBHOOK_URL + WEBHOOK_SECRET only — the same vars as the remote signer.

Compose / env

Net compose change vs main is the 127.0.0.1:8095 port binding for builder-api. .env.example documents the builder-api OpenMeter knobs (default plan, trial grant, allowance enforcement).

Verification

• Benthos suite passes against the merged collector.yaml (above)
• shellcheck clean on entrypoint.sh
• docker compose config validates
• docker build -f openmeter-collector/Dockerfile . succeeds; the built image has a working /usr/local/bin/builder-api and the entrypoint's required-env guard fires correctly

[Copilot]

Pull request overview

This PR updates the openmeter-collector container so it can run the Go builder-api service alongside the Benthos collector, including container build wiring, a supervising entrypoint, and collector configuration changes to support per-tenant ingest credentials resolution.

Changes:

• Build and ship the builder-api binary into the openmeter-collector image and run it alongside Benthos via entrypoint.sh.
• Extend the Benthos pipeline to optionally resolve per-tenant OpenMeter ingest URL/token from konnect-credentials and use per-message metadata for output auth/URL.
• Expose builder-api locally via docker-compose port binding and document additional env knobs in .env.example.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
openmeter-collector/entrypoint.sh	Adds env loading, OpenMeter URL normalization for mgmt vs ingest, starts/supervises Benthos + builder-api.
openmeter-collector/Dockerfile	Multi-stage build to compile builder-api and copy it into the Benthos collector image.
openmeter-collector/collector.yaml	Adds optional per-tenant ingest credential lookup and routes output URL/auth via message metadata.
docker-compose.yml	Publishes builder-api port 8095 from the openmeter-collector container to localhost.
.env.example	Documents new builder-api/OpenMeter settings and konnect-credentials-related variables.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

openmeter-collector/entrypoint.sh:92

• When a child process exits, the script always exits 1 and it discards the child’s exit status. This makes failures harder to diagnose and can incorrectly mark a clean shutdown as an error. Capture and propagate the exited child’s status after wait, then exit with that code.

  if ! kill -0 "$benthos_pid" 2>/dev/null; then
    wait "$benthos_pid" 2>/dev/null || true
    cleanup
    exit 1
  fi

openmeter-collector/entrypoint.sh:68

• The INT/TERM trap only runs cleanup but doesn’t exit the entrypoint. With set -e and the supervision loop, this can lead to confusing shutdown behavior (e.g., TERM triggers cleanup, then the loop continues until it notices a dead child and exits 1). It’s safer to exit directly from the trap with a signal-appropriate code.

trap cleanup INT TERM

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

openmeter-collector/entrypoint.sh:20

• Comment says the auth0-provisioner → builder-api env mapping happens "only when unset", but the ${var:=...} form also overwrites variables that are set to an empty string. Either update the comment to match behavior, or switch to ${var=...} if you truly only want the unset case.

# Map auth0-provisioner names → builder-api env (only when unset).

openmeter-collector/entrypoint.sh:86

• openmeter_mgmt_url is always non-empty because OPENMETER_URL is required and openmeter_mgmt_url is initialized from it. The extra if [ -n "$openmeter_mgmt_url" ] branch (and its "OPENMETER_URL unset" message) is effectively unreachable and can be simplified.

if [ -x /usr/local/bin/builder-api ]; then
  if [ -n "${AUTH0_MGMT_CLIENT_ID:-}" ] && [ -n "${AUTH0_MGMT_CLIENT_SECRET:-}" ]; then
    if [ -n "$openmeter_mgmt_url" ]; then
      OPENMETER_URL="$openmeter_mgmt_url" /usr/local/bin/builder-api &
      builder_pid=$!

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

openmeter-collector/collector.yaml:219

• This processor sets meta client_id, but that metadata is never referenced later in the config. It adds per-message work and can confuse future readers; suggest removing it (or wiring it into a log/metric if it’s meant to be used).

    - mapping: |
        meta client_id = this.data.client_id.or("")

openmeter-collector/collector.yaml:229

• OPENMETER_API_KEY is copied into message metadata (meta openmeter_token). Keeping secrets in per-message metadata increases the chance of accidental leakage (e.g., if metadata is ever logged or forwarded). You can avoid storing the token by reading it directly from env() in the Authorization header and only keeping the non-secret URL in metadata.

    - mapping: |
        meta openmeter_url = env("OPENMETER_URL").or("")
        meta openmeter_token = env("OPENMETER_API_KEY").or("")
        root = if meta("openmeter_url") == "" || meta("openmeter_token") == "" {
          throw("openmeter ingest unresolved: set OPENMETER_URL and OPENMETER_API_KEY")

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

openmeter-collector/collector.yaml:233

• The meta client_id processor is introduced here but the meta value is never referenced anywhere else in this config (no meta("client_id") usage). This extra processor adds noise and maintenance surface without affecting the output.

    - mapping: |
        meta client_id = this.data.client_id.or("")

openmeter-collector/entrypoint.sh:92

• This log message references AUTH0_MGMT_CLIENT_ID/SECRET, but the actual env var checked is AUTH0_MGMT_CLIENT_SECRET. Using the exact variable name makes the startup guidance unambiguous for operators.

  else
    echo "builder-api: skipped — AUTH0_MGMT_CLIENT_ID/SECRET not set (re-run auth0-provisioner/provision/bootstrap.sh)" >&2
  fi