Verify client certs for Prometheus deployments

[tnozicka]

Is this a bug report or feature request?

• Feature Request

What should the feature do:
Currently the managed Prometheus that is part of the new monitoring stack doesn't force mTLS certificate verification.

scylla-operator/assets/monitoring/prometheus/v1/prometheus.yaml

Lines 21 to 22 in f20887d

	# clientAuthType: "RequireAndVerifyClientCert"
	# TODO: we need the prometheus-operator not to require certs only for /-/readyz or to do exec probes that can read certs

This was done temporarily on purpose because the prometheus-operator sets up probes behind authenticated enpoints, which obviously doesn't work because kubelets don't have the client certs for mTLS. We need to start by creating a simple reproducer and report it to the prometheus-operator.

What is use case behind this feature:
Security

fyi @YvanDaSilva (so you are not surprised when this gets fixed)

Requires

Beta Give feedback

1. Option to disable security on Prometheus health endpoints, /-/healthy and /-/ready prometheus/prometheus#9166
   priority/Pmaybe +1
2. Support path exclusion from mTLS and Basic Auth in prometheus exporter-toolkit #2187
   priority/important-longterm +1
   
3. Readiness probe fails with mTLS enabled and "RequireAndVerifyClientCert" clientAuthType prometheus-operator/prometheus-operator#5419
   dependency/external kind/bug stale +3

[rzetelskik]

Opened prometheus-operator/prometheus-operator#5419
fyi @tnozicka

[rzetelskik]

Tried to follow it up in the most recent PR which tried to address it: prometheus/exporter-toolkit#106. It seems to have lost traction and I haven't received any replies so far. The developers seem to agree on an approach of excluding certain paths from cert verification - although I don't know how exactly they want to achieve that atm. Anyway waiting for a reply there to agree on an approach - I wouldn't want to invest into sending a PR if we don't get anyone to look at it.

[rzetelskik]

Just sent prometheus/exporter-toolkit#151. I will update this issue once (if?) it gets merged.

[rzetelskik]

No response from developers so far. Pinged the maintainers on prometheus/exporter-toolkit#151 and reached out on their slack channel in CNCF workspace. If I don't get a reply in the next couple of days, I'll attend Prometheus Developer Office Hours next Monday.

[rzetelskik]

I added the PR to the Developer Office Hours' agenda today but was only informed by the moderator that we'll just have to wait for the maintainers to reply in the PR. Pinged the maintainers again.

[rzetelskik]

As per prometheus/exporter-toolkit#151 (comment), this should be discussed on Prometheus Dev Summit today.

Update: it wasn't 🫠

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[rzetelskik]

/remove-lifecycle stale
/triage accepted

[rzetelskik]

Update: it seems like prometheus/exporter-toolkit#151 is likely to finally go through as I've got a first review. @tnozicka can we try accommodating this again in our roadmap?

[rzetelskik]

Update: prometheus/exporter-toolkit#151 got an approval and is waiting to be merged. The next step after that would be to update the prometheus-operator issue (or send a PR myself) but we'll have to wait for exporter-toolkit release and for it to propagate to prometheus itself. So at this point this item is blocked.