Can Mimir read secrets from Volume mounts

[wonderphil]

Describe the bug

When using secrets from volume mount mimir applications not starting

Values file:

alertmanager:
  extraVolumes:
    - name: secret-grafana-mimir
      secret:
        secretName: secret-grafana-mimir
  # Extra volume mounts that will be added to the alertmanager container
  extraVolumeMounts:
    - name: secret-grafana-mimir
      mountPath: "/etc/secret"
      readOnly: true

mimir:
  structuredConfig:
    
    alertmanager_storage:
      backend: azure
      azure:
        ## Recplaced the ${} var with the file path
        account_name: /etc/secret/MIMIR_STORAGE_ACCOUNT_NAME
        account_key: /etc/secret/MIMIR_STORAGE_ACCOUNT_KEY
        container_name: mimir-alaertmanager
    compactor:
      data_dir: "/data"
...

if I bash into a container and look at secret files I can see the correct content but I get the following errors for all the services

Error from Mimir Alert manager:

ts=2023-09-28T06:03:38.93765098Z caller=main.go:225 level=info msg="Starting application" version="(version=2.10.0, branch=HEAD, revision=77906f7)"
ts=2023-09-28T06:03:39.337442912Z caller=server.go:335 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2023-09-28T06:03:39.537634469Z caller=multitenant.go:156 level=warn msg="The configured Alertmanager HTTP prefix '/alertmanager' is different than the path specified in the external URL 'http://prometheus-alertmanager.observability:9093': the Alertmanager UI and API may not work as expected unless you have a reverse proxy exposing the Alertmanager endpoints under '/alertmanager' prefix"
ts=2023-09-28T06:03:39.538016272Z caller=log.go:87 level=error msg="error running application" err="decode account key: illegal base64 data at input byte 17\nerror initialising module: alertmanager\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

Error from Mimir Ingestor

ts=2023-09-28T06:02:59.75854223Z caller=main.go:225 level=info msg="Starting application" version="(version=2.10.0, branch=HEAD, revision=77906f7)"
ts=2023-09-28T06:02:59.762267683Z caller=log.go:87 level=error msg="error running application" err="decode account key: illegal base64 data at input byte 17\ncreate usage-stats bucket client\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initUsageStats\n\t/__w/mimir/mimir/pkg/mimir/modules.go:905\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nerror initialising module: usage-stats\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

All other services have similar errors (Alert manager, compactor, ingester, querier, querier-frontend, ruler, storage-gateway)

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version) - version="(version=2.10.0, branch=HEAD, revision=77906f7)"
2. add the above values
3. deploy to K8 cluster and wait for pods to deploy (K8 version v1.26.6, hosted on Azure)

Expected behavior

Expect each service to read the secrets file and use the contents

Environment

• Infrastructure: Azure Kubernetes version 1.26.6
• Deployment tool: argocd and helm charts

Additional Context

Logs are above along with config file

[charleskorn]

Reading secrets from a file is not supported by Mimir.

Have you tried mounting the secrets as environment variables and referring to them with the ${ENVIRONMENT_VARIABLE_NAME} syntax? Would that work for your situation?

[hobbsh]

@charleskorn I am going off of what was discussed/recommended here: #5359 (comment)

Alertmanager does support loading from files (for instance service_key_file) but when I try to use it with mimirtool, it gets kicked back:

mimirtool: error: POST request to https://<mimir_url>/api/v1/alerts failed: server returned HTTP status: 400 Bad Request, body: "error validating Alertmanager config: setting PagerDuty service_key_file is not allowed\n", try --help

[charleskorn]

Yep, that's what I meant when I said reading from files isn't supported - this is blocked in Mimir for security reasons.

[hobbsh]

So what is recommended for referencing secrets in the Alertmanager config?

[marshallford]

@hobbsh How did you end up resolving this?

[hobbsh]

I'm not proud of it but I ended up writing a bash script that pulls the secret data from SecretManager and updates specific YAML values in our templates, then calls mimirtool to sync. It's run as a GitHub Action so people don't have to worry about it. Not ideal at all and pretty frustrating since this is something that AlertManager/Prometheus supported originally.