Vault fails to restart with SIGSEGV after adding a ldap role with a subpath to the ldap secret engine

[ma44in]

Describe the bug

After adding a role with a subpath, like "ldap/static-role/dir1/user1" to the ldap secret engine, vault gets in following trouble.

1. It fails to start after restart and unseal.

...
2024-07-30T09:26:55.794Z [INFO]  core: usage gauge collection is disabled
2024-07-30T09:26:55.794Z [DEBUG] secrets.ldap.ldap_0ca1c775: unexpected nil static role found while loading managed users: name=dir1/
2024-07-30T09:26:55.794Z [INFO]  secrets.ldap.ldap_0ca1c775: initializing database rotation queue
2024-07-30T09:26:55.794Z [INFO]  secrets.ldap.ldap_0ca1c775: populating role rotation queue
2024-07-30T09:26:55.794Z [DEBUG] secrets.ldap.ldap_0ca1c775: no WAL entries found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6fd496d]

goroutine 465 [running]:
github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).populateQueue(0xc003830480, {0xd037178, 0xc00338e000}, {0xd037610, 0xc0031a4580})
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rotation.go:67 +0x42d
github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).initQueue(0xc003830480, {0xd037178, 0xc00338e000}, 0xc0030e61a0)
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rotation.go:461 +0x107
created by github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).initialize in goroutine 438
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/backend.go:101 +0xe8

2. Before restarting the view of the entry in the UI fails.

To Reproduce
Steps to reproduce the behavior:

The error can be reproduced with a clean container setup like follows.

#!/bin/sh

set -u -e -x

podman network create --ignore vault

podman volume rm -f vault || true

podman volume create --ignore vault

podman create --replace --rm \
  --name vault \
  --network vault \
  --env VAULT_ADDR=http://127.0.0.1:8200 \
  --env 'VAULT_LOCAL_CONFIG={"log_level": "Debug","disable_mlock": true, "storage": {"file": {"path": "/vault/file"}}, "listener": [{"tcp": { "address": "0.0.0.0:8200", "tls_disable": true}}], "default_lease_ttl": "168h", "max_lease_ttl": "720h", "ui": true}' \
  -v vault:/vault/file \
  -p 8200:8200 \
  hashicorp/vault server 

podman create --replace --rm \
  --name openldap \
  --network vault \
  --env LDAP_ADMIN_USERNAME=admin1 \
  --env LDAP_ADMIN_PASSWORD=admin1 \
  --env LDAP_USERS=user1 \
  --env LDAP_PASSWORDS=user1 \
  --env LDAP_ROOT=dc=example,dc=org \
  --env LDAP_ADMIN_DN=cn=admin1,dc=example,dc=org \
  -p 1389:1389 \
  bitnami/openldap:latest

podman start vault
podman start openldap

sleep 2

podman exec -it vault vault operator init -key-shares=1 -key-threshold=1 -format=json | tee ./vault_init.json

podman exec -it vault vault status || true

UNSEAL_KEY="$(cat ./vault_init.json | jq -r '.unseal_keys_b64[0]')"
ROOT_TOKEN="$(cat ./vault_init.json | jq -r '.root_token')"

podman exec -it vault vault operator unseal ${UNSEAL_KEY}

podman exec --env VAULT_TOKEN=${ROOT_TOKEN} -it vault vault secrets enable ldap

podman exec --env VAULT_TOKEN=${ROOT_TOKEN} -it vault vault write ldap/config binddn=CN=admin1,DC=example,DC=org bindpass=admin1 url=ldap://openldap:1389

podman exec --env VAULT_TOKEN=${ROOT_TOKEN} -it vault vault write ldap/static-role/dir1/user1 dn='cn=user1,ou=users,dc=example,dc=org' username='user1' rotation_period="24h"

podman restart vault

podman logs -f vault &

podman exec --env VAULT_TOKEN=${ROOT_TOKEN} -it vault vault operator unseal ${UNSEAL_KEY}

Log of the complete script output:

reproduce_output.log

Expected behavior

Vault should not allow to add a bad entry with a subpath like "ldap/static-role/dir1/user1" if it can't deal with it afterwards.

Environment:

• Vault Server Version (retrieve with vault status):

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.17.2
Build Date         2024-07-05T15:19:12Z
Storage Type       file
HA Enabled         false

• Vault CLI Version (retrieve with vault version):

Vault v1.17.2 (2af5655e364f697a15b1dc2db2c3f85f6ef949f2), built 2024-07-05T15:19:12Z

• Server Operating System/Architecture:

Vault server configuration file(s):

{"log_level": "Debug","disable_mlock": true, "storage": {"file": {"path": "/vault/file"}}, "listener": [{"tcp": { "address": "0.0.0.0:8200", "tls_disable": true}}], "default_lease_ttl": "168h", "max_lease_ttl": "720h", "ui": true}

[sbskas]

We're also bitten by this bug

Same config, same stackstrace, same subpath with ldap/Active Directory, clustered server with raft backend.
Happens as soon as you restart one of the server or you seal/unseal it, all of the other servers crash and restart.

Aug 28 11:34:28 server3 vault[297000]: 2024-08-28T11:34:28.212+0200 [INFO]  core: usage gauge collection is disabled
Aug 28 11:34:28 server3 vault[297000]: 2024-08-28T11:34:28.212+0200 [INFO]  secrets.ldap.ldap_d49f9119: initializing database rotation queue
Aug 28 11:34:28 server3 vault[297000]: 2024-08-28T11:34:28.212+0200 [INFO]  secrets.ldap.ldap_d49f9119: populating role rotation queue
Aug 28 11:34:28 server3 vault[297000]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 28 11:34:28 server3 vault[297000]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6fd978d]
Aug 28 11:34:28 server3 vault[297000]: goroutine 778 [running]:
Aug 28 11:34:28 server3 vault[297000]: github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).populateQueue(0xc002fbd780, {0xd03cb58, 0xc0039bc320}, {0xd03cff0, 0xc00342ca80})
Aug 28 11:34:28 server3 vault[297000]:         /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rotation.go:67 +0x42d
Aug 28 11:34:28 server3 vault[297000]: github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).initQueue(0xc002fbd780, {0xd03cb58, 0xc0039bc320}, 0xc003647ea0)
Aug 28 11:34:28 server3 vault[297000]:         /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rotation.go:461 +0x107
Aug 28 11:34:28 server3 vault[297000]: created by github.com/hashicorp/vault-plugin-secrets-openldap.(*backend).initialize in goroutine 789
Aug 28 11:34:28 server3 vault[297000]:         /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/backend.go:101 +0xe8

[biazmoreira]

@raskchanky can I mark this as reproduced?