Consul service is deregistered on SIGHUP but not synced afterwards

[easkay-castlight]

Describe the bug
Service status/registration in Consul is lost on reload/SIGHUP.

To Reproduce
Steps to reproduce the behavior:

1. Setup a Vault cluster with Consul service registration via Consul backend storage (Consul running as local agent on Vault server).
2. Unseal Vault as required.
3. Send a SIGHUP to Vault and note the logs in Consul.
4. Query the service status in Consul.
5. Restart & unseal Vault and compare Consul logs and service status.

Expected behavior
Upon reload if the service is deregistered with Consul, it should be re-registered when Vault has successfully launched.

Environment:

• Vault Server Version (retrieve with vault status): 1.17.3 (and 1.17.2)
• Vault CLI Version (retrieve with vault version): 1.17.3 (and 1.17.2)
• Server Operating System/Architecture: Ubuntu 20.04 amd64

Vault server configuration file(s):

{
  "pid_file": "/run/vault/vault.pid",
  "ui": true,
  "disable_mlock": true,
  "log_format": "json",
  "listener": [
    {
      "tcp": {
        "address": "0.0.0.0:8200",
        "proxy_protocol_behavior": "deny_unauthorized",
        "proxy_protocol_authorized_addrs": [
          "127.0.0.1",
          "<redacted>"
        ],
        "tls_cert_file": "/etc/vault.d/vault.crt",
        "tls_key_file": "/etc/vault.d/vault.key"
      }
    }
  ],
  "storage": {
    "consul": {
      "address": "127.0.0.1:8501",
      "scheme": "https",
      "token": "<redacted>",
      "tls_ca_file": "/etc/vault.d/dev-consul-ca.crt",
      "tls_cert_file": "/etc/vault.d/dev-consul-agents.crt",
      "tls_key_file": "/etc/vault.d/dev-consul-agents.key"
    }
  },
  "cluster_addr": "https://<redacted>:8201",
  "api_addr": "https://<redacted>"
}

Additional context
Relevant logs from system demonstrating behaviour attached, note the log entries for Consul showing the service status around the operations performed on Vault (SIGHUP and restart).
The HAProxy entries are included since data from Consul is used by consul-template to configure HAProxy for HA.
Vault_ServiceDeregistered_Logs.log

EDIT1: added additional steps around restart to reproduction steps, removed template comment from additional context section.

[pdohertybcov]

I think I've got the same issue. It started after upgrading vault from 1.16.3 to 1.17.2
I'm using logrotate to manage the audit logs, and it sends a SIGHUP to vault, which is then causing the service deregistration. It doesn't re-register until I go in and manually restart vault.

My suspicion is the behavior was introduced by #17598

Sep 04 00:00:02 hostname systemd[1]: vault.service: Sent signal SIGHUP to main process 2354486 (vault) on client request.
Sep 04 00:00:02 hostname vault[2354486]: ==> Vault reload triggered
Sep 04 00:00:02 hostname consul[2354476]: 2024-09-04T00:00:02.244Z [INFO]  agent: Deregistered service: service=vault:10.0.0.105:8200

[pdohertybcov]

My initial testing indicates that adding service_registration into the vault config works around this issue, but it's always been the case before that using the Consul storage backend automatically included the service registration.