We welcome contributions from the community! Please see our Contribution Guidelines for more information on how to get involved.
This project is licensed under the terms of the MIT license. See the LICENSE file for details.
+This project is licensed under the terms of the Apache-2.0 license. See the LICENSE file for details.
For any questions or support, please reach out to our team at directly by creating an issue on the project.
diff --git a/index.html b/index.html index 0bdc4d4..f027743 100644 --- a/index.html +++ b/index.html @@ -1053,7 +1053,7 @@We welcome contributions from the community! Please see our Contribution Guidelines for more information on how to get involved.
This project is licensed under the terms of the MIT license. See the LICENSE file for details.
+This project is licensed under the terms of the Apache-2.0 license. See the LICENSE file for details.
For any questions or support, please reach out to our team at directly by creating an issue on the project.
diff --git a/search/search_index.json b/search/search_index.json index 9cf5cf1..c65849d 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en","fr"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Vault DB Injector","text":""},{"location":"#overview","title":"Overview","text":"The Vault DB Injector is a Kubernetes-based application designed to dynamically generate database credentials using HashiCorp Vault and provide them as environment variables to Kubernetes Pods. The application leverages a Mutating Webhook to achieve this functionality, ensuring secure and automated management of database credentials.
"},{"location":"#key-features","title":"Key Features","text":""},{"location":"#1-dynamic-database-credential-injection","title":"1. Dynamic Database Credential Injection","text":"logrus for structured logging, supporting various log levels and JSON formatting.To get started with Vault DB Injector, refer to the Installation Guide and Configuration Guide for detailed instructions on setting up and configuring the application.
For a deeper understanding of how the application works, visit the How It Works section.
"},{"location":"#contributing","title":"Contributing","text":"We welcome contributions from the community! Please see our Contribution Guidelines for more information on how to get involved.
"},{"location":"#license","title":"License","text":"This project is licensed under the terms of the MIT license. See the LICENSE file for details.
"},{"location":"#contact","title":"Contact","text":"For any questions or support, please reach out to our team at directly by creating an issue on the project.
"},{"location":"getting-started/build/","title":"Build","text":""},{"location":"getting-started/build/#build","title":"Build","text":"Requirements for building
A Makefile is provided for building tasks. The options are as follows
Getting started is as simple as:
go get -u github.com/numberly/vault-db-injector\ncd $GOPATH/src/github.com/numberly/vault-db-injector\nmake setup\nmake\nall-rw vault-policyK/V Vaultvault databases mountvault databases backend connectionkubernetes auth backendkubernetes auth backend rolevault databases backend roleDatabase configurationTo work correctly, the vault db injector need the following :
In this documentation, we assume that both of those following requirements are already installed.
"},{"location":"getting-started/getting-started/#2-vocabulary","title":"2. Vocabulary","text":"Here are some vocabulary that you need before starting the installation.
name description default K/V Vault Vault KVv2 path for injector state vault-injector vault policy Vault Injector policy to work properly all-rw vault databases mount Vault database engine used to connect to our database databases vault databases backend connection Vault databases backend connection where we allow role to generate credentials rw-pgsql2-pr vault databases backend role Vault databases role used to allow application on specific database test-role kubernetes auth backend Vault Backend used by Kubernetes Application to authenticate under Vault kubernetes kubernetes auth backend role Vault Backend Role used by injector to authenticate under Vault all-rw postgresql Database engine used to connecte under vault rw-pgsql2-pr database database name to connect into test"},{"location":"getting-started/getting-started/#3-vault-configuration","title":"3. Vault Configuration","text":"First, we need to configure Vault to allow the injector to generate credentials on the databases engine and to store them inside the dedicated K/V Vault.
all-rw vault-policy","text":"We are going to start by create the all-rw vault policy that the injector will use :
path \"vault-injector/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"vault-injector/data/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"vault-injector/metadata/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"rw-pgsql2-pr/creds/*\" {\n capabilities = [\"read\"]\n}\npath \"auth/kubernetes/role/*\" {\n capabilities = [\"read\"]\n}\npath \"sys/leases/renew\" {\n capabilities = [\"create\"]\n}\npath \"auth/token/renew-self\" {\n capabilities = [\"create\"]\n}\npath \"auth/token/renew\" {\n capabilities = [\"create\", \"update\"]\n}\npath \"auth/token/revoke\" {\n capabilities = [\"create\", \"update\"]\n}\npath \"auth/token/create\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}\npath \"auth/token/create-orphan\" {\n capabilities = [\"create\", \"update\", \"read\", \"sudo\"]\n}\npath \"auth/token/revoke-orphan\" {\n capabilities = [\"create\", \"update\", \"sudo\"]\n}\nK/V Vault","text":"We need to create the K/V Vault in version v2 named vault-injector.
You can use the following documentation : vault-injector
"},{"location":"getting-started/getting-started/#33-create-vault-databases-mount","title":"3.3. Createvault databases mount","text":"We need to create a vault databases mount engine named databases.
Here is a terraform example :
resource \"vault_mount\" \"databases\" {\n path = \"databases\"\n type = \"database\"\n description = \"databases authentication automation\"\n max_lease_ttl_seconds = \"31536000\"\n}\nvault databases backend connection","text":"We need to create a vault databases backend connection on the vault databases engine. As you can see below, we allow the test-role role.
Here is a terraform example :
resource \"vault_database_secret_backend_connection\" \"pgsql2\" {\n backend = vault_mount.databases.path\n name = \"pgsql2\"\n allowed_roles = [\n \"test-role\",\n ]\n\n postgresql {\n connection_url = \"postgres://{{username}}:{{password}}@rw-pgsql2-pr:5432/postgres?sslmode=verify-full\"\n username = \"postgres\"\n password = \"my-password\"\n username_template = \"{{.RoleName}}-{{unix_time}}-{{random 8}}\"\n }\n}\nkubernetes auth backend","text":"We need to create a kubernetes auth backend to allow serviceAccount to connect under Vault.
You can use the following documentation : kubernetes
"},{"location":"getting-started/getting-started/#36-create-a-kubernetes-auth-backend-role","title":"3.6. Create akubernetes auth backend role","text":"We need to create a kubernetes auth backend role to allow the service account of the vault-db-injector to connect under Vault.
Here is a terraform example :
resource \"vault_kubernetes_auth_backend_role\" \"all_rw\" {\n backend = \"kubernetes\"\n role_name = \"all-rw\"\n bound_service_account_names = [\"vault-db-injector\"]\n bound_service_account_namespaces = [\"vault-db-injector\"]\n token_ttl = 3600\n token_policies = [vault_policy.all_rw.name] # remember the one created before\n token_bound_cidrs = [\"10.17.0.0/16\"] # Your pod CIDR\n}\nWe need to create a kubernetes auth backend role to allow the service account of our application to get generated credentials from vault Here is a terraform example :
resource \"vault_policy\" \"policy\" {\n provider = vault.main\n name = var.service_account\n\n policy = <<EOT\npath \"pgsql2/creds/test-role\" {\n capabilities = [\"read\"]\n}\nEOT\n}\n\nresource \"vault_kubernetes_auth_backend_role\" \"role\" {\n provider = vault.main\n backend = kubernetes\n role_name = \"test\"\n bound_service_account_names = [\"test\"]\n bound_service_account_namespaces = [\"test\"]\n token_ttl = 3600\n token_policies = [vault_policy.policy.name]\n token_bound_cidrs = [\"10.17.0.0/16\"]\n}\nvault databases backend role","text":"We need to create a vault databases backend role to allow our application to consume vault databases backend connection
Here is a terraform example :
resource \"vault_database_secret_backend_role\" \"role\" {\n provider = vault.main\n backend = \"databases\"\n name = test-role\n db_name = test\n default_ttl = 3600\n creation_statements = [\n \"CREATE ROLE \\\"{{name}}\\\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \\\"test\\\";\",\n \"ALTER ROLE \\\"{{name}}\\\" SET ROLE \\\"test\\\";\",\n ]\n revocation_statements = [\n \"DROP ROLE \\\"{{name}}\\\";\"\n ]\n}\nDatabase configuration","text":"You need to create you database under postgres, here is an example for postgres :
CREATE DATABASE test;\nCREATE ROLE test;\nrevoke all on database test from public cascade;\ngrant connect on database test to test;\n\\c test\ngrant create, usage on schema public to test;\ngrant temporary on database test to test;\nALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO test;\nREVOKE ALL ON pg_user FROM public;\nREVOKE ALL ON pg_roles FROM public;\nREVOKE ALL ON pg_group FROM public;\nREVOKE ALL ON pg_authid FROM public;\nREVOKE ALL ON pg_auth_members FROM public;\nREVOKE ALL ON pg_database FROM public;\nREVOKE ALL ON pg_tablespace FROM public;\nREVOKE ALL ON pg_settings FROM public;\nNow that you have a vault correctly configured and a database ready to be used, we can deploy our vault-db-injector application :
For this, its quit easy, you just need to use the help chart.
kubectl create namespace vault-db-injector\nhelm upgrade --install vault-db-injector . --namespace vault-db-injector\nWhen everything is Okay, you should have something like this :
NAME READY STATUS RESTARTS AGE\nvault-db-injector-7f74977b7c-88vvg 1/1 Running 0 29s\nvault-db-injector-7f74977b7c-rq6mt 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-77skb 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-96zz4 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-tdp4r 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-wpd8x 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-2m5qp 1/1 Running 0 28s\nvault-db-injector-revoker-7965857f75-5msv6 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-n29wp 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-th9vs 1/1 Running 0 28s\nHere is an example application deploy on the namespace test that will connect to our database test with the service account test.
apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: test\n namespace: test\n---\napiVersion: v1\nkind: Pod\nmetadata:\n annotations:\n db-creds-injector.numberly.io/test.env-key-uri: POSTGRES_URL\n db-creds-injector.numberly.io/test.template: postgresql://@rw-pgsql2-pr:5432/test?sslmode=require\n db-creds-injector.numberly.io/test.mode: uri\n db-creds-injector.numberly.io/test.role: test-role\n labels:\n client: numberly\n vault-db-injector: \"true\"\n name: test\n namespace: test\nspec:\n containers:\n - image: postgres:15.5\n command: [\"/bin/bash\", \"-c\"]\n args: [\"sleep 3000000\"]\n name: test\n resources: {}\n securityContext:\n allowPrivilegeEscalation: false\n capabilities:\n drop:\n - ALL\n readOnlyRootFilesystem: true\n runAsNonRoot: true\n runAsUser: 65534\n dnsPolicy: ClusterFirst\n restartPolicy: Always\n serviceAccount: test\n serviceAccountName: test\nTo deploy it :
kubectl apply -f test.yaml\nWait until your application is ready and you should be able to exec inside the pods and connect to the database.
"},{"location":"how-it-works/configuration/","title":"Configuration","text":"* 1. [Configuration](#Configuration)\n * 1.1. [Mode Injector](#ModeInjector)\n * 1.2. [Mode token-renewer](#Modetoken-renewer)\n * 1.3. [Mode token-renewer](#Modetoken-renewer-1)\nHere is the configuration for Vault Injector:
"},{"location":"how-it-works/configuration/#11-mode-injector","title":"1.1. Mode Injector","text":"The Injector mode is basic one that will handle all api-server request and handle all requests to vault to generate credentials to our DB The config file path can be parse by adding the path with : - \"--config=/injector/config.yaml\"
certFile: /tls/tls.crt\nkeyFile: /tls/tls.key\nvaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: injector\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nThe Renewer one is a process that will run every hour and validate that all orphan token won't expire before pod is deleted The config file path can be parse by adding the path with : - \"--config=/renewer/config.yaml\"
vaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: renewer\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\nSyncTTLSecond: 300\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nThe Revoker one is a process that is going to watch pod deletion Kubernetes events filtered with the label vault-db-injector: true and will revoke token attached to the pod when it is deleted The config file path can be parse by adding the path with : - \"--config=/revoker/config.yaml\"
vaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: revoker\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nKey File: pkg/healthcheck/healthcheck.go
Health Checks are a crucial feature that monitors the application's status and readiness, ensuring it is functioning correctly and is ready to handle requests. This feature provides endpoints that external systems can query to check the health and readiness of the application.
"},{"location":"how-it-works/healthcheck/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application exposes HTTP endpoints for health (/healthz) and readiness (/readyz). These endpoints provide information about the application's operational status.
Regular Monitoring:
The health check service regularly monitors the internal state of the application and updates the health and readiness status accordingly.
Integration with Kubernetes:
Health checks ensure that the application is running correctly and can handle incoming requests. This reliability is critical for maintaining user trust and satisfaction.
Proactive Issue Detection:
By regularly monitoring the application's status, health checks can detect issues early, allowing for proactive resolution before they impact users.
Kubernetes Compatibility:
The Vault DB Injector is a Go program that is design to retrieve databases credentials from Hashicorp Vault, it use Kubernetes Mutating Webhook to intercept pod creation activated with a label and configured with annotations. After the credentials are provided, it will store them in a specific Vault KV and will handle the lifecycle of them such as : - Renew them periodically - Revoke them after the pod is deleted
"},{"location":"how-it-works/how-it-work/#11-mechanism","title":"1.1. Mechanism","text":""},{"location":"how-it-works/how-it-work/#111-vault-injector","title":"1.1.1. Vault Injector:","text":"Their is also a fonctionnality that permit to rotate token directly from the injector. Objectif is to keep token as secure as possible. For this purpose, we have made the choice to store the token directly on Vault in a specific KV. We provide a generated random UUID to every pod which will be use as a unique identifier. As long as the pod is alive, the injector will rotate his token 5 minutes by default. If the pods is deleted, the revoker will revoke the token and the lease and the renewer which is design to keep the state will also delete the secret in the KV.
If the injector reboot or fail for any reason, the token can stil be renewed manually or by restarting the injector correctly. He can retrieve all token that was created previously. The renewer will renew all tokens and leases every 5 minutes by default.
We are using periodic token for this purpose which need absolutely \"sudo\" policy. Periodic token permit to add a max_ttl and to have infinite token that can be renewed until the pod is deleted. Token TTL permit to fix the max_ttl of the token which mean : \"How many time my token can belong without been renewed.\" By default, if the token is not renewed, hes timelife is 32 days, that mean, if the injector fail, you have 31 days and 23 hours before all your token will expire, which i think is enough to understand why and repair it to a working state.
With this setup, the advantage is that now, vault is the only real SPOF. The Injector in renewer mode will only renew TOKEN and LEASE and cleanup KV/STORE that has been created by the Injector. The Injector in revoker mode will only revoke TOKEN when a pod has been deleted If the revoker fail to revoke a Token, the renewer can do it periodically every 5 minutes (This will be probably removed in future version.)
"},{"location":"how-it-works/how-it-work/#14-authentication","title":"1.4. Authentication","text":"It uses a service account and the Kubernetes mount point to retrieve and generate its information. It then sends the following information to the application:
It will store on a specific vault KV/Store the following :
They are two kind of token that are created in the lifecycle of the Vault injector :
kubeRole: in the yaml configvaultSecretName: vault-injector and prefix : vaultSecretPrefix: kubernetes1-dv-par5db-creds-injector.numberly.io/role: and db-creds-injector.numberly.io/dbname.role:tokenTTL: 768hvault-db-injector: \"true\", annotations db-creds-injector.numberly.io/cluster: & db-creds-injector.numberly.io/role: and a serviceaccountLeaseId, TokenId, namespace in a folder named with the pod UUIDThe available annotations can be declare like this :
db-creds-injector.numberly.io/cluster, [OPTIONAL] default to databases which is the database engine to usedb-creds-injector.numberly.io/role, [MANDATORY] Role to be use to get db credentialsdb-creds-injector.numberly.io/dbname.role, [OPTIONAL] Role to be use to get db credentials for this specific databasedb-creds-injector.numberly.io/dbname.env-key-dbuser, [OPTIONAL] overwrite DB user env variable, default to DBUSERdb-creds-injector.numberly.io/dbname.env-key-dbpassword, [OPTIONAL] overwrite DB password env variable, default to DBPASSWORDdb-creds-injector.numberly.io/dbname.mode, [MANDATORY] the mode for the injector, default to classic.# To be added in the pod spec\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.env-key-dbpassword: DB_PASSWORD\n db-creds-injector.numberly.io/dbname.env-key-dbuser: DB_USER\n db-creds-injector.numberly.io/dbname.role: db-role # the one created from vault with terraform\n db-creds-injector.numberly.io/dbname.mode: classic\nlabels:\n vault-db-injector: \"true\"\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.template: postgres://@postgres-server.tld:5432/dbname?sslmode=require\n db-creds-injector.numberly.io/dbname.role: db-role\n db-creds-injector.numberly.io/dbname.env-key-uri: POSTGRES_URL\n db-creds-injector.numberly.io/dbname.mode: uri\nlabels:\n vault-db-injector: \"true\"\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.env-key-dbpassword: DB_PASSWORD\n db-creds-injector.numberly.io/dbname.env-key-dbuser: DB_USER\n db-creds-injector.numberly.io/dbname.role: db-role # the one created from vault with terraform\n db-creds-injector.numberly.io/dbname.mode: classic\n db-creds-injector.numberly.io/other_dbname.template: postgres://@postgres-server.tld:5432/dbname?sslmode=require\n db-creds-injector.numberly.io/other_dbname.role: another-vault-role\n db-creds-injector.numberly.io/other_dbname.env-key-uri: POSTGRES_URL,ANOTHER_ENV\n db-creds-injector.numberly.io/other_dbname.mode: uri\nlabels:\n vault-db-injector: \"true\"\ndbname and other_dbname."},{"location":"how-it-works/injector/","title":"Injector","text":"Key File: pkg/injector/injector.go
The injector is responsible for injecting database credentials into Kubernetes Pods using a Mutating Admission Webhook.
When a Pod is created, the webhook intercepts the request and modifies the Pod specification to include environment variables with the credentials.
Credential Injection:
By automating the injection of credentials, the injector ensures that Pods have the necessary credentials without storing them statically, enhancing security.
Transparent Operation:
The Mutating Admission Webhook operates transparently, modifying Pod specifications on-the-fly without manual intervention.
Security:
Key Files: pkg/k8s/connect.go, pkg/k8s/pod_utils.go, pkg/k8s/parse_annotations.go
Kubernetes Integration is a fundamental feature that enables the application to interact with the Kubernetes API. This integration allows the application to manage and manipulate Kubernetes resources, facilitating tasks such as credential injection, pod management, and more.
"},{"location":"how-it-works/kubernetes/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application initializes a Kubernetes client that can interact with the Kubernetes API. This client is used to perform various operations such as reading pod annotations, accessing secrets, and more.
Service Account Token Retrieval:
The application retrieves the service account token from the Kubernetes environment. This token is used for authenticating API requests made by the application.
CA Certificate Retrieval:
The application retrieves the Kubernetes CA certificate. This certificate is used to establish secure communication with the Kubernetes API server.
Pod Annotation Parsing:
By integrating with the Kubernetes API, the application can perform a wide range of operations directly within the Kubernetes cluster. This seamless integration simplifies the management of Kubernetes resources.
Security:
Using service account tokens and CA certificates ensures that all interactions with the Kubernetes API are secure and authenticated. This enhances the security of the application's operations.
Dynamic Configuration:
Each injector annotation are read by the Injector pod and permit to configure properly how Database Credetials need to be handled.
"},{"location":"how-it-works/leaderelection/","title":"Leader Election","text":"Key File: pkg/leadership/leadership.go
Leader Election is a mechanism that ensures high availability and fault tolerance by designating a single instance of the application to perform critical tasks at any given time. This feature is particularly important in distributed systems where multiple instances of an application might be running concurrently.
"},{"location":"how-it-works/leaderelection/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"Leader Election uses Kubernetes resource locks to manage which instance of the application is the current leader. The application instances compete for a lock, and the one that acquires it becomes the leader.
Performing Critical Tasks:
The leader is responsible for performing tasks that should not be duplicated. These tasks could include credential renewal, revocation, and other maintenance activities that need to be managed centrally.
Failover Handling:
By ensuring that there is always one active leader performing critical tasks, the system can provide high availability and resilience. If the current leader fails, another instance takes over, maintaining operational continuity.
Fault Tolerance:
Leader Election allows the system to handle failures gracefully. The loss of a leader does not disrupt critical operations because another instance will quickly assume leadership.
Efficient Resource Management:
This feature is essential for applications deployed in Kubernetes clusters, where ensuring that critical tasks are managed by a single, designated instance enhances both reliability and performance.
"},{"location":"how-it-works/renewer/","title":"Renewer","text":"Key File: pkg/renewer/renewer.go
The Renewer is a crucial component responsible for ensuring that database credentials remain valid over time. It periodically checks the status of the credentials and renews them before they expire. This process is critical in environments where credentials have a limited lifespan and need to be refreshed to maintain access.
"},{"location":"how-it-works/renewer/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The Renewer periodically checks the expiration status of the current database credentials. It ensures that credentials are always valid and do not reach their expiration date unnoticed.
Credential Renewal:
By automating the renewal of credentials, the Renewer eliminates the need for manual intervention, reducing administrative overhead and the risk of human error.
Continuous Availability:
Ensures that applications have uninterrupted access to the database by renewing credentials well before they expire, thus preventing potential downtime.
Security:
This feature is particularly useful for maintaining secure and continuous database access in dynamic cloud environments, where credential rotation is a best practice for security.
"},{"location":"how-it-works/revoker/","title":"Revoker","text":"Key File: pkg/revoker/revoker.go
The Revoker is responsible for ensuring that unused or stale database credentials are properly revoked. This process is essential for maintaining the security of your system by ensuring that credentials that are no longer needed are not left active.
"},{"location":"how-it-works/revoker/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The Revoker identifies credentials that are no longer in use or have become stale. This could be because the application instance that was using them has been terminated, or the credentials have exceeded their intended lifespan.
Revoking Credentials:
Once stale or unused credentials are identified, the Revoker sends a request to HashiCorp Vault to revoke these credentials. This action ensures that these credentials can no longer be used to access the database.
Updating System State:
The Revoker updates the system's state to reflect that certain credentials have been revoked. This helps in maintaining an accurate and secure overview of active and inactive credentials.
Automatic revocation:
DELETE event on pod so it can revoke the credentials direcly after the pod is Deleted, this permit to be sure credentials are not revoked before the pods completely deleted.By revoking unused or stale credentials, the Revoker minimizes the risk of unauthorized access. This is crucial in environments where security is a top priority.
Reduced Attack Surface:
Removing credentials that are no longer needed reduces the number of potential entry points for attackers, thus lowering the overall risk profile of the system.
Compliance and Audit:
This feature is particularly important in dynamic environments where resources and their associated credentials are frequently created and destroyed. It ensures that credentials do not linger beyond their useful life, thereby maintaining a high standard of security.
"},{"location":"how-it-works/vault/","title":"Vault Integration","text":"Key Files: pkg/vault/handle_token.go, pkg/vault/vault.go
Vault Integration is a crucial feature that handles interactions with HashiCorp Vault for generating and managing database credentials. This integration ensures that the application can securely request and use credentials from Vault.
"},{"location":"how-it-works/vault/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application initializes a Vault client using the provided configuration. This client is used to authenticate and communicate with the Vault server.
Authentication with Vault:
The application authenticates with Vault using a Kubernetes authentication method
Requesting Credentials:
When the application needs database credentials, it sends a request to Vault. Vault generates the credentials and returns them to the application.
Handling Tokens:
By integrating with Vault, the application can securely manage database credentials. Vault ensures that credentials are generated securely and rotated regularly, enhancing overall security.
Dynamic Credential Generation:
Credentials are generated on demand, which means they are always fresh and have limited lifespans. This dynamic generation reduces the risk of credential compromise.
Centralized Secret Management:
This configuration defines a set of alerts for monitoring the VaultDb Injector within a Kubernetes environment. Each alert is designed to notify the team of potential issues that could impact the availability, security, or functionality of the services relying on Vault for secret management.
"},{"location":"monitoring/alertmanager/#1-alerts-configuration","title":"1. Alerts Configuration","text":""},{"location":"monitoring/alertmanager/#11-service-account-denied","title":"1.1. Service Account Denied","text":"- alert: VaultDbInjectorServiceAccountDenied\n annotations:\n description: \"Service Account (SA) `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied access to db_role `{{ $labels.db_role }}` due to `{{ $labels.cause }}` on cluster `{{ $labels.k8s_cluster }}`. Immediate investigation is recommended to ensure proper access controls and service configurations.\"\n summary: \"Service Account `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied by the injector.\"\n expr: increase(vault_injector_service_account_denied_count{}[2m]) > 0\n for: 1m\n labels:\n severity: critical\nResponse Actions: - Verify the service account permissions and roles. - Check the db_role configurations to ensure they are correctly set up. - Investigate the cause for denial to prevent future occurrences.
"},{"location":"monitoring/alertmanager/#12-token-renewal-failure","title":"1.2. Token Renewal Failure","text":"- alert: VaultDbInjectorFailToRenewToken\n annotations:\n description: \"VaultDbInjector encountered an error while attempting to renew a token. This might affect the continuous operation of dependent services. Check for errors and ensure the token renewal process is configured correctly.\"\n summary: \"VaultDbInjector token renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: increase(vault_injector_renew_token_count_error{}[2m]) > 0\n for: 1m\n labels:\n severity: warning\nResponse Actions: - Review the injector logs for errors related to token renewal. - Ensure the Vault policies allow for token renewal by the injector. - Check for network issues that might prevent the injector from communicating with Vault.
"},{"location":"monitoring/alertmanager/#13-lease-renewal-failure","title":"1.3. Lease Renewal Failure","text":"- alert: VaultDbInjectorFailToRenewLease\n annotations:\n description: \"VaultDbInjector encountered an error while attempting to renew a lease. Similar to token renewal failures, this can disrupt service operations if not addressed.\"\n summary: \"VaultDbInjector lease renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: increase(vault_injector_renew_lease_count_error{}[2m]) > 0\n for: 1m\n labels:\n severity: warning\nResponse Actions: - Inspect the injector logs for specific errors related to lease renewal. - Confirm that the Vault configuration allows the injector to renew leases. - Investigate any network or configuration issues that might affect communication with Vault.
"},{"location":"monitoring/alertmanager/#14-token-expiration-warnings","title":"1.4. Token Expiration Warnings","text":"- alert: VaultDbInjectorTokenExpirationLessThan14Days\n annotations:\n description: \"A token is nearing expiration (less than 2 weeks). Renewing or rotating the token promptly ensures continuous service operation without interruption.\"\n summary: \"Token nearing expiration in namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: vault_injector_token_expiration - time() < 1209600\n for: 90m\n labels:\n severity: warning\n\n- alert: VaultDbInjectorTokenExpirationLessThan7Days\n annotations:\n description: \"A token will expire in less than 7 days. Immediate action is required to renew or rotate the token to avoid service disruption.\"\n summary: \"Urgent: Token expiration warning for namespace `{{ $labels.exported_namespace }}`.\"\n expr: vault_injector_token_expiration - time() < 604800\n for: 5m\n labels:\n severity: critical\nResponse Actions: - For both alerts, identify the service or application using the token. - Initiate the token renewal or rotation process. - Review token policies to ensure they're aligned with security and operational requirements.
"},{"location":"monitoring/alertmanager/#15-lease-expiration-warnings","title":"1.5. Lease Expiration Warnings","text":"- alert: VaultDbInjectorLeaseExpirationLessThan4Days\n annotations:\n description: \"A lease is nearing expiration (less than 4 days). Addressing this promptly can prevent potential access issues for services relying on leased credentials or secrets.\"\n summary: \"Lease nearing expiration for namespace `{{ $labels.namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: vault_injector_lease_expiration - time() < 345600\n for: 3m\n labels:\n severity: warning\n\n- alert: VaultDbInjectorLeaseExpirationLessThan1Day\n annotations:\n description: \"A lease will expire in less than 1 day. Immediate renewal is critical to maintaining access for the dependent services.\"\n summary: \"Critical: Lease expiration imminent for namespace `{{ $labels.namespace }}`.\"\n expr: vault_injector_lease_expiration - time() < 86400\n for: 3m\n labels:\n severity: critical\nResponse Actions: - Quickly identify and renew the leases for the affected services or credentials. - Review the lease durations and renewal policies to prevent future alerts.
"},{"location":"monitoring/alertmanager/#2-conclusion","title":"2. Conclusion","text":"Monitoring VaultDb Injector with these alerts helps ensure the reliability and security of services depending on Vault for secret management and access control. Each alert is designed to provide actionable insights for maintaining operational efficiency and security compliance. Responding promptly to these alerts will mitigate potential risks and disruptions to your services.
"},{"location":"monitoring/grafana/","title":"Grafana","text":"A grafana dashboard is available on the repository and permit you to easily monitore you vault-db-injector infrastructure.
"},{"location":"monitoring/grafana/#dashboard","title":"Dashboard","text":""},{"location":"monitoring/prometheus/","title":"1. Prometheus Metrics","text":"Our application exports several Prometheus metrics for monitoring and observability purposes. Below are the details of the available metrics:
Metric Name Description Labelsvault_injector_renew_token_count_success Vault injector token renewed with success count uuid, namespace vault_injector_renew_token_count_error Vault injector token renewed with error count uuid, namespace vault_injector_renew_lease_count_success Vault injector lease renewed with success count uuid, namespace vault_injector_renew_lease_count_error Vault injector lease renewed with error count uuid, namespace vault_injector_revoke_token_count_success Vault injector token revoked with success count uuid, namespace vault_injector_revoke_token_count_error Vault injector token revoked with error count uuid, namespace vault_injector_token_expiration Vault injector expiration time for tokens uuid, namespace vault_injector_lease_expiration Vault injector expiration time for leases uuid, namespace vault_injector_token_last_renewed Last vault token successful renewal uuid, namespace vault_injector_synchronization_count_success Vault injector synchronization with success vault_injector_synchronization_count_error Vault injector synchronization with error vault_injector_pod_cleanup_count_success Vault injector PodCleanup with success vault_injector_pod_cleanup_count_error Vault injector PodCleanup with error vault_injector_last_synchronization_success Last vault token successful renewal vault_injector_orphan_ticket_created_count_success Vault injector orphan ticket created with success vault_injector_orphan_ticket_created_count_error Vault injector orphan ticket created with error vault_injector_store_data_count_success Vault injector data stored with success uuid, namespace vault_injector_store_data_count_error Vault injector data stored with error uuid, namespace vault_injector_delete_data_count_success Vault injector data delete with success uuid, namespace vault_injector_delete_data_count_error Vault injector data deleted with error uuid, namespace vault_injector_connect_vault_count_success Vault injector connect to vault with success vault_injector_connect_vault_count_error Vault injector connect to vault with error vault_injector_service_account_authorized_count Vault injector service account is authorized to assume dbRole vault_injector_service_account_denied_count Vault injector service account is not authorized to assume dbRole service_account_name, namespace, db_role, cause vault_injector_last_synchronization_duration Vault injector last duration of synchronization vault_injector_is_leader Return 1 if the vault injector is leader, else 0 lease_name vault_injector_leader_election_attempts_total Total number of attempts to acquire leadership lease_name vault_injector_leader_election_duration_seconds Duration in seconds that this instance has been the leader lease_name, leader_name, mode vault_injector_fetch_pods_success_count Count that increase when their is no error retrieving pods vault_injector_fetch_pods_error_count Count that increase when their is an error retrieving pods vault_injector_mutated_pods_error_count Count that increase when their is an error mutating pods vault_injector_mutated_pods_error_count Count that increase when their is an error mutating pods"},{"location":"fr/getting-started/build/#construction","title":"Construction","text":"Exigences pour la construction
Un Makefile est fourni pour les t\u00e2ches de construction. Les options sont les suivantes :
Pour commencer, c'est aussi simple que :
go get -u github.com/numberly/vault-db-injector\ncd $GOPATH/src/github.com/numberly/vault-db-injector\nmake setup\nmake\nThe Vault DB Injector is a Kubernetes-based application designed to dynamically generate database credentials using HashiCorp Vault and provide them as environment variables to Kubernetes Pods. The application leverages a Mutating Webhook to achieve this functionality, ensuring secure and automated management of database credentials.
"},{"location":"#key-features","title":"Key Features","text":""},{"location":"#1-dynamic-database-credential-injection","title":"1. Dynamic Database Credential Injection","text":"logrus for structured logging, supporting various log levels and JSON formatting.To get started with Vault DB Injector, refer to the Installation Guide and Configuration Guide for detailed instructions on setting up and configuring the application.
For a deeper understanding of how the application works, visit the How It Works section.
"},{"location":"#contributing","title":"Contributing","text":"We welcome contributions from the community! Please see our Contribution Guidelines for more information on how to get involved.
"},{"location":"#license","title":"License","text":"This project is licensed under the terms of the Apache-2.0 license. See the LICENSE file for details.
"},{"location":"#contact","title":"Contact","text":"For any questions or support, please reach out to our team at directly by creating an issue on the project.
"},{"location":"getting-started/build/","title":"Build","text":""},{"location":"getting-started/build/#build","title":"Build","text":"Requirements for building
A Makefile is provided for building tasks. The options are as follows
Getting started is as simple as:
go get -u github.com/numberly/vault-db-injector\ncd $GOPATH/src/github.com/numberly/vault-db-injector\nmake setup\nmake\nall-rw vault-policyK/V Vaultvault databases mountvault databases backend connectionkubernetes auth backendkubernetes auth backend rolevault databases backend roleDatabase configurationTo work correctly, the vault db injector need the following :
In this documentation, we assume that both of those following requirements are already installed.
"},{"location":"getting-started/getting-started/#2-vocabulary","title":"2. Vocabulary","text":"Here are some vocabulary that you need before starting the installation.
name description default K/V Vault Vault KVv2 path for injector state vault-injector vault policy Vault Injector policy to work properly all-rw vault databases mount Vault database engine used to connect to our database databases vault databases backend connection Vault databases backend connection where we allow role to generate credentials rw-pgsql2-pr vault databases backend role Vault databases role used to allow application on specific database test-role kubernetes auth backend Vault Backend used by Kubernetes Application to authenticate under Vault kubernetes kubernetes auth backend role Vault Backend Role used by injector to authenticate under Vault all-rw postgresql Database engine used to connecte under vault rw-pgsql2-pr database database name to connect into test"},{"location":"getting-started/getting-started/#3-vault-configuration","title":"3. Vault Configuration","text":"First, we need to configure Vault to allow the injector to generate credentials on the databases engine and to store them inside the dedicated K/V Vault.
all-rw vault-policy","text":"We are going to start by create the all-rw vault policy that the injector will use :
path \"vault-injector/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"vault-injector/data/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"vault-injector/metadata/*\" {\n capabilities = [\"read\", \"list\", \"update\", \"create\", \"delete\", \"sudo\"]\n}\npath \"rw-pgsql2-pr/creds/*\" {\n capabilities = [\"read\"]\n}\npath \"auth/kubernetes/role/*\" {\n capabilities = [\"read\"]\n}\npath \"sys/leases/renew\" {\n capabilities = [\"create\"]\n}\npath \"auth/token/renew-self\" {\n capabilities = [\"create\"]\n}\npath \"auth/token/renew\" {\n capabilities = [\"create\", \"update\"]\n}\npath \"auth/token/revoke\" {\n capabilities = [\"create\", \"update\"]\n}\npath \"auth/token/create\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}\npath \"auth/token/create-orphan\" {\n capabilities = [\"create\", \"update\", \"read\", \"sudo\"]\n}\npath \"auth/token/revoke-orphan\" {\n capabilities = [\"create\", \"update\", \"sudo\"]\n}\nK/V Vault","text":"We need to create the K/V Vault in version v2 named vault-injector.
You can use the following documentation : vault-injector
"},{"location":"getting-started/getting-started/#33-create-vault-databases-mount","title":"3.3. Createvault databases mount","text":"We need to create a vault databases mount engine named databases.
Here is a terraform example :
resource \"vault_mount\" \"databases\" {\n path = \"databases\"\n type = \"database\"\n description = \"databases authentication automation\"\n max_lease_ttl_seconds = \"31536000\"\n}\nvault databases backend connection","text":"We need to create a vault databases backend connection on the vault databases engine. As you can see below, we allow the test-role role.
Here is a terraform example :
resource \"vault_database_secret_backend_connection\" \"pgsql2\" {\n backend = vault_mount.databases.path\n name = \"pgsql2\"\n allowed_roles = [\n \"test-role\",\n ]\n\n postgresql {\n connection_url = \"postgres://{{username}}:{{password}}@rw-pgsql2-pr:5432/postgres?sslmode=verify-full\"\n username = \"postgres\"\n password = \"my-password\"\n username_template = \"{{.RoleName}}-{{unix_time}}-{{random 8}}\"\n }\n}\nkubernetes auth backend","text":"We need to create a kubernetes auth backend to allow serviceAccount to connect under Vault.
You can use the following documentation : kubernetes
"},{"location":"getting-started/getting-started/#36-create-a-kubernetes-auth-backend-role","title":"3.6. Create akubernetes auth backend role","text":"We need to create a kubernetes auth backend role to allow the service account of the vault-db-injector to connect under Vault.
Here is a terraform example :
resource \"vault_kubernetes_auth_backend_role\" \"all_rw\" {\n backend = \"kubernetes\"\n role_name = \"all-rw\"\n bound_service_account_names = [\"vault-db-injector\"]\n bound_service_account_namespaces = [\"vault-db-injector\"]\n token_ttl = 3600\n token_policies = [vault_policy.all_rw.name] # remember the one created before\n token_bound_cidrs = [\"10.17.0.0/16\"] # Your pod CIDR\n}\nWe need to create a kubernetes auth backend role to allow the service account of our application to get generated credentials from vault Here is a terraform example :
resource \"vault_policy\" \"policy\" {\n provider = vault.main\n name = var.service_account\n\n policy = <<EOT\npath \"pgsql2/creds/test-role\" {\n capabilities = [\"read\"]\n}\nEOT\n}\n\nresource \"vault_kubernetes_auth_backend_role\" \"role\" {\n provider = vault.main\n backend = kubernetes\n role_name = \"test\"\n bound_service_account_names = [\"test\"]\n bound_service_account_namespaces = [\"test\"]\n token_ttl = 3600\n token_policies = [vault_policy.policy.name]\n token_bound_cidrs = [\"10.17.0.0/16\"]\n}\nvault databases backend role","text":"We need to create a vault databases backend role to allow our application to consume vault databases backend connection
Here is a terraform example :
resource \"vault_database_secret_backend_role\" \"role\" {\n provider = vault.main\n backend = \"databases\"\n name = test-role\n db_name = test\n default_ttl = 3600\n creation_statements = [\n \"CREATE ROLE \\\"{{name}}\\\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \\\"test\\\";\",\n \"ALTER ROLE \\\"{{name}}\\\" SET ROLE \\\"test\\\";\",\n ]\n revocation_statements = [\n \"DROP ROLE \\\"{{name}}\\\";\"\n ]\n}\nDatabase configuration","text":"You need to create you database under postgres, here is an example for postgres :
CREATE DATABASE test;\nCREATE ROLE test;\nrevoke all on database test from public cascade;\ngrant connect on database test to test;\n\\c test\ngrant create, usage on schema public to test;\ngrant temporary on database test to test;\nALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO test;\nREVOKE ALL ON pg_user FROM public;\nREVOKE ALL ON pg_roles FROM public;\nREVOKE ALL ON pg_group FROM public;\nREVOKE ALL ON pg_authid FROM public;\nREVOKE ALL ON pg_auth_members FROM public;\nREVOKE ALL ON pg_database FROM public;\nREVOKE ALL ON pg_tablespace FROM public;\nREVOKE ALL ON pg_settings FROM public;\nNow that you have a vault correctly configured and a database ready to be used, we can deploy our vault-db-injector application :
For this, its quit easy, you just need to use the help chart.
kubectl create namespace vault-db-injector\nhelm upgrade --install vault-db-injector . --namespace vault-db-injector\nWhen everything is Okay, you should have something like this :
NAME READY STATUS RESTARTS AGE\nvault-db-injector-7f74977b7c-88vvg 1/1 Running 0 29s\nvault-db-injector-7f74977b7c-rq6mt 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-77skb 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-96zz4 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-tdp4r 1/1 Running 0 29s\nvault-db-injector-renewer-6496b84df-wpd8x 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-2m5qp 1/1 Running 0 28s\nvault-db-injector-revoker-7965857f75-5msv6 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-n29wp 1/1 Running 0 29s\nvault-db-injector-revoker-7965857f75-th9vs 1/1 Running 0 28s\nHere is an example application deploy on the namespace test that will connect to our database test with the service account test.
apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: test\n namespace: test\n---\napiVersion: v1\nkind: Pod\nmetadata:\n annotations:\n db-creds-injector.numberly.io/test.env-key-uri: POSTGRES_URL\n db-creds-injector.numberly.io/test.template: postgresql://@rw-pgsql2-pr:5432/test?sslmode=require\n db-creds-injector.numberly.io/test.mode: uri\n db-creds-injector.numberly.io/test.role: test-role\n labels:\n client: numberly\n vault-db-injector: \"true\"\n name: test\n namespace: test\nspec:\n containers:\n - image: postgres:15.5\n command: [\"/bin/bash\", \"-c\"]\n args: [\"sleep 3000000\"]\n name: test\n resources: {}\n securityContext:\n allowPrivilegeEscalation: false\n capabilities:\n drop:\n - ALL\n readOnlyRootFilesystem: true\n runAsNonRoot: true\n runAsUser: 65534\n dnsPolicy: ClusterFirst\n restartPolicy: Always\n serviceAccount: test\n serviceAccountName: test\nTo deploy it :
kubectl apply -f test.yaml\nWait until your application is ready and you should be able to exec inside the pods and connect to the database.
"},{"location":"how-it-works/configuration/","title":"Configuration","text":"* 1. [Configuration](#Configuration)\n * 1.1. [Mode Injector](#ModeInjector)\n * 1.2. [Mode token-renewer](#Modetoken-renewer)\n * 1.3. [Mode token-renewer](#Modetoken-renewer-1)\nHere is the configuration for Vault Injector:
"},{"location":"how-it-works/configuration/#11-mode-injector","title":"1.1. Mode Injector","text":"The Injector mode is basic one that will handle all api-server request and handle all requests to vault to generate credentials to our DB The config file path can be parse by adding the path with : - \"--config=/injector/config.yaml\"
certFile: /tls/tls.crt\nkeyFile: /tls/tls.key\nvaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: injector\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nThe Renewer one is a process that will run every hour and validate that all orphan token won't expire before pod is deleted The config file path can be parse by adding the path with : - \"--config=/renewer/config.yaml\"
vaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: renewer\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\nSyncTTLSecond: 300\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nThe Revoker one is a process that is going to watch pod deletion Kubernetes events filtered with the label vault-db-injector: true and will revoke token attached to the pod when it is deleted The config file path can be parse by adding the path with : - \"--config=/revoker/config.yaml\"
vaultAddress: https://vault1.tld:8200\nvaultAuthPath: pgsql2-dv-kubernetes1-dv-par5\nlogLevel: info\nkubeRole: all-rw\ntokenTTL: 768h\nvaultSecretName: vault-injector\nvaultSecretPrefix: kubernetes1-dv-par5\nmode: revoker\nsentry: true\nsentryDsn: https://my-sentry-url@sentry.tld/660\ninjectorLabel: vault-db-injector\ndefaultEngine: databases\nKey File: pkg/healthcheck/healthcheck.go
Health Checks are a crucial feature that monitors the application's status and readiness, ensuring it is functioning correctly and is ready to handle requests. This feature provides endpoints that external systems can query to check the health and readiness of the application.
"},{"location":"how-it-works/healthcheck/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application exposes HTTP endpoints for health (/healthz) and readiness (/readyz). These endpoints provide information about the application's operational status.
Regular Monitoring:
The health check service regularly monitors the internal state of the application and updates the health and readiness status accordingly.
Integration with Kubernetes:
Health checks ensure that the application is running correctly and can handle incoming requests. This reliability is critical for maintaining user trust and satisfaction.
Proactive Issue Detection:
By regularly monitoring the application's status, health checks can detect issues early, allowing for proactive resolution before they impact users.
Kubernetes Compatibility:
The Vault DB Injector is a Go program that is design to retrieve databases credentials from Hashicorp Vault, it use Kubernetes Mutating Webhook to intercept pod creation activated with a label and configured with annotations. After the credentials are provided, it will store them in a specific Vault KV and will handle the lifecycle of them such as : - Renew them periodically - Revoke them after the pod is deleted
"},{"location":"how-it-works/how-it-work/#11-mechanism","title":"1.1. Mechanism","text":""},{"location":"how-it-works/how-it-work/#111-vault-injector","title":"1.1.1. Vault Injector:","text":"Their is also a fonctionnality that permit to rotate token directly from the injector. Objectif is to keep token as secure as possible. For this purpose, we have made the choice to store the token directly on Vault in a specific KV. We provide a generated random UUID to every pod which will be use as a unique identifier. As long as the pod is alive, the injector will rotate his token 5 minutes by default. If the pods is deleted, the revoker will revoke the token and the lease and the renewer which is design to keep the state will also delete the secret in the KV.
If the injector reboot or fail for any reason, the token can stil be renewed manually or by restarting the injector correctly. He can retrieve all token that was created previously. The renewer will renew all tokens and leases every 5 minutes by default.
We are using periodic token for this purpose which need absolutely \"sudo\" policy. Periodic token permit to add a max_ttl and to have infinite token that can be renewed until the pod is deleted. Token TTL permit to fix the max_ttl of the token which mean : \"How many time my token can belong without been renewed.\" By default, if the token is not renewed, hes timelife is 32 days, that mean, if the injector fail, you have 31 days and 23 hours before all your token will expire, which i think is enough to understand why and repair it to a working state.
With this setup, the advantage is that now, vault is the only real SPOF. The Injector in renewer mode will only renew TOKEN and LEASE and cleanup KV/STORE that has been created by the Injector. The Injector in revoker mode will only revoke TOKEN when a pod has been deleted If the revoker fail to revoke a Token, the renewer can do it periodically every 5 minutes (This will be probably removed in future version.)
"},{"location":"how-it-works/how-it-work/#14-authentication","title":"1.4. Authentication","text":"It uses a service account and the Kubernetes mount point to retrieve and generate its information. It then sends the following information to the application:
It will store on a specific vault KV/Store the following :
They are two kind of token that are created in the lifecycle of the Vault injector :
kubeRole: in the yaml configvaultSecretName: vault-injector and prefix : vaultSecretPrefix: kubernetes1-dv-par5db-creds-injector.numberly.io/role: and db-creds-injector.numberly.io/dbname.role:tokenTTL: 768hvault-db-injector: \"true\", annotations db-creds-injector.numberly.io/cluster: & db-creds-injector.numberly.io/role: and a serviceaccountLeaseId, TokenId, namespace in a folder named with the pod UUIDThe available annotations can be declare like this :
db-creds-injector.numberly.io/cluster, [OPTIONAL] default to databases which is the database engine to usedb-creds-injector.numberly.io/role, [MANDATORY] Role to be use to get db credentialsdb-creds-injector.numberly.io/dbname.role, [OPTIONAL] Role to be use to get db credentials for this specific databasedb-creds-injector.numberly.io/dbname.env-key-dbuser, [OPTIONAL] overwrite DB user env variable, default to DBUSERdb-creds-injector.numberly.io/dbname.env-key-dbpassword, [OPTIONAL] overwrite DB password env variable, default to DBPASSWORDdb-creds-injector.numberly.io/dbname.mode, [MANDATORY] the mode for the injector, default to classic.# To be added in the pod spec\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.env-key-dbpassword: DB_PASSWORD\n db-creds-injector.numberly.io/dbname.env-key-dbuser: DB_USER\n db-creds-injector.numberly.io/dbname.role: db-role # the one created from vault with terraform\n db-creds-injector.numberly.io/dbname.mode: classic\nlabels:\n vault-db-injector: \"true\"\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.template: postgres://@postgres-server.tld:5432/dbname?sslmode=require\n db-creds-injector.numberly.io/dbname.role: db-role\n db-creds-injector.numberly.io/dbname.env-key-uri: POSTGRES_URL\n db-creds-injector.numberly.io/dbname.mode: uri\nlabels:\n vault-db-injector: \"true\"\nannotations:\n db-creds-injector.numberly.io/cluster: databases\n db-creds-injector.numberly.io/dbname.env-key-dbpassword: DB_PASSWORD\n db-creds-injector.numberly.io/dbname.env-key-dbuser: DB_USER\n db-creds-injector.numberly.io/dbname.role: db-role # the one created from vault with terraform\n db-creds-injector.numberly.io/dbname.mode: classic\n db-creds-injector.numberly.io/other_dbname.template: postgres://@postgres-server.tld:5432/dbname?sslmode=require\n db-creds-injector.numberly.io/other_dbname.role: another-vault-role\n db-creds-injector.numberly.io/other_dbname.env-key-uri: POSTGRES_URL,ANOTHER_ENV\n db-creds-injector.numberly.io/other_dbname.mode: uri\nlabels:\n vault-db-injector: \"true\"\ndbname and other_dbname."},{"location":"how-it-works/injector/","title":"Injector","text":"Key File: pkg/injector/injector.go
The injector is responsible for injecting database credentials into Kubernetes Pods using a Mutating Admission Webhook.
When a Pod is created, the webhook intercepts the request and modifies the Pod specification to include environment variables with the credentials.
Credential Injection:
By automating the injection of credentials, the injector ensures that Pods have the necessary credentials without storing them statically, enhancing security.
Transparent Operation:
The Mutating Admission Webhook operates transparently, modifying Pod specifications on-the-fly without manual intervention.
Security:
Key Files: pkg/k8s/connect.go, pkg/k8s/pod_utils.go, pkg/k8s/parse_annotations.go
Kubernetes Integration is a fundamental feature that enables the application to interact with the Kubernetes API. This integration allows the application to manage and manipulate Kubernetes resources, facilitating tasks such as credential injection, pod management, and more.
"},{"location":"how-it-works/kubernetes/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application initializes a Kubernetes client that can interact with the Kubernetes API. This client is used to perform various operations such as reading pod annotations, accessing secrets, and more.
Service Account Token Retrieval:
The application retrieves the service account token from the Kubernetes environment. This token is used for authenticating API requests made by the application.
CA Certificate Retrieval:
The application retrieves the Kubernetes CA certificate. This certificate is used to establish secure communication with the Kubernetes API server.
Pod Annotation Parsing:
By integrating with the Kubernetes API, the application can perform a wide range of operations directly within the Kubernetes cluster. This seamless integration simplifies the management of Kubernetes resources.
Security:
Using service account tokens and CA certificates ensures that all interactions with the Kubernetes API are secure and authenticated. This enhances the security of the application's operations.
Dynamic Configuration:
Each injector annotation are read by the Injector pod and permit to configure properly how Database Credetials need to be handled.
"},{"location":"how-it-works/leaderelection/","title":"Leader Election","text":"Key File: pkg/leadership/leadership.go
Leader Election is a mechanism that ensures high availability and fault tolerance by designating a single instance of the application to perform critical tasks at any given time. This feature is particularly important in distributed systems where multiple instances of an application might be running concurrently.
"},{"location":"how-it-works/leaderelection/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"Leader Election uses Kubernetes resource locks to manage which instance of the application is the current leader. The application instances compete for a lock, and the one that acquires it becomes the leader.
Performing Critical Tasks:
The leader is responsible for performing tasks that should not be duplicated. These tasks could include credential renewal, revocation, and other maintenance activities that need to be managed centrally.
Failover Handling:
By ensuring that there is always one active leader performing critical tasks, the system can provide high availability and resilience. If the current leader fails, another instance takes over, maintaining operational continuity.
Fault Tolerance:
Leader Election allows the system to handle failures gracefully. The loss of a leader does not disrupt critical operations because another instance will quickly assume leadership.
Efficient Resource Management:
This feature is essential for applications deployed in Kubernetes clusters, where ensuring that critical tasks are managed by a single, designated instance enhances both reliability and performance.
"},{"location":"how-it-works/renewer/","title":"Renewer","text":"Key File: pkg/renewer/renewer.go
The Renewer is a crucial component responsible for ensuring that database credentials remain valid over time. It periodically checks the status of the credentials and renews them before they expire. This process is critical in environments where credentials have a limited lifespan and need to be refreshed to maintain access.
"},{"location":"how-it-works/renewer/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The Renewer periodically checks the expiration status of the current database credentials. It ensures that credentials are always valid and do not reach their expiration date unnoticed.
Credential Renewal:
By automating the renewal of credentials, the Renewer eliminates the need for manual intervention, reducing administrative overhead and the risk of human error.
Continuous Availability:
Ensures that applications have uninterrupted access to the database by renewing credentials well before they expire, thus preventing potential downtime.
Security:
This feature is particularly useful for maintaining secure and continuous database access in dynamic cloud environments, where credential rotation is a best practice for security.
"},{"location":"how-it-works/revoker/","title":"Revoker","text":"Key File: pkg/revoker/revoker.go
The Revoker is responsible for ensuring that unused or stale database credentials are properly revoked. This process is essential for maintaining the security of your system by ensuring that credentials that are no longer needed are not left active.
"},{"location":"how-it-works/revoker/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The Revoker identifies credentials that are no longer in use or have become stale. This could be because the application instance that was using them has been terminated, or the credentials have exceeded their intended lifespan.
Revoking Credentials:
Once stale or unused credentials are identified, the Revoker sends a request to HashiCorp Vault to revoke these credentials. This action ensures that these credentials can no longer be used to access the database.
Updating System State:
The Revoker updates the system's state to reflect that certain credentials have been revoked. This helps in maintaining an accurate and secure overview of active and inactive credentials.
Automatic revocation:
DELETE event on pod so it can revoke the credentials direcly after the pod is Deleted, this permit to be sure credentials are not revoked before the pods completely deleted.By revoking unused or stale credentials, the Revoker minimizes the risk of unauthorized access. This is crucial in environments where security is a top priority.
Reduced Attack Surface:
Removing credentials that are no longer needed reduces the number of potential entry points for attackers, thus lowering the overall risk profile of the system.
Compliance and Audit:
This feature is particularly important in dynamic environments where resources and their associated credentials are frequently created and destroyed. It ensures that credentials do not linger beyond their useful life, thereby maintaining a high standard of security.
"},{"location":"how-it-works/vault/","title":"Vault Integration","text":"Key Files: pkg/vault/handle_token.go, pkg/vault/vault.go
Vault Integration is a crucial feature that handles interactions with HashiCorp Vault for generating and managing database credentials. This integration ensures that the application can securely request and use credentials from Vault.
"},{"location":"how-it-works/vault/#2-key-responsibilities","title":"2. Key Responsibilities:","text":"The application initializes a Vault client using the provided configuration. This client is used to authenticate and communicate with the Vault server.
Authentication with Vault:
The application authenticates with Vault using a Kubernetes authentication method
Requesting Credentials:
When the application needs database credentials, it sends a request to Vault. Vault generates the credentials and returns them to the application.
Handling Tokens:
By integrating with Vault, the application can securely manage database credentials. Vault ensures that credentials are generated securely and rotated regularly, enhancing overall security.
Dynamic Credential Generation:
Credentials are generated on demand, which means they are always fresh and have limited lifespans. This dynamic generation reduces the risk of credential compromise.
Centralized Secret Management:
This configuration defines a set of alerts for monitoring the VaultDb Injector within a Kubernetes environment. Each alert is designed to notify the team of potential issues that could impact the availability, security, or functionality of the services relying on Vault for secret management.
"},{"location":"monitoring/alertmanager/#1-alerts-configuration","title":"1. Alerts Configuration","text":""},{"location":"monitoring/alertmanager/#11-service-account-denied","title":"1.1. Service Account Denied","text":"- alert: VaultDbInjectorServiceAccountDenied\n annotations:\n description: \"Service Account (SA) `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied access to db_role `{{ $labels.db_role }}` due to `{{ $labels.cause }}` on cluster `{{ $labels.k8s_cluster }}`. Immediate investigation is recommended to ensure proper access controls and service configurations.\"\n summary: \"Service Account `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied by the injector.\"\n expr: increase(vault_injector_service_account_denied_count{}[2m]) > 0\n for: 1m\n labels:\n severity: critical\nResponse Actions: - Verify the service account permissions and roles. - Check the db_role configurations to ensure they are correctly set up. - Investigate the cause for denial to prevent future occurrences.
"},{"location":"monitoring/alertmanager/#12-token-renewal-failure","title":"1.2. Token Renewal Failure","text":"- alert: VaultDbInjectorFailToRenewToken\n annotations:\n description: \"VaultDbInjector encountered an error while attempting to renew a token. This might affect the continuous operation of dependent services. Check for errors and ensure the token renewal process is configured correctly.\"\n summary: \"VaultDbInjector token renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: increase(vault_injector_renew_token_count_error{}[2m]) > 0\n for: 1m\n labels:\n severity: warning\nResponse Actions: - Review the injector logs for errors related to token renewal. - Ensure the Vault policies allow for token renewal by the injector. - Check for network issues that might prevent the injector from communicating with Vault.
"},{"location":"monitoring/alertmanager/#13-lease-renewal-failure","title":"1.3. Lease Renewal Failure","text":"- alert: VaultDbInjectorFailToRenewLease\n annotations:\n description: \"VaultDbInjector encountered an error while attempting to renew a lease. Similar to token renewal failures, this can disrupt service operations if not addressed.\"\n summary: \"VaultDbInjector lease renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: increase(vault_injector_renew_lease_count_error{}[2m]) > 0\n for: 1m\n labels:\n severity: warning\nResponse Actions: - Inspect the injector logs for specific errors related to lease renewal. - Confirm that the Vault configuration allows the injector to renew leases. - Investigate any network or configuration issues that might affect communication with Vault.
"},{"location":"monitoring/alertmanager/#14-token-expiration-warnings","title":"1.4. Token Expiration Warnings","text":"- alert: VaultDbInjectorTokenExpirationLessThan14Days\n annotations:\n description: \"A token is nearing expiration (less than 2 weeks). Renewing or rotating the token promptly ensures continuous service operation without interruption.\"\n summary: \"Token nearing expiration in namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: vault_injector_token_expiration - time() < 1209600\n for: 90m\n labels:\n severity: warning\n\n- alert: VaultDbInjectorTokenExpirationLessThan7Days\n annotations:\n description: \"A token will expire in less than 7 days. Immediate action is required to renew or rotate the token to avoid service disruption.\"\n summary: \"Urgent: Token expiration warning for namespace `{{ $labels.exported_namespace }}`.\"\n expr: vault_injector_token_expiration - time() < 604800\n for: 5m\n labels:\n severity: critical\nResponse Actions: - For both alerts, identify the service or application using the token. - Initiate the token renewal or rotation process. - Review token policies to ensure they're aligned with security and operational requirements.
"},{"location":"monitoring/alertmanager/#15-lease-expiration-warnings","title":"1.5. Lease Expiration Warnings","text":"- alert: VaultDbInjectorLeaseExpirationLessThan4Days\n annotations:\n description: \"A lease is nearing expiration (less than 4 days). Addressing this promptly can prevent potential access issues for services relying on leased credentials or secrets.\"\n summary: \"Lease nearing expiration for namespace `{{ $labels.namespace }}` on cluster `{{ $labels.k8s_cluster }}`.\"\n expr: vault_injector_lease_expiration - time() < 345600\n for: 3m\n labels:\n severity: warning\n\n- alert: VaultDbInjectorLeaseExpirationLessThan1Day\n annotations:\n description: \"A lease will expire in less than 1 day. Immediate renewal is critical to maintaining access for the dependent services.\"\n summary: \"Critical: Lease expiration imminent for namespace `{{ $labels.namespace }}`.\"\n expr: vault_injector_lease_expiration - time() < 86400\n for: 3m\n labels:\n severity: critical\nResponse Actions: - Quickly identify and renew the leases for the affected services or credentials. - Review the lease durations and renewal policies to prevent future alerts.
"},{"location":"monitoring/alertmanager/#2-conclusion","title":"2. Conclusion","text":"Monitoring VaultDb Injector with these alerts helps ensure the reliability and security of services depending on Vault for secret management and access control. Each alert is designed to provide actionable insights for maintaining operational efficiency and security compliance. Responding promptly to these alerts will mitigate potential risks and disruptions to your services.
"},{"location":"monitoring/grafana/","title":"Grafana","text":"A grafana dashboard is available on the repository and permit you to easily monitore you vault-db-injector infrastructure.
"},{"location":"monitoring/grafana/#dashboard","title":"Dashboard","text":""},{"location":"monitoring/prometheus/","title":"1. Prometheus Metrics","text":"Our application exports several Prometheus metrics for monitoring and observability purposes. Below are the details of the available metrics:
Metric Name Description Labelsvault_injector_renew_token_count_success Vault injector token renewed with success count uuid, namespace vault_injector_renew_token_count_error Vault injector token renewed with error count uuid, namespace vault_injector_renew_lease_count_success Vault injector lease renewed with success count uuid, namespace vault_injector_renew_lease_count_error Vault injector lease renewed with error count uuid, namespace vault_injector_revoke_token_count_success Vault injector token revoked with success count uuid, namespace vault_injector_revoke_token_count_error Vault injector token revoked with error count uuid, namespace vault_injector_token_expiration Vault injector expiration time for tokens uuid, namespace vault_injector_lease_expiration Vault injector expiration time for leases uuid, namespace vault_injector_token_last_renewed Last vault token successful renewal uuid, namespace vault_injector_synchronization_count_success Vault injector synchronization with success vault_injector_synchronization_count_error Vault injector synchronization with error vault_injector_pod_cleanup_count_success Vault injector PodCleanup with success vault_injector_pod_cleanup_count_error Vault injector PodCleanup with error vault_injector_last_synchronization_success Last vault token successful renewal vault_injector_orphan_ticket_created_count_success Vault injector orphan ticket created with success vault_injector_orphan_ticket_created_count_error Vault injector orphan ticket created with error vault_injector_store_data_count_success Vault injector data stored with success uuid, namespace vault_injector_store_data_count_error Vault injector data stored with error uuid, namespace vault_injector_delete_data_count_success Vault injector data delete with success uuid, namespace vault_injector_delete_data_count_error Vault injector data deleted with error uuid, namespace vault_injector_connect_vault_count_success Vault injector connect to vault with success vault_injector_connect_vault_count_error Vault injector connect to vault with error vault_injector_service_account_authorized_count Vault injector service account is authorized to assume dbRole vault_injector_service_account_denied_count Vault injector service account is not authorized to assume dbRole service_account_name, namespace, db_role, cause vault_injector_last_synchronization_duration Vault injector last duration of synchronization vault_injector_is_leader Return 1 if the vault injector is leader, else 0 lease_name vault_injector_leader_election_attempts_total Total number of attempts to acquire leadership lease_name vault_injector_leader_election_duration_seconds Duration in seconds that this instance has been the leader lease_name, leader_name, mode vault_injector_fetch_pods_success_count Count that increase when their is no error retrieving pods vault_injector_fetch_pods_error_count Count that increase when their is an error retrieving pods vault_injector_mutated_pods_error_count Count that increase when their is an error mutating pods vault_injector_mutated_pods_error_count Count that increase when their is an error mutating pods"},{"location":"fr/getting-started/build/#construction","title":"Construction","text":"Exigences pour la construction
Un Makefile est fourni pour les t\u00e2ches de construction. Les options sont les suivantes :
Pour commencer, c'est aussi simple que :
go get -u github.com/numberly/vault-db-injector\ncd $GOPATH/src/github.com/numberly/vault-db-injector\nmake setup\nmake\n