Providing secrets as files in shared volume

[InbalZilberman]

Introduction

It has been shown by CyberArk Labs and other security experts that files are more secure than environment variables. Moreover, they are common to use when it comes to keeping secrets.

Providing the secrets as files means also keeping the secrets only in "CyberArk" hands and not in other places (like K8s secrets) enabling a more direct path from the secret source to the application making it more secure.

Saving secrets as files sets the ground to rotation in the future (although it is out of scope in this feature).

In this feature we will provide the ability for apps to consume Conjur/DAP secrets as files hence improve the delivery of secrets's security while keeping the consumption of the secrets by the apps common and native.

The secrets will be saved as files in a memory based shared volume and will be provided before the application container is running as a one timer using init container.

Only init container is in scope.

The feature will be supported as OSS (like other phases) as community level.

Out of scope - There is no need to support multiple applications while delivery as files

Current mechanism

Current K8s secret provider saves the secrets in K8s secrets which is considered not as secure as CyberArk/DAP and is available to more than one pod. Once the secrets are in K8s secrets the RBAC is spread out to actually the operation team and not the security team.

Use cases

AS an application developer in K8s later on Liz the operator

I would like to control the secrets the applications is expecting & their location

So That the application can run

AS an application developer in K8s

I would like to place my secrets in files in memory based shared volume

So That no code change is needed for integration with Conjur\DAP

AS an application in K8s

I would like to fetch my secrets from files in memory based shared volume

So That I can use them in a common and a secure way (with no code change of the application before using secret provider)

The secrets are expected to be provided in advance before the app is initialized.

Process logic

1. Conjur admin, Martin, is creating Conjur policies with
   1. Secrets needed to be consumed by the application.
   2. An identity for the application's pod in which the secret provider relays as well- as the application pod is being identified using our K8s authentication the host name should follow the authentication identities guidelines and the application Identity guidelines.
   3. Permissions to the application on the secrets
      For example:

- !policy
  id: conjur/authn-k8s/my_cluster_auth
  body:
  - !webservice
 
  - !layer apps
 
  - !permit
    role: !layer apps
    privilege: [ read, authenticate ]
    resource: !webservice



- !policy
  id: allowed_apps
  annotations:
    description: Apps and services in cluster.
  body:
 
  - !layer k8sApps
  - &apps
    - !host
      id: namespace-based-app
      annotations:
        authn-k8s/namespace: my_namespace
        authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s 
 
  - !grant
    role: !layer k8sApps
    members: *apps
 
  - &variables
    - !variable password
    - !variable username
 
 
  - !permit
    role: !layer k8sApps
    privileges: [ read, execute ]
    resources: *variables

2. Liz/Martin set a config map
   The config map will have groups of settings each group holds a map similar to conjur_map we have today.

   2 alternatives are presented here one with folder path as group name and the other with a must have key called "folder-path". Other : pairs will have the following structure : where the secret value of conjur variable will be saved in a file according to the folder-path.

In our flow:

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: my_config_map
  data:
    <folder-path>: |-   
    username: db_credentials/username   
    password: db_credentials/password

    <other-folder-path>: |-
    username1: other_credentials/username
    password1: other_credentials/password

  For example
  /secrets: |-   
    username: db_credentials/username   
    password: db_credentials/password


  Alternatively 

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: my_config_map
  data:
    group-name: |-   
    username: db_credentials/username   
    password: db_credentials/password
    folder-path: "/secrets"

    other-group-name: |-
    username1: other_credentials/username
    password1: other_credentials/password
    folder-path: "/mysecrets"

We need to understand if there is an issue here with special characters in the folder path that will require option 2 or that we like option 2 anyhow

Moti Cohen suggested to go with mountPath instead of folder-path if the path can be not relative than this is a great name change.

3. The operator, Liz, is creating a manifest in Openshift or K8s in which she defines a new image that can be an init container

For example:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: test-app
  name: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      serviceAccountName: < APP_SERVICE_ACCOUNT_NAME >
      containers:
      - image: < TEST_APP_DOCKER_IMAGE >
        imagePullPolicy: Always
        name: test-app
        ports:
        - containerPort: 8080
        volumeMounts:
          - mountPath: /cyberark <to be decided by dev - should match the path in secrets_folder_path>
            name: cyberark-secrets
            readOnly: true
    initContainers:
      - image: cyberark/cyberark-secret-provider-for-k8s
        imagePullPolicy: Always
        name: cyberark-secret-provider-for-k8s-file-flavor
        env:
          - name: CONTAINER_MODE
            value: init
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: CONTAINER_MODE
              value: init

            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name

            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace

            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

            - name: CONJUR_APPLIANCE_URL
              value: < CONJUR_APPLIANCE_URL >

            - name: CONJUR_AUTHN_URL
              value: < CONJUR_AUTHN_URL >

            - name: CONJUR_ACCOUNT
              value: < CONJUR_ACCOUNT >

            - name: CONJUR_AUTHN_LOGIN
              value: host/conjur/authn-k8s/<authenticator-id>/apps/namespace-based-app


            - name: CONJUR_SSL_CERTIFICATE
              valueFrom:
                configMapKeyRef:
                  name: < CONFIG_MAP_NAME >
                  key: ssl-certificate

		  - name: SECRET_DESTINATION
            value: "files"
          
                -----NO KEY ------
        volumeMounts:
          - mountPath: /cyberark/secrets
            name: cyberark-secrets
         - mountPath: /cyberark/secrets-map
            name: secrets-map-configmap
      imagePullSecrets: 
        - name: dockerpullsecret
      volumes:
        - name: cyberark-secrets
          emptyDir:
            medium: Memory
		configMap:
          name: secrets-map-configmap
          items:
            - key: my_config_map
              path: secrets-map

This new container name ,called in our example "secret provider for K8s - files flavour", will have the same env variables as the image in phase 1 had with few differences:

1. The value of SECRET_DESTINATION would be "files".
2. the memory volume should be of type memory
3. Subject to design - a new file called secrets-map will be given with a reference to the config map that was defined in previous step and stored in a mounted folder.

**Note the name of the image is a design decision

4. Liz, the operator deploys the pod.

5. The pod initialize the during the initialization

5.1 The new container "secret provider for K8s - files flavor" is authenticating to Conjur using the pod's characteristics and host identity then fetches the needed secrets from Conjur and place them in the right path as files.

In our example

mountPath in the app container is /cyberark folder path of group 1 is /secrets

variable db_credentials/username is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/secrets/username

variable db_credentials/password is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/secrets/password

folder path of group 2 is /mysecrets hence

variable other_credentials/username is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/mysecrets/username1

variable other_credentials/password is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/mysecrets/password1

If the container is an init container it conclude his role here

5.2 The app container life begin - the application fetches the secrets from the files

/cyberark/secrets/username

/cyberark/secrets/password

Out of scope -If the new container is a sidecar it perform the above steps every 5 minutes, meaning it fetches the secrets from conjur and if changed place them on to the shared volume. That way we support rotation of secr

Requirements

The functionality that is given as today should not be erased or damaged as it provide ability to provide secrets as K8s secrets.
We will support only K8s applications that are running as containers in a pod.

DAP & OSS

The secret provider should work with DAP (Enterprise edition) and Conjur (OSS)

####Security or permissions

No special security or permissions configuration should be done in order to provide the application's container/pod its secrets by k8s secret provider.

The secret provider should serve only applications under at least the same namespace.

Should the volume be readOnly?

Configuration and Ease of usage

We should try and understand if it is possible to have the same secret provider image running in these different modes.

The value of the configuration called SECRET_DESTINATION should be "files" instead of "k8s_secrets".

Deployment

Deployment mechanism should be native and think of full container life cycle like upgrade.

we should think wat guidelines we need to provide to customers who want to move from init container with K8s secrets to init container with files

Performance

The application should start running with minimum delay in time.

Lets create an avg time test where an app consume 25 vars (5 PAS accounts)

Audits

We should investigate the need for audit when writing to files.

Failure Scenarios

Failure Scenario	Error message	Expected behavior/UX
Secret cant be fetchedConjur path is not correctIt can happen because no permissions where provided to the host ofConjur-authn-client or because the path is invalid.	key can not be fetched from Conjur according to path . Error:	A log is written with the right error code and message. The log can be seen in regular cloud tools like openshift pod log, elastic search...Fail Pod
Failed to authenticate using Access token provided	Failed to authenticate using Access token provided here <>. Error: <>	proper log and fail POD
Failed to save secret to file file path can be incorrect permissions to write		proper log and fail POD

Monitoring

Monitoring Metrics Analysis

The container should be monitored using Openshift and K8s tools - probably as part of the POD.

We should take care that it is not stucked - we should do it by liveliness probe https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Logging

They should have an error code and log level

The logs should appear in Cloud tools like elasticsearch**

Supportability

We should provide guidelines and troubleshooting info if the deployment of this image fails or it fails during runtime.

Conditions of satisfaction

1. The new process works as expected: including deployment of the init container, fetching of the Conjur secrets and writing them to files, initialization of the app container with all the right values
2. The old process works as before.
3. Error flows were discussed, defined and implemented
4. The Mapping mechanism of Conjur secrets works and all end cases are handled
5. The authentication mechanism is k8s authn
6. Secrets are being fetched according to the permissions of the container's host and are save as files in the shared volume
7. Logs are being written as these actions take place.
8. LATER STAGE - Out of scope for this phase - In case of a sidecar If a secret is rotated or changed the app would get a hold of the latest and greatest in a 5 min cycle.
9. Error handling
10. Retry once before failing
11. Test on a common env of a scaled pod X3 and few namespaces were on follower namespace serve all of them.

DoD

1. Test plan written and reviewed by PO & QAA
2. Automatic integration tests written according to a test plan and passed successfully
3. UT written for all classes\functions\major logic flows and passed successfully
4. Security review was done
5. Security action items were taken
6. Performance tests were done
7. Supportability tasks were written
8. Enhance logs and supportability
9. Logs were reviewed by TW and PO
10. Fill in the configurations in this feature doc
11. Configurations were reviewed by PO .
12. Documentation HO to TW and review docs_
13. Release as community level feature

[izgeri]

References

• Functional spec (private)
• PRD (private)

Conjur server changes to consider:

• cyberark/conjur#843 - batch retrieve secrets matching variable prefix
• cyberark/conjur#2071 - batch retrieval endpoint doesn't fail when only a subset of requested secrets can be retrieved
  • Related: cyberark/conjur#1132 - rethink batch secret retrieval audit
• cyberark/conjur#1519 - Conjur has an /info endpoint
• cyberark/conjur#1520 - Conjur has a /metrics endpoint
• cyberark/conjur#2072 - Conjur has an endpoint to create authenticator instances in policy

[alexkalish]

Closing in favor of #333.