Flow proposal: remote.kubernetes.configmap and remote.kubernetes.secret

[rfratto]

To support people who wish to migrate off of Grafana Agent Operator onto Flow, we can add native support for components to read values from ConfigMaps and Secrets.

This can be used to create a tool which discovers a GrafanaAgent resource hierarchy, and converts it to a matching Flow configuration as closely as possible.

The remote.kubernetes.configmap and remote.kubernetes.secret components can be used specifically to create a mapping for CRDs like MetricsInstance, where it can define a remote_write endpoint whose credentials are synchronized with a Kubernetes Secret.

remote.kubernetes.configmap

remote.kubernetes.configmap will watch a specific ConfigMap in Kubernetes, exporting all keys within the ConfigMap:

remote.kubernetes.configmap "example" {
  namespace = "CONFIGMAP_NAMESPACE"
  name      = "CONFIGMAP_NAME"

  client {
    // Omitted: Kubernetes client settings 
  }
}

If the namespace argument is empty, the value will be inferred from the namespace the agent is running in.

remote.kubernetes.configmap exports a value called data which is a map of data fields.

For example, for the given ConfigMap:

apiVersion: v1 
kind: ConfigMap 
metadata:
  namespace: default
  name: example-configmap
data:
  foo: bar
  hello: world

the data export would be an object matching:

{
  foo   = "bar",
  hello = "world",
}

remote.kubernetes.secret

remote.kubernetes.secret will watch a specific Secret in Kubernetes, exporting all keys within the Secret:

remote.kubernetes.secret "example" {
  namespace = "SECRET_NAMESPACE"
  name      = "SECRET_NAME"

  client {
    // Omitted: Kubernetes client settings 
  }
}

remote.kubernetes.secret exports a value called data which is a map of data fields. base64 values retrieved from the Kubernetes API are decoded prior to exporting. Each value in the exported data value is a secret.

If the namespace argument is empty, the value will be inferred from the namespace the agent is running in.

For example, for the given Secret:

apiVersion: v1 
kind: Secret 
metadata:
  namespace: default
  name: example-secret
data:
  foo: YmFy       # base64 encoding of "bar" 
  hello: d29ybGQ= # base64 encoding of "world"  

the data export would be an object matching:

{
  foo   = secret /* "bar" */,
  hello = secret /* "world" */,
}

The exported values in the map are secrets, allowing them to only be passed to other arguments expecting secrets, and may not be implicitly converted back to strings.

Considerations

To reduce the load on the Kubernetes API server, these components should probably not start informers, and should rather directly watch just the configured configmap or secret.

Alternatives

The alternative to these components is to predefine the list of secrets used by the Flow deployment and mount them to the Grafana Agent pods.

These components are functionally equivalent to mounting, but makes the migration from the Operator to be slightly more convenient for the user.

[rfratto]

Closed by #4854.