In Fluent Bit, every log record is given a tag, which defines how it is routed through the pipeline and which plugin configurations apply to it. Additionally, in the cloudwatch and cloudwatch_logs AWS output plugins, the log_stream_prefix option can be used to create CloudWatch Log Streams with the name prefix + tag.
Therefore, users commonly want to customize the tag to include kubernetes metadata. This can be accomplished using the rewrite_tag filter and kubernetes filter.
The kubernetes filter can add metadata to your records that looks like the following:
kubernetes: {
annotations: {
"kubernetes.io/psp": "eks.privileged"
},
container_hash: "<some hash>",
container_name: "myapp",
docker_id: "<some id>",
host: "ip-10-1-128-166.us-east-2.compute.internal",
labels: {
app: "myapp",
"pod-template-hash": "<some hash>"
},
namespace_name: "default",
pod_id: "198f7dd2-2270-11ea-be47-0a5d932f5920",
pod_name: "myapp-5468c5d4d7-n2swr"
}
With the rewrite tag filter, you can then customize the tag using these keys. Below is an example configuration which will set the tag to be applogs-{k8s host}.{namespace name}.{pod name}.{container name}.
[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
[FILTER]
Name rewrite_tag
Match application.*
Rule $kubernetes['namespace_name'] ^[a-zA-Z0-9-_]*$ applogs-$kubernetes['host'].$kubernetes['namespace_name'].$kubernetes['pod_name'].$kubernetes['container_name'] false
[OUTPUT]
Name cloudwatch_logs
Match applogs*
region ${AWS_REGION}
log_group_name /eks/rewrite-tag-example/application
log_stream_prefix eks-
auto_create_group true
Remember that rewrite_tag will change the tag and re-emit the records into the head of the pipeline like an input. Therefore, be very careful about creating cycles in your configuration- notice that the tag the filter creates is very different than the tag it matches.
If you want to deploy this example yourself, included is an altered version of the Amazon CloudWatch Container Insights Daemonset that uses this technique the customize the tag and log stream name. Follow the steps 1 & 2 to create a namespace and config map. Then, instead of step 3, apply the file in this example with:
kubectl apply -f fluent-bit.yaml
- rewrite_tag re-emits records at the head of the pipeline like an input, therefore, it increases the processing required for every single record, and may slow down the total throughput that Fluent Bit can handle, and can increase its memory usage.