Merge pull request #195 from Phantom-Intruder/fluentbit-sidecar

Fluentbit sidecar

Logging101/filebeat-sidecar.md

@@ -109,6 +109,12 @@ Note the name of the pod, and use it in the below command:
 kubectl describe pod <POD_NAME> --watch
 ```
 
-You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`.  When the container running the sleep command goes to a `successful` state, the container running filebeat should immediately. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves.
+You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`.  When the container running the sleep command goes to a `successful` state, the container running filebeat should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves.
 
-This brings us to the end of this section on logging with filebeat sidecars. You can use the same concept with similar tools such as fluentd if you plan to scale up your jobs/logs massively. Just make sure that there are no bottlenecks in any other points such as logstash and elasticsearch.
+This brings us to the end of this section on logging with filebeat sidecars. You can use the same concept with similar tools such as fluentd if you plan to scale up your jobs/logs massively. Just make sure that there are no bottlenecks in any other points such as logstash and elasticsearch.
+
+We have already covered fluent bit, so you know that it is way more lightweight than either filebeat or fluentd. In fact, according to [benchmarks by AWS](https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/), fluent bit uses 5 times less memory compared to fluentd, and 2 times less than filebeat. So in the case where we run hundreds of jobs at the same time, it makes a lot of sense to use a logger that pushes all your log lines with as little resource consumption as possible since we will be creating a logger instance per each pod.
+
+In the next section, let's take a look at how we can use fluentbit as a sidecar container to push logs.
+
+[Next: fluentbit as a sidecar container](./fluentbit-sidecar.md)

Logging101/fluentbit-sidecar.md

@@ -0,0 +1,76 @@
+## fluentbit as a sidecar container
+
+The concept behind the fluentbit sidecar container will be basically the same as with the filebeat sidecar. The differences will be in the fluentbit conf since that will obviously use a different syntax and the way the fluentbit container will be loaded into the pod. We will continue to use the same Ubuntu job that we were using before, and the same concept of using a completion flag to tell when the container should stop will continue to apply. We will also be using the same shared volume, and we will be using a ConfigMap to load the fluent bit conf as well. Below is the fluent bit conf that matches the filebeat config that we had in the previous section:
+
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: fluentbit-configmap
+data:
+  fluent-bit.conf: |
+    [SERVICE]
+        Flush        1
+        Log_Level    info
+        Daemon       off
+
+    [INPUT]
+        Name         tail
+        Path         /data/*.log
+        Tag          mixlog
+
+    [OUTPUT]
+        Name         http
+        Match        mixlog
+        Host         logstash-logstash
+        Port         5044
+```
+
+The first five lines are already familiar to you. We then start the fluentbit config. We first have some information on the service, followed by the definition of the input. As with before, we use the tail plugin to get all the log files found in /data/ and tag them with the tag "mixlog". We then match these tagged items in the output plugin and stream the logs into the logstash service. You will notice that while filebeat natively had an input source to logstash called "beats", fluent bit does not. However, we can use "http" to do this instead. From the logstash side, you will have to change the input to point to use "http" instead of "beats", but apart from that, everything should work just fine.
+
+Now let's look at what should be done from the Kubernetes manifest side. It will be basically the same thing as what we had with filebeat, except we will use the fluent bit image. We will also be pointing the overriding config to fluent-bit.conf which will be mounted in a shared volume, the same as the filebeat yaml. Apart from that, everything will be the same.
+
+```
+- name: fluent-bit-sidecar
+  image: cr.fluentbit.io/fluent/fluent-bit:2.2.2
+  volumeMounts:
+    - name: fluent-bit-config
+      mountPath: /fluent-bit/etc/
+      readOnly: true
+    - name: shared-data
+      mountPath: /data/
+  command: ["/fluent-bit/bin/fluent-bit"]
+  args: ["-c", "/fluent-bit/etc/fluent-bit.conf & while [ ! -f /data/completion-flag ]; do sleep 1; done && exit 0"]
+  ```
+
+Now that we have covered both areas that need to be changed, let's go ahead and give this a test run. First off, deploy the ConfigMap:
+
+```
+kubectl apply -f fluentbit-configmap.yaml
+```
+
+Next, apply the deployment.yaml:
+
+```
+kubectl apply -f non-parallel-job.yml
+```
+
+Now let's observe the containers in the same way we did with the filebeat sidecars.
+
+```
+kubectl get po
+```
+
+Note the name of the pod, and use it in the below command:
+
+```
+kubectl describe pod <POD_NAME> --watch
+```
+
+You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`.  When the container running the sleep command goes to a `successful` state, the container running fluentbit should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves.
+
+## Conclusion
+
+This brings us to the end of the section on running fluent bit as a sidecar container. Now, you may be asking the question: if fluentbit does the same things as filebeat with a much smaller resource footprint, why use filebeat at all? The answer to this is features. For example, logstash supports the Beats protocol natively. However, it does not do this for fluentbit. Instead, you will have to use HTTP, which might mess up the output that is presented in Kibana. Larger loggers such as fluentd support in-built grok parsing which fluentbit doesn't. Instead, you will have to push logs from fluent bit to fluentd (or logstash as we do here), which adds another resource that acts as a mediator. Since logstash also handles buffering so that elasticsearch doesn't get overwhelmed, this isn't a particularly terrible idea. Additionally, you might notice that fluent bit does not have tools like bash or sh, which means that if you want to look inside the fluent bit container for some reason, you won't be able to do so.
+
+So there is a trade-off and you will have to consider what is best for your use case.