Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Learnings from the field – Running Fluent Bit on Azure Kubernetes Service – Part 1

This is the first part of a three-part series about “Learnings from the field – Running Fluent Bit on Azure Kubernetes Service”.

Logging is one of the central aspects when operating Kubernetes. The easiest way to get started with it is by using the solution your cloud provider provides. On Azure, this is Azure Monitor Container Insights that can also be used on Google Kubernetes Engine and Amazon Elastic Kubernetes Service via Azure Arc.

When you look for a platform-agnostic approach that is also highly customizable, you probably end up with Fluent Bit. Besides running Fluent Bit on Kubernetes for your container logs, you can run it on VMs or bare-metal servers for logging. Nevertheless, the focus in this series is on Fluent Bit running on Azure Kubernetes Service and using Azure Log Analytics as the log backend.

I share with you specific learnings from the field operating Fluent Bit on Azure Kubernetes Service.

Kubernetes API endpoint vs. Kubelet endpoint

Per default, the Kubernetes filter plugin talks to the Kubernetes API endpoint https://kubernetes.default.svc:443 to enrich the log data with information about the Kubernetes pod.

For small and mid-sized Kubernetes clusters, this is not an issue, and you do not need to worry about overloading the API endpoint with those requests. On larger clusters, it can become an issue that the API endpoint gets unresponsive.

Hence, it is recommended to use the Kubelet endpoint instead. Fluent Bit gets deployed as a daemon set on a Kubernetes cluster. So, each node has its own Fluent Bit pod. The advantage of using the Kubelet endpoint is a faster response time to get the Kubernetes pod information and a reduced load on the API endpoint. The API endpoint approach is a 1:n relation where n is the number of Fluent Bit pods in the cluster. Whereas the Kubelet endpoint approach is a 1:1 relation.

AKS Azure portal - Fluent Bit daemon set overview

Looking at Azure Kubernetes Service, there is another advantage using the Kubelet endpoint. When you run an Azure Kubernetes Service cluster that is not a private cluster, the API endpoint has a public IP. That means when you are familiar with the topic of SNAT port exhaustion, every call from the Fluent Bit Kubernetes filter plugin to the API endpoint counts toward your available SNAT ports.

-> https://www.danielstechblog.io/detecting-snat-port-exhaustion-on-azure-kubernetes-service/
-> https://www.danielstechblog.io/preventing-snat-port-exhaustion-on-azure-kubernetes-service-with-virtual-network-nat/

My recommendation at that point is always to use the Kubelet endpoint option without considering if your Azure Kubernetes Service cluster is a small or a large one.

The configuration is straightforward and needs to be done for each Kubernetes filter configuration you use.

...
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Alias               logs_filter_1
        Match               kubernetes.logs.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        ...
        Use_Kubelet         On
        Kubelet_Host        ${NODE_IP}
        Kubelet_Port        10250
        tls.verify          Off
...

As seen above this is a working Fluent Bit configuration for Azure Kubernetes Service using the Kubelet endpoint. The option tls.verify must be set to Off. Otherwise, we cannot connect to the Kubelet endpoint. Furthermore, we use the Kubernetes downward API to dynamically hand in the node’s IP address as an environment variable that is referenced as value for the Kubelet_Host.

...
    spec:
      containers:
      - name: fluent-bit
        ...
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
...

The code snippet above is part of the Kubernetes template used to deploy Fluent Bit as a daemon set.

-> https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
-> https://kubernetes.io/docs/concepts/workloads/pods/downward-api/#available-fields

When we look into the Fluent Bit log output, we see that Fluent Bit successfully connects to the Kubelet endpoint.

...
[2023/01/09 20:49:40] [ info] [filter:kubernetes:logs_filter_1] https=1 host=10.240.0.4 port=10250
[2023/01/09 20:49:40] [ info] [filter:kubernetes:logs_filter_1]  token updated
[2023/01/09 20:49:40] [ info] [filter:kubernetes:logs_filter_1] local POD info OK
[2023/01/09 20:49:40] [ info] [filter:kubernetes:logs_filter_1] testing connectivity with Kubelet...
[2023/01/09 20:49:41] [ info] [filter:kubernetes:logs_filter_1] connectivity OK
...

Do not lose Kubernetes metadata information

Another important setting for the Kubernetes filter plugin is the setting Buffer_Size. The buffer size specifies the maximum size of the buffer for reading Kubernetes API responses. When Kubernetes metadata information exceeds the buffer size that information is discarded. Per default, the buffer size is 32 KB and too small. Even with a buffer size of 2 MB, you might receive the following warning message.

...
[2023/01/09 20:48:14] [ warn] [http_client] cannot increase buffer: current=32000 requested=64768 max=32000
...

That means some of the log data could not be enriched with the pod information. You only see then the time stamp and log message in the log backend. Such a log entry is not helpful as you cannot identify to which pod the log message belongs.

From my current experience, the only value that makes sense for the buffer size is 0. 0 means no limit for the buffer, and the buffer expands as needed.

...
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Alias               logs_filter_1
        Match               kubernetes.logs.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        ...
        Use_Kubelet         On
        Kubelet_Host        ${NODE_IP}
        Kubelet_Port        10250
        tls.verify          Off
        Buffer_Size         0
...

Setting the buffer size to 0 guarantees that we do not lose Kubernetes metadata for the log enrichment. The only exception will be that the Fluent Bit pod runs into an out-of-memory exception.

Keep that in mind when you specify the memory requests for the Fluent Bit daemon set. I have used the values from the Azure Monitor Container Insights solution, that by the way uses Fluent Bit as one of its components under the hood. The values are 325Mi for the memory requests and 750Mi for the limits.

Outlook

That is all for part one of the series “Learnings from the field – Running Fluent Bit on Azure Kubernetes Service”.

In the second part, I talk about my learnings regarding log ingestion to Azure Log Analytics. Stay tuned.


Posted

in

WordPress Cookie Notice by Real Cookie Banner