Azure Data Explorer can be used as a logging backend for Fluent Bit in three different ways. In today’s blog post, we focus on one of them that in my opinion is the best one out of those three.
This is the way
Let us start first with a brief overview of the three different solutions and why I have chosen the not-so-obvious one. In my last blog post, I already talked about two of them: Azure Storage and Azure Event Hub. Not particularly for Fluent Bit but as ingestion methods for Azure Data Explorer.
-> https://www.danielstechblog.io/ingesting-azure-diagnostic-logs-into-azure-data-explorer/
Before we cover those two options let us look into the first option that comes to our mind looking into Fluent Bit’s output plugin offerings. There is a native output plugin for Azure Data Explorer.
-> https://docs.fluentbit.io/manual/pipeline/outputs/azure_kusto
The Azure Data Explorer output plugin uses the queued ingestion method which can be optimized by applying a batch ingestion policy. If a 10-second ingestion delay for your logs is acceptable then the Azure Data Explorer output plugin is the best option. During my tests, I found out that the core oauth2 implementation in Fluent Bit is not always working as expected. Especially, when you restart Fluent Bit pods on Kubernetes. For instance, on my three-node Azure Kubernetes Service cluster two Fluent Bit pods were working after the initial deployment. The version I used for my tests was Fluent Bit 3.0.7.
[2024/07/02 06:50:18] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-adxaks.northeurope.kusto.windows.net', database='Kubernetes', table='FluentBitLogs' [2024/07/02 06:50:32] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs [2024/07/02 06:50:32] [ info] [oauth2] HTTP Status=200 [2024/07/02 06:50:32] [ info] [oauth2] access token from 'login.microsoftonline.com:443' retrieved
The other one was stuck in the following state.
[2024/07/02 06:50:18] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-adxaks.northeurope.kusto.windows.net', database='Kubernetes', table='FluentBitLogs' [2024/07/02 06:50:32] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs
After simulating several restarts of the Fluent Bit pods, all ended up in this stuck state. Unfortunately, I have not had the time to dive deeper into this or open a GitHub issue on the project’s repository. That said you should keep that in mind when you choose the Azure Data Explorer output plugin.
Another option is the Azure Blob output plugin which is officially developed by Microsoft.
-> https://docs.fluentbit.io/manual/pipeline/outputs/azure_blob
I already highlighted in my above-mentioned blog post that the Azure Storage Account option gets expensive when a lot of files are written and read from the Storage Account due to transaction costs. Especially, with a lot of pods running in a Kubernetes cluster, the costs will explode.
Here is a small example from my three-node Azure Kubernetes Service cluster with only running a small number of pods.
Within four hours I had 25.000 transactions in total. Now imagine a couple of large Kubernetes clusters emit their container logs to an Azure Storage Account. Besides that, the Storage Account option requires an Event Grid and Event Hub to get the container logs into Azure Data Explorer.
Therefore, the only viable option to get container logs from Fluent Bit into Azure Data Explorer is the usage of the Kafka output plugin.
-> https://docs.fluentbit.io/manual/pipeline/outputs/kafka
The Kafka output plugin gets connected to an Azure Event Hub as Event Hub provides an Apache Kafka endpoint. Using the Event Hub option for the Azure Data Explorer ingestion allows us to enable the streaming ingestion into Azure Data Explorer for container logs we gather with Fluent Bit. Streaming ingestion allows a near real-time ingestion.
Let us start with the Azure Data Explorer and Fluent Bit configuration.
Prepare Azure Data Explorer for Fluent Bit
As mentioned, before we use the Kafka output plugin in Fluent Bit to get our container logs via Event Hub into Azure Data Explorer.
Now, we begin with the Azure Data Explorer configuration. Our first step is the provisioning of a new database in our Azure Data Explorer cluster, which is called Kubernetes with the default settings.
Once created, we open the query editor to prepare the table for the container logs that match the configured output of Fluent Bit.
.create table ContainerLogs (
TimeGenerated: datetime, Region: string, PodNamespace: string, PodName: string, ContainerName: string, LogSource: string, LogMessage: dynamic,
ContainerImage: string, ContainerImageDigest: string, PodId: guid, ContainerId: string,
Environment: string, Cluster: string, Computer: string, NodeIp: string
)
The next step is the ingestion mapping to ensure a correct ingestion into the table.
.create-or-alter table ContainerLogs ingestion json mapping "FluentBitMapping"
```[
{"column": "TimeGenerated", "datatype": "datetime", "Properties": {"Path": "$.TimeGenerated"}},
{"column": "Region", "datatype": "string", "Properties": {"Path": "$.Region"}},
{"column": "PodNamespace", "datatype": "string", "Properties": {"Path": "$.PodNamespace"}},
{"column": "PodName", "datatype": "string", "Properties": {"Path": "$.PodName"}},
{"column": "ContainerName", "datatype": "string", "Properties": {"Path": "$.ContainerName"}},
{"column": "LogSource", "datatype": "string", "Properties": {"Path": "$.LogSource"}},
{"column": "LogMessage", "datatype": "dynamic", "Properties": {"Path": "$.LogMessage"}},
{"column": "ContainerImage", "datatype": "string", "Properties": {"Path": "$.ContainerImage"}},
{"column": "ContainerImageDigest", "datatype": "string", "Properties": {"Path": "$.ContainerImageDigest"}},
{"column": "PodId", "datatype": "guid", "Properties": {"Path": "$.PodId"}},
{"column": "ContainerId", "datatype": "string", "Properties": {"Path": "$.ContainerId"}},
{"column": "Environment", "datatype": "string", "Properties": {"Path": "$.Environment"}},
{"column": "Cluster", "datatype": "string", "Properties": {"Path": "$.Cluster"}},
{"column": "Computer", "datatype": "string", "Properties": {"Path": "$.Computer"}},
{"column": "NodeIp", "datatype": "string", "Properties": {"Path": "$.NodeIp"}}
]```
Using Event Hub for the Azure Data Explorer ingestion allows us to enable streaming ingestion for near real-time ingestion. Hence, we enable the streaming ingestion policy for the whole Kubernetes database.
.alter database Kubernetes policy streamingingestion enable
After provisioning the table for the data ingestion, we create the necessary data connection between Event Hub and the Azure Data Explorer database.
As seen in the screenshot, we provide a name for the data connection and select the appropriate Event Hub. The compression setting is kept with its default setting None. Furthermore, we provide the table name with the corresponding ingestion mapping. Last but not least, we select the managed identity type for the data connection. In our case, from type system-assigned.
Once the data connection has been created, we can monitor the connection and see how many events have been received and processed when we start to send container logs with Fluent Bit via the Kafka output plugin.
The Event Hub Namespace was created upfront with an Event Hub called Kubernetes with 8 partitions and an enabled auto-inflate functionality for the Event Hub throughput units.
For a production environment, I would set the partition count to the max value of 32 partitions.
Configure Fluent Bit
The entire Fluent Bit configuration is available on my GitHub repository under the following link.
-> https://github.com/neumanndaniel/kubernetes/tree/master/fluent-bit/azure-data-explorer
Instead, we focus on the two filters after the kubernetes filter to prepare the container logs and transfer them into the correct key format that matches the Azure Data Explorer ingestion mapping. Afterward, the configuration of the Kafka output plugin follows.
...
[FILTER]
Name nest
Alias logs_filter_2
Match kubernetes.logs.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
...
The first filter that follows the kubernetes filter is the nest filter that makes Kubernetes pod metadata information under the prefix kubernetes_ available.
...
[FILTER]
Name modify
Alias logs_filter_3
Match kubernetes.logs.*
Add Cluster ${CLUSTER}
Add Region ${REGION}
Add Environment ${ENVIRONMENT}
Add NodeIp ${NODE_IP}
Rename time TimeGenerated
Rename message LogMessage
Rename kubernetes_pod_name PodName
Rename kubernetes_namespace_name PodNamespace
Rename kubernetes_container_image ContainerImage
Rename kubernetes_container_hash ContainerImageDigest
Rename kubernetes_docker_id ContainerId
Rename kubernetes_container_name ContainerName
Rename kubernetes_pod_id PodId
Rename kubernetes_host Computer
Rename stream LogSource
Remove logtag
...
We then use the modify filter to add additional keys to the log output and rename existing keys to match the Azure Data Explorer ingestion mapping. Configuration placeholders like ${CLUSTER} are environment variables passed into the Fluent Bit pod via the daemon set configuration as seen in the below snippet.
...
env:
- name: FLUENT_BIT_EVENT_HUB_NAMESPACE
valueFrom:
secretKeyRef:
name: azureeventhub
key: namespace
- name: FLUENT_BIT_EVENT_HUB
valueFrom:
secretKeyRef:
name: azureeventhub
key: topic
- name: FLUENT_BIT_EVENT_HUB_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: azureeventhub
key: connection_string
...
Before we dive deeper into the Kafka output plugin we generate a new shared access policy for our Kubernetes Event Hub.
Using the shared access policy of the Event Hub and not of the entire Event Hub Namespace allows us to restrict Fluent Bit’s access to only this particular Event Hub.
...
output-kubernetes.conf: |
[OUTPUT]
Name kafka
Alias logs_output
Match kubernetes.logs.*
Brokers ${FLUENT_BIT_EVENT_HUB_NAMESPACE}.servicebus.windows.net:9093
Topics ${FLUENT_BIT_EVENT_HUB}
Retry_Limit False
Log_Level info
Queue_Full_Retries 0
Timestamp_Key @TimeGenerated
Timestamp_Format iso8601_ns
Format json
rdkafka.client.id fluent-bit
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.mechanism PLAIN
rdkafka.sasl.username $ConnectionString
rdkafka.sasl.password ${FLUENT_BIT_EVENT_HUB_CONNECTION_STRING}
rdkafka.request.required.acks 1
rdkafka.log.connection.close false
rdkafka.message.timeout.ms 0
One of the most important things when using Fluent Bit is to ensure that we do not lose any log data on the log collection side. Hence, the configuration parameters Retry_Limit, Queue_Full_Retries, and rdkafka.message.timeout.ms try to ensure that. Then we use some best practice configurations for the parameters rdkafka.request.required.acks and rdkafka.connection.close. The rdkafka.sasl.mechanism is set to PLAIN for using a connection string for authentication.
A full list of the rdkafka configuration parameters and Fluent Bit’s Kafka output plugin can be found under the following links.
-> https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md
-> https://docs.fluentbit.io/manual/pipeline/outputs/kafka
The required Kubernetes secret in our setup that provides the Event Hub Namespace name, the Event Hub name, and the connection string to Fluent Bit is created by running the following script that also deploys Fluent Bit.
❯ ./deploy-fluent-bit-kafka.sh RESOURCE_GROUP EVENT_HUB_NAMESPACE EVENT_HUB SHARED_ACCESS_POLICY_NAME ❯ ./deploy-fluent-bit-kafka.sh adx adxaks kubernetes fluent-bit
After a successful deployment of Fluent Bit, we should see the first container logs in Azure Data Explorer.
Summary
Fluent Bit supports several output plugins that can be used for data ingestion into Azure Data Explorer. The most suitable and versatile is from my current experience the Kafka output plugin with an Azure Event Hub. This option has some key advantages compared to the Azure Data Explorer and Azure Blob output plugin. First, we can use the streaming ingestion for near real-time ingestion compared to the queued ingestion method that the Azure Data Explorer output plugin uses. Second, it is cost-efficient compared to the Azure Blob output plugin which suffers in the end from the storage transaction costs in Azure.
So, my recommendation at the moment is the Kafka output plugin with an Azure Event Hub to ingest container logs via Fluent Bit into Azure Data Explorer.
Unfortunately, I had issues getting the Azure Data Explorer output plugin into a stable operation which would have been my preferred solution.
The example KQL file and the entire Fluent Bit deployment configuration can be found on my GitHub repository.
-> https://github.com/neumanndaniel/scripts/tree/main/Azure_Data_Explorer/Fluent_Bit_Kubernetes
-> https://github.com/neumanndaniel/kubernetes/tree/master/fluent-bit/azure-data-explorer