In my last blog post, I wrote about the different options for using Azure Data Explorer as a logging backend for Fluent Bit.
-> https://www.danielstechblog.io/using-azure-data-explorer-as-logging-backend-for-fluent-bit/
Especially, about my issues getting the Azure Data Explorer output plugin working and why this led to the decision to use the Kafka output plugin in combination with Azure Event Hub to ingest the container log data.
Shortly after publishing the blog post the Azure Data Explorer engineering team reached out to me and asked if I was interested in a demo session as they are currently working on releasing the v2 version of the Azure Data Explorer output plugin.
Besides the other new functionality they added, they made the authentication and initialization part more robust and configurable. So, here we are again revisiting the topic “Using Azure Data Explorer as logging backend for Fluent Bit” with the Azure Data Explorer output plugin version v2.
INFO:
The Azure Data Explorer engineering team provided me with a pre-built Fluent Bit image that contained the latest v2 version of the Azure Data Explorer output plugin for evaluation. As of writing this blog post, it might be that the v2 version is still under review and not available yet to the entire Fluent Bit community. You can have a look at whether the following pull request is still open or was merged. The latter means that the new Azure Data Explorer output plugin version is available.
After that important note let us get started.
Prepare Azure Data Explorer for Fluent Bit
Now, we begin with the Azure Data Explorer configuration. Our first step is the provisioning of a new database in our Azure Data Explorer cluster, which is called Kubernetes with the default settings.
Once created, we open the query editor to prepare the table for the container logs that match the configured output of Fluent Bit.
.create table ContainerLogs (
TimeGenerated: datetime, Region: string, PodNamespace: string, PodName: string, ContainerName: string, LogSource: string, LogMessage: dynamic,
ContainerImage: string, ContainerImageDigest: string, PodId: guid, ContainerId: string,
Environment: string, Cluster: string, Computer: string, NodeIp: string
)
The next step is the ingestion mapping to ensure a correct ingestion into the table.
.create-or-alter table FluentBitLogs ingestion json mapping "FluentBitMapping"
```[
{"column": "TimeGenerated", "datatype": "datetime", "Properties": {"Path": "$.log.TimeGenerated"}},
{"column": "Region", "datatype": "string", "Properties": {"Path": "$.log.Region"}},
{"column": "PodNamespace", "datatype": "string", "Properties": {"Path": "$.log.PodNamespace"}},
{"column": "PodName", "datatype": "string", "Properties": {"Path": "$.log.PodName"}},
{"column": "ContainerName", "datatype": "string", "Properties": {"Path": "$.log.ContainerName"}},
{"column": "LogSource", "datatype": "string", "Properties": {"Path": "$.log.LogSource"}},
{"column": "LogMessage", "datatype": "dynamic", "Properties": {"Path": "$.log.LogMessage"}},
{"column": "ContainerImage", "datatype": "string", "Properties": {"Path": "$.log.ContainerImage"}},
{"column": "ContainerImageDigest", "datatype": "string", "Properties": {"Path": "$.log.ContainerImageDigest"}},
{"column": "PodId", "datatype": "guid", "Properties": {"Path": "$.log.PodId"}},
{"column": "ContainerId", "datatype": "string", "Properties": {"Path": "$.log.ContainerId"}},
{"column": "Environment", "datatype": "string", "Properties": {"Path": "$.log.Environment"}},
{"column": "Cluster", "datatype": "string", "Properties": {"Path": "$.log.Cluster"}},
{"column": "Computer", "datatype": "string", "Properties": {"Path": "$.log.Computer"}},
{"column": "NodeIp", "datatype": "string", "Properties": {"Path": "$.log.NodeIp"}}
]```
Whereas in the previous blog post, we could reference the keys directly, for instance, $.PodName, the Azure Data Explorer output plugin puts all relevant container log data under the log key section, for instance, $.log.PodName.
Compared to the Kafka output plugin with an Azure Event Hub we only have the queue ingestion option with the Azure Data Explorer output plugin available instead of the streaming ingestion option.
Hence, we apply the following batch ingestion policy on the entire database level.
.alter database Kubernetes policy ingestionbatching
```
{
"MaximumBatchingTimeSpan": "00:00:10",
"MaximumNumberOfItems": 500,
"MaximumRawDataSizeMB": 1024
}
```
Without any further optimization on the Azure Data Explorer output plugin side and depending on the ingestion volume and Azure Data Explorer cluster size we can expect the same ingestion performance as someone is used to with Azure Log Analytics in combination with Fluent Bit.
The last step that is required on the Azure Data Explorer side is the Database Ingestor permission for the service principal we use later for Fluent Bit.
.add database Kubernetes ingestors ('aadapp=<Application_ID>;<Tenant_ID>')
We set the permissions on the database level.
Configure Fluent Bit
The entire Fluent Bit configuration is available on my GitHub repository under the following link.
-> https://github.com/neumanndaniel/kubernetes/tree/master/fluent-bit/azure-data-explorer
Instead, we focus on the two filters after the kubernetes filter to prepare the container logs and transfer them into the correct key format that matches the Azure Data Explorer ingestion mapping. Afterward, the configuration of the Azure Data Explorer output plugin follows.
...
[FILTER]
Name nest
Alias logs_filter_2
Match kubernetes.logs.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
...
The first filter that follows the kubernetes filter is the nest filter that makes Kubernetes pod metadata information under the prefix kubernetes_ available.
...
[FILTER]
Name modify
Alias logs_filter_3
Match kubernetes.logs.*
Add Cluster ${CLUSTER}
Add Region ${REGION}
Add Environment ${ENVIRONMENT}
Add NodeIp ${NODE_IP}
Rename time TimeGenerated
Rename message LogMessage
Rename kubernetes_pod_name PodName
Rename kubernetes_namespace_name PodNamespace
Rename kubernetes_container_image ContainerImage
Rename kubernetes_container_hash ContainerImageDigest
Rename kubernetes_docker_id ContainerId
Rename kubernetes_container_name ContainerName
Rename kubernetes_pod_id PodId
Rename kubernetes_host Computer
Rename stream LogSource
Remove logtag
...
We then use the modify filter to add additional keys to the log output and rename existing keys to match the Azure Data Explorer ingestion mapping. Configuration placeholders like ${CLUSTER} are environment variables passed into the Fluent Bit pod via the daemon set configuration as seen in the below snippet.
...
env:
- name: FLUENT_ADX_TENANT_ID
valueFrom:
secretKeyRef:
name: azuredataexplorer
key: tenant_id
- name: FLUENT_ADX_CLIENT_ID
valueFrom:
secretKeyRef:
name: azuredataexplorer
key: client_id
- name: FLUENT_ADX_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: azuredataexplorer
key: client_secret
...
The below Azure Data Explorer output plugin configuration contains some of the new configuration options. In my example, I am using the standard default values for them.
...
output-kubernetes.conf: |
[OUTPUT]
Name azure_kusto
Match kubernetes.logs.*
Tenant_Id ${FLUENT_ADX_TENANT_ID}
Client_Id ${FLUENT_ADX_CLIENT_ID}
Client_Secret ${FLUENT_ADX_CLIENT_SECRET}
Ingestion_Endpoint https://ingest-adxaks.northeurope.kusto.windows.net
Database_Name Kubernetes
Table_Name ContainerLogs
Ingestion_Mapping_Reference FluentBitMapping
Log_Key log
Include_Tag_Key Off
Include_Time_Key Off
Retry_Limit False
Log_Level info
compression_enabled on
ingestion_endpoint_connect_timeout 60
ingestion_resources_refresh_interval 3600
buffering_enabled false
Beside the necessary settings Tenant_Id, Client_Id, Client_Secret, Ingestion_Endpoint, Database_Name, Table_Name, and Ingestion_Mapping_Reference I want to highlight Retry_Limit and Log_Level.
One of the most important things when using Fluent Bit is to ensure that we do not lose any log data on the log collection side by setting the configuration parameter Retry_Limit to False.
Having the Log_Level set to info, the default value, allows us to retrieve information about the authentication and initialization part of the Azure Data Explorer output plugin.
❯ kubectl logs fluent-bit-pph4m
Fluent Bit v3.0.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
___________.__ __ __________.__ __ ________
\_ _____/| | __ __ ____ _____/ |_ \______ \__|/ |_ ___ _\_____ \
| __) | | | | \_/ __ \ / \ __\ | | _/ \ __\ \ \/ / _(__ <
| \ | |_| | /\ ___/| | \ | | | \ || | \ / / \
\___ / |____/____/ \___ >___| /__| |______ /__||__| \_/ /______ /
\/ \/ \/ \/ \/
[2024/08/23 19:45:07] [ info] [fluent bit] version=3.0.7, commit=598e92b0f6, pid=1
[2024/08/23 19:45:07] [ info] [storage] ver=1.5.2, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128
[2024/08/23 19:45:07] [ info] [storage] backlog input plugin: storage_backlog.1
[2024/08/23 19:45:07] [ info] [cmetrics] version=0.9.1
[2024/08/23 19:45:07] [ info] [ctraces ] version=0.5.1
[2024/08/23 19:45:07] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2024/08/23 19:45:07] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2024/08/23 19:45:07] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 47.7M
[2024/08/23 19:45:07] [ info] [filter:kubernetes:logs_filter_1] https=1 host=kubernetes.default.svc port=443
[2024/08/23 19:45:07] [ info] [filter:kubernetes:logs_filter_1] token updated
[2024/08/23 19:45:07] [ info] [filter:kubernetes:logs_filter_1] local POD info OK
[2024/08/23 19:45:07] [ info] [filter:kubernetes:logs_filter_1] testing connectivity with Kubelet...
[2024/08/23 19:45:08] [ info] [filter:kubernetes:logs_filter_1] connectivity OK
[2024/08/23 19:45:08] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-adxaks.northeurope.kusto.windows.net', database='Kubernetes', table='ContainerLogs'
[2024/08/23 19:45:08] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/08/23 19:45:08] [ info] [sp] stream processor started
[2024/08/23 19:45:22] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resources and refresh interval is 3904
[2024/08/23 19:45:22] [ info] [oauth2] HTTP Status=200
[2024/08/23 19:45:22] [ info] [oauth2] access token from 'login.microsoftonline.com:443' retrieved
As seen above the authentication and initialization part is now working like a charm compared to the v1 version of the output plugin.
The required Kubernetes secret in our setup that provides the service principal credentials is created by running the following script that also deploys Fluent Bit.
❯ ./deploy-fluent-bit-adx.sh TENANT_ID CLIENT_ID CLIENT_SECRET ❯ ./deploy-fluent-bit-adx.sh 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 Pa$$W0rd
After a successful deployment of Fluent Bit, we should see the first container logs in Azure Data Explorer.
Summary
The new v2 version of the Azure Data Explorer output plugin fixes the issues in the authentication and initialization part and adds additional new configuration parameters to fine tune the ingestion on the plugin side.
The example KQL file and the entire Fluent Bit deployment configuration can be found on my GitHub repository.
-> https://github.com/neumanndaniel/scripts/tree/main/Azure_Data_Explorer/Fluent_Bit_Kubernetes
-> https://github.com/neumanndaniel/kubernetes/tree/master/fluent-bit/azure-data-explorer