The new Fluent Bit version 4 contains an interesting new feature for the Azure Data Explorer output plugin. Besides the standard option of using an Azure Service Principal for authentication, it now additionally supports an Azure Managed Identity for authentication.
Azure_kusto (Output)
- azure managed identity support added (#10036)
-> https://fluentbit.io/announcements/v4.0.0/
-> https://github.com/fluent/fluent-bit/pull/10036
Azure Managed Identities provide a more secure way than Azure Service Principals as they eliminate credential management and are bound to only one or several Azure resources. The latter depends on the type of the managed identity.
In this blog post today, we walk through the necessary steps to use Fluent Bit on Azure Kubernetes Service with an Azure Managed Identity for the Azure Data Explorer output plugin.
Configuration
On Azure Kubernetes Service, we can use managed identities in two ways, with or without workload identities. We focus on the option without workload identities where we use the managed identity of the Azure Kubernetes Service cluster node pool. This is the so-called kubelet identity and is the easiest option that does not require additional configuration on the Azure Kubernetes Service cluster side.
For the Azure Data Explorer output plugin configuration option managed_identity_client_id we require the GUID of the managed identity on the first look as Azure Kubernetes Service uses a user-assigned managed identity for the node pool.
Set to SYSTEM for system-assigned managed identity, or set to the MI client ID (GUID) for user-assigned managed identity.
However, as this is the only managed identity for the node pool in a default configuration, we can use SYSTEM as well.
Nevertheless, let us walk through the necessary steps to get the GUID of the kubelet identity, as this is required to assign the necessary permissions on the Azure Data Explorer database. We achieve this by running the following Azure CLI command.
❯ az aks show -g rg-azst-1 -n aks-azst-1 --query 'identityProfile.kubeletidentity'
{
"clientId": "<REDACTED>",
"objectId": "<REDACTED>",
"resourceId": "/subscriptions/<REDACTED>/resourcegroups/rg-aks-azst-1-nodes/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aks-azst-1-agentpool"
}
Once we have the GUID of the managed identity, we assign the Database Ingestor role to the managed identity on the Azure Data Explorer database.
The last step is the configuration of the Azure Data Explorer output plugin. As mentioned earlier, we can either use SYSTEM or the managed identity’s GUID as a value for the managed_identity_client_id configuration option.
[OUTPUT]
Name azure_kusto
Match kubernetes.logs.*
managed_identity_client_id SYSTEM
Ingestion_Endpoint https://ingest-adxaks.northeurope.kusto.windows.net
Database_Name Kubernetes
Table_Name ContainerLogs
Ingestion_Mapping_Reference FluentBitMapping
Log_Key log
Include_Tag_Key Off
Include_Time_Key Off
Retry_Limit False
Log_Level info
compression_enabled on
ingestion_endpoint_connect_timeout 60
ingestion_resources_refresh_interval 3600
Once applied to the Azure Kubernetes Service cluster, we check the logs of the Fluent Bit pods.
❯ kubectl logs fluent-bit-x5tzg Fluent Bit v4.0.0 * Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io ______ _ _ ______ _ _ ___ _____ | ___| | | | | ___ (_) | / || _ | | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | | _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ [2025/04/25 20:33:30] [ info] [fluent bit] version=4.0.0, commit=712b48d7c3, pid=1 [2025/04/25 20:33:30] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2025/04/25 20:33:30] [ info] [simd ] disabled [2025/04/25 20:33:30] [ info] [cmetrics] version=0.9.9 [2025/04/25 20:33:30] [ info] [ctraces ] version=0.6.2 [2025/04/25 20:33:30] [ info] [filter:kubernetes:logs_filter_1] https=1 host=kubernetes.default.svc port=443 [2025/04/25 20:33:30] [ info] [filter:kubernetes:logs_filter_1] token updated [2025/04/25 20:33:30] [ info] [filter:kubernetes:logs_filter_1] local POD info OK [2025/04/25 20:33:30] [ info] [filter:kubernetes:logs_filter_1] testing connectivity with Kubelet... [2025/04/25 20:33:30] [ info] [filter:kubernetes:logs_filter_1] connectivity OK [2025/04/25 20:33:30] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-adxaks.northeurope.kusto.windows.net', database='Kubernetes', table='ContainerLogs' [2025/04/25 20:33:30] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2025/04/25 20:33:30] [ info] [sp] stream processor started [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3399 [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3880 [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3801 [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3803 [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3900 [2025/04/25 20:33:45] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 4165 [2025/04/25 20:33:45] [ info] [azure msi auth] HTTP Status=200 [2025/04/25 20:33:45] [ info] [azure msi auth] access token from '169.254.169.254:80' retrieved
After examining the Fluent Bit pods’ logs, we check the Azure Data Explorer database to see if new log data arrives correctly.
Summary
The new managed identity option for the Azure Data Explorer output plugin in Fluent Bit makes the day-to-day operation of the log ingestion way easier. We do not require credential management anymore, and if we do not use a user-assigned managed identity, the permissions for a managed identity are automatically removed from the Azure Data Explorer database when the Azure resource of the managed identity gets deleted.
-> https://docs.fluentbit.io/manual/pipeline/outputs/azure_kusto