In the recent 3.2.2 release, the new Azure Data Explorer output plugin version is available.
Azure_kusto (Output)
- fix multiple files tail issue and timeout issue (#8430)
-> https://fluentbit.io/announcements/v3.2.2/
-> https://github.com/fluent/fluent-bit/pull/8430
The previous version had a couple of issues that have now been fixed. For instance, I was running into an unreliable authentication with the earlier version where in the end after several pod restarts the Fluent Bit pods were stuck in the authentication part against the Azure Data Explorer.
In the new version, the authentication and initialization part has been made more robust and configurable. It works like a charm as I have tested it in a preview version of the output plugin.
❯ kubectl logs fluent-bit-vxptf Fluent Bit v3.2.2 * Copyright (C) 2015-2024 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io ______ _ _ ______ _ _ _____ _____ | ___| | | | | ___ (_) | |____ |/ __ \ | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`' / /' | _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ / / | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /./ /___ \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)_____/ [2024/11/28 21:16:53] [ info] [fluent bit] version=3.2.2, commit=a59c867924, pid=1 [2024/11/28 21:16:53] [ info] [storage] ver=1.5.2, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128 [2024/11/28 21:16:53] [ info] [storage] backlog input plugin: storage_backlog.1 [2024/11/28 21:16:53] [ info] [simd ] disabled [2024/11/28 21:16:53] [ info] [cmetrics] version=0.9.9 [2024/11/28 21:16:53] [ info] [ctraces ] version=0.5.7 [2024/11/28 21:16:53] [ info] [input:storage_backlog:storage_backlog.1] initializing [2024/11/28 21:16:53] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only) [2024/11/28 21:16:53] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 47.7M [2024/11/28 21:16:53] [ info] [filter:kubernetes:logs_filter_1] https=1 host=kubernetes.default.svc port=443 [2024/11/28 21:16:53] [ info] [filter:kubernetes:logs_filter_1] token updated [2024/11/28 21:16:53] [ info] [filter:kubernetes:logs_filter_1] local POD info OK [2024/11/28 21:16:53] [ info] [filter:kubernetes:logs_filter_1] testing connectivity with Kubelet... [2024/11/28 21:16:53] [ info] [filter:kubernetes:logs_filter_1] connectivity OK [2024/11/28 21:16:53] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-adxaks.northeurope.kusto.windows.net', database='Kubernetes', table='ContainerLogs' [2024/11/28 21:16:53] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2024/11/28 21:16:53] [ info] [sp] stream processor started [2024/11/28 21:17:07] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs and refresh interval is 3487 [2024/11/28 21:17:08] [ info] [oauth2] HTTP Status=200 [2024/11/28 21:17:08] [ info] [oauth2] access token from 'login.microsoftonline.com:443' retrieved ...
Unfortunately, the PR for the updated documentation has not been merged yet. Hence, I am linking the PR here.
-> https://github.com/fluent/fluent-bit-docs/pull/1405
When you are using Fluent Bit with the Azure Data Explorer output plugin, I highly recommend updating to Fluent Bit 3.2.2 to benefit from the improvements that have been made with this new output plugin version.