v4.0.4

KNOWLEDGE BASE

Release Notes v4.0.4

Fluent Bit is a Fast and Lightweight Data Processor and Forwarder for Linux, BSD, Windows and macOS. We are proud to announce the availability of Fluent Bit v4.0.4.

This release brings significant improvements to OpenTelemetry support, including a new internal OTLP interface and enhanced Lua scripting with metadata awareness. This release also introduces native character encoding conversion in in_tail, full support for AWS MSK IAM authentication in Kafka plugins, and performance boosts via Kafka consumer auto-commit. In addition, we’ve included numerous stability, compatibility, and memory handling enhancements across the core engine and plugins — with special attention to NFS-friendly log rotation, improved upstream connection reuse, and expanded platform compatibility.

Below is a detailed list of changes included in this release…

What’s new ?

Core

• network
  • add new net.backlog property (somaxconn) (PR #10550)
  • on dns lookup with c-ares, unregister socket from the event loop (PR #10466)
  • on sync connection always check socket status (PR #10466)
• downstream
  • expose new net.backlog config property (PR #10550)
  • fix timeouts handling by fixing property check (PR #10552)
• http_client: validate return of host and content_length check (PR #10466)
• opentelemetry
  • new abstraction for OTel data handling (PR #10549)
• log_event_encoder_dynamic_field: ensure outstanding scopes are cleaned up before releasing the underlying buffer (PR #10549)
• pack: extra protection on error handling on JSON to msgpack (PR #10549)
• stream_processor: update yylex() signatures in lexer definition files (PR #10551)
• reload: fix handling hot reload when reload failure (PR #10547)
• hash_table
  • fix case-insensitive memory leak (PR #10424)
  • simplify logic to free heap allocated key (PR #10424)
• io: on network connect exceptions, invalidate file descriptor (PR #10425)
• internal: config: implement storage inheritance option (PR #10396)
• log_event_decoder: validate memory allocation (PR #10433)
• upstream
  • fix connection count when net.max_worker_connections is set (fix #10342) (PR #10423)
  • prioritize the reuse of connections instead of establishing new ones (PR #10491)
  • fix handling of the busy_flag for keepalive connections (PR #10466)
  • add missing return value in conn recycle (PR #10466)
• processor: add function to allow a direct string property value (PR #10444)
• multiline: release unpacked contexts on exceptions (PR #10449)
• http_server: allow to set a maximum buffer size for a client payload (PR #10448)
• config_format: yaml: add detailed error message on bad format (PR #10499)
• go: plugin_proxy: update function pointer signatures with correct args for C23 (PR #10489)
• build
  • Isolate onigmo headers for building process (PR #10517)
  • check for SASL before linking to it (PR #10525)
  • remove double-linking to ssl and crypto (PR #10525)
  • src: Build generic part of unicode interface (PR #10464)
  • adjust order for Kafka dependencies and AWS defs (PR #10536)
  • deprecate option FLB_FILTER_LUA_USE_MPACK (PR #10457)
• gzip: Process boundary conditions strictly (PR #10526)
• input_chunk: Keep consistent for appending chunks (PR #10527)
• aws
  • msk_iam: add support for MSK IAM auth (PR #10525)
  • only include aws_msk_iam if Kafka is enabled (PR #10525)
  • http_credentials: release config_key_profile on exception (PR #10466)
• kafka: add new API to manage a context with opaque pointers (PR #10525)
• packaging: add pkg-config (PR #10525)
• unicode
  • defs: Add conversion mapping for UTF-8 (PR #10464)
  • conv: Implement conversion rules for encoding conversion (PR #10464)
  • tests: internal: Add more aliases for encodings (PR #10464)
• conv: Unicode: Add a search function for encoding types (PR #10464)
• st: regparse: Handle C23 strict checking for function signatures (PR #10521)
• record_accessor
  • extend API to support binary and string refs (PR #10548)
  • update yylex() signatures in lexer definition files (PR #10551)

Libraries

• librdkafka: upgrade to v2.10.1 (PR #10525)
• cmetrics: upgrade to v1.0.4 (PR #10557)

Plugins

• Elasticsearch (Input)
  • use new http server api to set max buffer size (PR #10448)
• HTTP (Input)
  • use new http server api to set max buffer size (PR #10448)
• OpenTelemetry (Input)
  • use new http server api to set max buffer size (PR #10448)
  • use new opentelemetry interface for JSON logs (PR #10549)
• Prometheus remote write (Input)
  • use new http server api to set max buffer size (PR #10448)
• Splunk (Input)
  • use new http server api to set max buffer size (PR #10448)
  • continue with the following token if size doesn’t match (PR #10484)
• Calyptia_fleet (Input)
  • fix memory leak in get_fleet_id_from_header, found by @alecholmes. (PR #10467)
  • report the configuration version that is currently loaded when refreshing configuration. (PR #10496)
  • destroy context when failure is encountered during init. (PR #10502)
• Docker (Input)
  • update function pointer signatures with correct args for C23 (PR #10490)
• Kafka (Input)
  • add support for AWS MSK IAM auth (PR #10525)
  • auto-configure rdkafka internals when aws_msk_iam is enabled (PR #10525)
  • add enable_auto_commit option for Kafka consumer (PR #10558)
• Prometheus scrape metrics (Input)
  • release context if upstream context fails (PR #10539)
• Kernel Logs (Input)
  • use strtoull() to fix timestamp parsing on 32-bit systems (PR #10543)
• Tail (Input)
  • Handle encoding conversion engine for non UTF-16 encodings (PR #10542)
  • tests: runtime: Add test cases for encoding conversions from non UTF-16 encodings (PR #10542)
  • Fix build failures on CentOS7 (PR #10542)
  • only rely on fstat() to detect file rotation: useful for NFS. (PR #10405)
• Event_test (Input)
  • use new server api with backlog parameter (PR #10550)
• Winevtlog (Input)
  • Launch w/o valid subscription when using ignore_missing_channels parameter (PR #10555)
• Modify (Filter)
  • fix leaks on exceptions (PR #10434)
• Lua (Filter)
  • add groups metadata and body support (PR #10444)
  • add new function prototype to support groups and metadata access (PR #10457)
  • remove references to FLB_FILTER_LUA_USE_MPACK (PR #10457)
  • tests: Wait proper time with timeout (PR #10553)
• Content modifier (Processor)
  • fix leak when converting to JSON (PR #10450)
  • fix data type in comparison (PR #10450)
• Sampling (Processor)
  • fix handling of dropped spans (PR #10538)
• Sql (Processor)
  • update yylex() signatures in a lexer definition file (PR #10551)
• OpenTelemetry (Output)
  • fix leaks on exceptions (PR #10451)
  • fix handling of retry status (fix #10481) (PR #10497)
  • fix logic for grouping when using resource_id and scope_id (PR #10500)
  • fix behavior of record attributes fix the merge process (PR #10562)
• Kafka (Output)
  • add support for AWS MSK IAM auth (PR #10525)
  • auto-configure rdkafka internals when aws_msk_iam is enabled (PR #10525)
• Calyptia (Output)
  • destroy context when failure is encountered during init. (PR #10502)
• Cloudwatch logs (Output)
  • fix strcasestr() _GNU_SOURCE (PR #10452)
• Loki (Output)
  • prevent race conditions when multiple workers use the Remove_Keys (PR #10563)

Join us

We want to hear about you, our community is growing, and you can be part of it!, you can contact us at:

• Github: http://github.com/fluent/fluent-bit
• Slack: http://slack.fluentd.org
• Twitter: @fluentbit