๐Ÿ”ฅ Fluent Bit v4.0.4 is here !

TL;DR: Fluent Bit v4.0.4 introduces a powerful OpenTelemetry interface for better encoding and decoding, enhanced Lua scripting with OTLP metadata access, character encoding conversion for non-UTF-8 logs, and AWS IAM support for Kafka/MSK โ€” plus performance boosts, NFS-tail fixes, and dozens of other improvements.

New OpenTelemetry Interface

Fluent Bit continues to strengthen its position as a modern OpenTelemetry data pipeline. In v4.0.4, we introduce a new internal OpenTelemetry interface for handling encoding and decoding โ€” making OTLP log processing faster, safer, and more testable.

Key Improvements:

  • ๐Ÿ”„ Unified OTLP handling across JSON and Protobuf
  • โœ… Cleaner separation of concerns between input, processing, and output stages
  • ๐Ÿงช Better unit testing and error reporting for encoding issues
  • ๐Ÿ”ง New support for custom logs_body_key in out_opentelemetry and in_opentelemetry to structure OTLP log bodies as needed

This foundation enables stronger support for trace/metrics/logs ingestion, and positions Fluent Bit as a robust forwarder in OTLP-native environments.

๐Ÿง  Lua Filter Superpowers: OTLP Metadata + Group Access

In v4.0.4, the Lua filter is now OpenTelemetry-aware, enabling advanced record-level and group-level processing. This is ideal for modifying OTLP logs on the fly based on resource or scope attributes.

New Function Signature:

1

function cb(tag, timestamp, group, metadata, record)

What You Can Do:
  • Inject resource.attributes (like service name) into log records
  • Modify OTLP severity levels or labels
  • Return multiple records and metadata per invocation
  • Mix both simple and complex scripting with backward compatibility
Example: Adjust OTLP Severity and Add Service Name
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

function cb(tag, ts, group, metadata, record)
  if group['resource']['attributes']['service.name'] then
    record['service_name'] = group['resource']['attributes']['service.name']
  end

  if metadata['otlp']['severity_number'] == 9 then
    metadata['otlp']['severity_number'] = 13
    metadata['otlp']['severity_text'] = 'WARN'
  end

  return 1, ts, metadata, record
end

๐Ÿงฉ Compatible with OpenTelemetry Collector logs and any OTLP log producers.

๐ŸŒ Encoding Support for Asian Markets: Native Transcoding

For many organizations in Asia, logs are still written using encodings like GBK, Big5, or Shift_JIS โ€” leading to ๆ–‡ๅญ—ๅŒ–ใ‘ (mojibake) when sent to UTF-8-based backends.

Fluent Bit now includes a native character encoding engine in in_tail. With the new generic.encoding property, you can transcode logs to UTF-8 as they are read.

Example: Transcode from GBK to UTF-8
1
2
3
4
5
6
7
8
9

pipeline:
  inputs:
    - name: tail
      path: /var/log/my_legacy_app.log
      parser: json
      generic.encoding: GBK

  outputs:
    - name: stdout

๐Ÿ” Ensures clean UTF-8 logs for Elasticsearch, OpenSearch, S3, or any backend. ๐ŸŽŒ Perfect for smooth Fluentd โ†’ Fluent Bit migrations.

๐Ÿ” Kafka + AWS MSK IAM + Performance Boosts

Fluent Bit now includes native support for AWS IAM-based authentication when talking to Amazon MSK (both input and output plugins).

Highlights:
  • ๐Ÿ›ก๏ธ Uses AWS SigV4 signing via OAuthBearer
  • โ˜๏ธ Works with EC2 IAM roles, STS tokens, and serverless MSK
  • ๐Ÿงผ Stateless token refresh implementation using librdkafka

No more credentials in configs โ€” just IAM and go.

Bonus: enable_auto_commit for Kafka Input

Kafka consumers now support:

1

enable_auto_commit: true
  • false (default) = safety-first, only commit after full processing
  • true = performance-oriented, batch commit behavior, better throughput

Use this if your backend can tolerate replays and you want maximum ingestion performance.

๐Ÿงฑ in_tail File Rotation Improvements (NFS-Friendly)

Working with log files on NFS or remote mounts can be tricky โ€” especially when files are rotated or overwritten.

In v4.0.4, in_tail now uses fstat()-based rotation detection, making it much more reliable in environments like:

  • ๐Ÿ“ Network File Systems (NFS)
  • ๐Ÿชต Centralized log directories
  • ๐Ÿณ Containerized applications with volume mounts

๐Ÿงฏ Prevents duplicate logs, dropped records, or missed tailing when files are moved or rotated externally.

โš™๏ธ Other Highlights

Plugins
  • out_opentelemetry: fixes for retry, metadata merging, and grouping
  • out_loki: prevents race conditions when multiple workers use Remove_Keys
  • filter_modify / filter_lua: memory fixes and cleanup
  • in_kafka / out_kafka: improved AWS MSK integration and token validation
Core & Build System
  • ๐Ÿ“ฆ librdkafka upgraded to v2.10.1
  • ๐Ÿงช New tests for encodings, filters, upstream behavior
  • ๐Ÿงต Better upstream connection reuse logic
  • ๐Ÿง Support for Rocky Linux and AlmaLinux in installer
  • ๐Ÿงน Code cleanup and memory leak fixes across the board

๐Ÿ› ๏ธ How to Upgrade

Fluent Bit v4.0.4 is available now:

๐Ÿ™Œ Thanks to the Community

This release wouldnโ€™t be possible without contributions, bug reports, and feedback from our users and partners around the world.