Fluent Bit is a lightweight and efficient log collector and forwarder that enables seamless log gathering in a production environment.

In the realm of observability, logs play a crucial role by enhancing data analysis capabilities and facilitating troubleshooting. While our previous posts primarily focused on application logs, it’s important to note that numerous other solutions generate logs, including databases, systems, web servers, and access logs, among others.

To gain a comprehensive understanding of current events within our environments, it becomes essential to collect and correlate these logs effectively. This is precisely where Fluent Bit excels.

In our previous HowTo series on AWS, we talked about creating a custom AWS Fluent Bit image. In this article, you’ll learn what Fluent Bit is, how to configure it for production log gathering and what are its limitations.

What is Fluent Bit?

Fluent Bit is a member of the log collectors/forwarders family, serving as a powerful logging process tool. It operates as a lightweight and highly efficient log processor, functioning as the younger sibling of Fluentd.

In other words, Fluent Bit acts as a log collector, gathering logs from diverse sources such as traditional servers, Linux environments, containers, Kubernetes clusters, and pods.

It enhances the log data with contextual information, assigns labels to the logs, and transforms the log stream into a structured key-value pair format. These enriched logs can then be seamlessly sent to popular log storage solutions such as Elasticsearch, Kafka, and more.

Fluent Bit is a project hosted by CNCF and licensed under the Apache License v2.0, is a CNCF graduated sub-project within the Fluentd ecosystem. It was initially developed by Eduardo Silva, and as a community-driven initiative under CNCF, it maintains a fully vendor-neutral approach.

Configuring Fluent Bit for production log gathering:

Here’s a general guide on how to use Fluent Bit for production log gathering:

1. Installation and Configuration:

  • Install Fluent Bit on the desired host or machine. You can find installation instructions specific to your operating system on the Fluent Bit official documentation.
  • Configure Fluent Bit by creating a configuration file (e.g., `fluent-bit.conf`) that specifies the input, filters, and output plugins you want to use. This file determines how Fluent Bit collects, processes, and forwards logs.
  • You can refer to the Fluent Bit documentation for detailed configuration options and examples.

2. Define Input Sources:

  • Specify the input sources from which Fluent Bit should collect logs. This can include log files, syslog, Docker containers, or other sources depending on your setup.
  • Configure the appropriate input plugin in the configuration file, providing the necessary parameters such as log paths, listening ports, or environment variables.

3. Apply Filters (Optional):

  • Utilize filters to modify, parse, or enrich logs before forwarding them. Fluent Bit offers a range of filters, such as the `grep` filter for pattern matching, `parser` filter for parsing structured logs, and `record modifier` filter for altering log records.
  • Choose and configure the filters that suit your specific log processing requirements.

4. Define Output Destinations:

  • Specify the output destinations where Fluent Bit should forward the collected logs. This can include various options such as Elasticsearch, Kafka, Amazon S3, or other log aggregation systems.
  • Configure the appropriate output plugin in the configuration file, providing the necessary parameters like host addresses, credentials, or specific formatting options.

5. Start Fluent Bit:

  • Once you have completed the configuration, run the Fluent Bit service or execute the Fluent Bit binary with the path to the configuration file as a parameter. For example: `fluent-bit -c fluent-bit.conf`.
  • Ensure that Fluent Bit is running as a background service or as a managed process depending on your operating system and preferences.

6. Monitor and Troubleshoot:

  • Monitor Fluent Bit’s logs and metrics to ensure smooth operation and identify any potential issues. Fluent Bit provides its own logging mechanism, and you can configure output plugins to send logs to external monitoring systems if desired.
  • Monitor system resources to ensure Fluent Bit operates within acceptable limits.

7. Scale and Optimize:

  • As your log volume increases or your infrastructure grows, consider scaling Fluent Bit horizontally to handle the load. You can distribute Fluent Bit instances across multiple machines or containers and configure them to aggregate and forward logs to centralized destinations.
  • Continuously optimize the configuration and plugins to improve performance and meet evolving requirements.

By following these steps, you can effectively utilize Fluent Bit for log gathering in a production environment. Remember to consult the Fluent Bit documentation and leverage its extensive features and plugins to tailor the setup to your specific use case.

Configuration Steps

To configure Fluent Bit for production log gathering using the HTTP output plugin and enabling storage persistence, follow these steps:

1. Open the Fluent Bit configuration file (typically named `fluent-bit.conf`) for editing.

2. Add the following configuration settings under the `[SERVICE]` section:

				
					[SERVICE]
    Flush                      1
    Parsers_File              /etc/td-agent-bit/parsers.conf
    Log_Level                 error
    Storage.type              filesystem
    Storage.path              /var/log/flb_storage_
    Buffer storage.sync       normal
    Storage.checksum          On
    Storage.backlog.mem_limit 700kb
    Storage.metrics           On

The above settings configure the flushing frequency, specify the parsers file path, set the log level to `error`, enable filesystem storage, set the storage path, define buffer synchronization as `normal`, enable checksum verification, set the memory limit for the backlog, and enable storage metrics.

The `Log_Level` setting in the `[SERVICE]` section of the Fluent Bit configuration file determines the verbosity of log messages.

While the example above sets it to `error`, it can also be configured to `debug` for more detailed logging. By setting `Log_Level` to `debug`, Fluent Bit will provide additional log messages that can aid in debugging and troubleshooting any potential issues or errors.

However, it’s important to note that enabling debug logging may generate a higher volume of log output, which could impact system performance and should be used judiciously for production environments.

3. Add the input configuration section:

				
					[INPUT]
    Name              tail
    Path              /var/log/*.log
    Path_Key          filename
    Tag               Apica
    Buffer_Max_Size   1024k
    Read_from_Head    On
    Mem_Buf_Limit     1MB
    Refresh_Interval  5
    Storage.type      filesystem

This configuration sets the input plugin as `tail`, specifies the path of the log files to monitor (in this case, all files matching the `*.log` pattern in the `/var/log/` directory), uses the filename as a key for each log record, assigns the tag as `Apica`, sets the maximum buffer size, enables reading logs from the beginning of the file, sets the memory buffer limit, specifies the refresh interval, and sets the storage type as `filesystem`.

4. Add the filter configuration sections:

				
					[FILTER]
    Name               record_modifier
    Match              Apica
    Record cluster_id  flash

[FILTER]
    Name             record_modifier
    Match            Apica
    Record namespace  xyz

[FILTER]
    Name            record_modifier
    Match           Apica
    Record app_name system_logs

[FILTER]
    Name            throttle
    Match           *
    Rate            700
    Window          300
    Interval        1s

These filters modify log records for the `Apica` tag. The `record_modifier` plugin is utilized to enhance log records by adding the `cluster_id`, `namespace`, and `app_name` fields.

This modification provides additional context and facilitates better log organization. On the other hand, the `throttle` plugin plays a crucial role in controlling log ingestion rates.

By limiting the log rate to 700 logs per second within a 300-second window, the throttle plugin ensures smooth ingestion by buffering the logs. This prevents sudden spikes in log ingestion, stabilizes the system, and helps maintain a balanced and controlled flow of log data.

It also mitigates the risk of overwhelming downstream systems, optimizes resource utilization, and improves overall log processing efficiency.

Furthermore, the throttle plugin aids in maintaining a consistent and predictable log ingestion rate, making it easier to manage and analyze log data effectively.

5. Add the output configuration section:

				
					[OUTPUT]
    Name          http
    Match         *
    Host          lq5955.apica.io
    Port          80
    URI           /v1/json_batch
    Format        json
    tls           off
    tls.verify    off
    net.keepalive off
    compress      gzip
    Header Authorization Bearer ${Apica_TOKEN}

This configuration specifies the output plugin as `HTTP`, sets the matching condition for all logs, defines the host as `lq5955.apica.io`, uses port `80`, specifies the URI `/v1/json_batch` for the HTTP endpoint, sets the log format as JSON disables TLS, disables TLS verification, disables TCP keepalive, enables gzip compression, and includes an Authorization header with a token provided by the `${Apica_TOKEN}` environment variable.

6. Save and close the configuration file.

Ensure that you have the necessary permissions and access to the log files specified in the `Path` configuration.

Verify that the HTTP endpoint, including the host, port, and URI, is correct and that the required credentials or tokens are accurately provided. Once you’ve completed these steps, start or restart Fluent Bit with the updated configuration file, and it will start sending log events to the specified HTTP endpoint, using the configured plugins and settings for storage persistence.

Fluent Bit Limitations

Fluentbit, like any other log processing tool, has certain limitations with its default settings that can result in data loss and imbalances in data input and output.

However, these limitations can be mitigated by using storage persistence mechanisms to ensure that data is not lost.

The following are some limitations that stick out:

Data Loss:

  • By default, Fluent Bit prioritizes performance over durability, which can result in potential data loss during high-volume scenarios.
  • To mitigate data loss, enable storage persistence to store logs in a reliable and persistent manner, minimizing the risk of losing log data.

In/Out Imbalances:

  • In certain situations, the input rate of logs may exceed the output rate, causing a backlog of logs and potential data loss.
  • Configure storage persistence to provide a buffer for logs, allowing Fluent Bit to handle fluctuations in log input and output rates more effectively.

Storage Persistence:

  • Enabling storage persistence ensures that log data is safely stored on disk, even in the event of system failures or unexpected shutdowns.
  • By persisting logs to disk, Fluent Bit can recover and resume processing logs from where they left off, reducing the chances of data loss.

Configuring Storage:

  • Define the storage type (e.g., filesystem) and specify the storage path where logs will be stored.
  • Adjust the storage settings, such as the memory limit, to accommodate the volume of logs and the available resources.

Monitoring and Capacity Planning:

  • Regularly monitor Fluent Bit’s performance and resource usage, especially when handling large amounts of log data.
  • Plan and allocate sufficient storage space to avoid running out of disk space, which could lead to data loss.

By understanding and addressing these limitations through the use of storage persistence and careful configuration, you can enhance the reliability and durability of Fluent Bit, ensuring that log data is not lost and maintaining data integrity in log processing workflows.

What’s Ahead?

We hope this post has been useful in helping you understand the potential limitations of using Fluent Bit to process log data. Along with the nits and grits of Fluent Bit for Production Log Gathering.

For more information on Fluent Bit and its capabilities, check out our documentation and contact us if you have any questions. We’d love to hear from you!

In the next few posts, we’ll explore how you can use Fluent Bit to ship Windows logs to Apica, How to forward Amazon-Linux logs to Apica using Fluent Bit, and finally how to install Fluent Bit on Ubuntu.

Until then, happy logging!