From 206b8eb4dc3c1b7bf562f2e44d3c035d2128a307 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Sat, 1 Aug 2026 14:13:47 +0200 Subject: [PATCH] docs: document behavior changes for s3 log_key and HTTP server Two merged source changes alter existing behavior in ways users can hit on upgrade. - pipeline/outputs/s3.md: note in the log_key row that the record key name must match exactly, linking to the new section below. - pipeline/outputs/s3.md: add a "Key matching for log_key" section describing the prefix matching used before 5.1, where a record key that was a prefix of the configured value matched (log_key log_level selecting a log field instead), and the "Could not find log_key" error that identifies an affected configuration after upgrading. - administration/monitoring.md: document that a monitoring HTTP server that cannot bind now fails startup with "could not start HTTP server", where earlier versions continued running without the endpoint. Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell --- administration/monitoring.md | 2 ++ pipeline/outputs/s3.md | 9 ++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/administration/monitoring.md b/administration/monitoring.md index 5e9a8bbd6..e51ba8c20 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -79,6 +79,8 @@ Fluent Bit starts and generates output in your terminal: [2020/03/10 19:08:24] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 ``` +In Fluent Bit 5.1 and later, a monitoring endpoint that can't start is treated as a startup failure. If the HTTP server can't bind, for example because `http_port` is already in use or `http_listen` isn't a valid local address, Fluent Bit logs `could not start HTTP server` and exits instead of running without the endpoint. In earlier versions, Fluent Bit continued to start and process data, and the failure only surfaced when something tried to reach the endpoint. + Use `curl` to gather information about the HTTP server. The following command sends the command output to the `jq` program, which outputs human-readable JSON data to the terminal. ```shell diff --git a/pipeline/outputs/s3.md b/pipeline/outputs/s3.md index d3bc77cf4..70a487160 100644 --- a/pipeline/outputs/s3.md +++ b/pipeline/outputs/s3.md @@ -55,7 +55,7 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor | `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | | `json_date_format` | Specify the format of the date. Accepted values: `double`, `epoch`, `epoch_ms`, `iso8601` (2018-05-30T09:39:52.000681Z), `_java_sql_timestamp_` (2018-05-30 09:39:52.000681). | _none_ | | `json_date_key` | Specify the name of the date key in the output record. To disable the time key, set the value to `false`. | `date` | -| `log_key` | By default, the whole log record is sent to S3. When specifying a key name with this option, only the value of that key is sent to S3. For example, when using Docker you can specify `log_key log` and only the log message is sent to S3. | _none_ | +| `log_key` | By default, the whole log record is sent to S3. When specifying a key name with this option, only the value of that key is sent to S3. For example, when using Docker you can specify `log_key log` and only the log message is sent to S3. The key name must match exactly. See [Key matching for `log_key`](#key-matching-for-log_key). | _none_ | | `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | | `log_suppress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` means no suppression. | `0` | | `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | @@ -115,6 +115,13 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor | `use_put_object` | Use the S3 `PutObject` API instead of the multipart upload API. When enabled, the key extension is only available when `$UUID` is specified in `s3_key_format`. If `$UUID` isn't included, a random string appends format string and the key extension can't be customized. | `false` | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | +### Key matching for `log_key` + +A record key matches `log_key` only when the two names are identical. + +In Fluent Bit versions earlier than 5.1, the comparison stopped at the length of the record key, so a record key that was a prefix of the configured value also matched. With `log_key log_level`, a record containing a `log` key matched on `log`, and Fluent Bit uploaded the value of `log` instead of `log_level`. + +If a record has no key matching `log_key`, Fluent Bit logs `Could not find log_key '' in records` and uploads nothing for those records. After upgrading, check for this error in your logs. It indicates a configured `log_key` that was previously matching a different field by prefix. ## TLS / SSL