ml: ensure context is registered for REGEX type

[mirko-lazarevic]

This fix ensures that when the buffer is
flushed, the record will have proper timestamp
and metadata instead of just the "log" field.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes

  • Multiline processing now registers first-line context earlier (including a pre‑concatenation registration when starting fresh) and skips packing metadata for records truncated during processing, preventing metadata loss across continuation boundaries.

• Tests

  • Added regression tests that verify full per‑record metadata (time, stream, file, log) is preserved across multiline flushes, slow arrivals, and truncation/continuation scenarios.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

The multiline parser registers the first-line context earlier when the group's first-line buffer is empty and suppresses metadata packing for truncated content; new unit tests were added to verify that flushed multiline records retain stream and file metadata and that metadata is isolated across truncation boundaries.

Changes

Cohort / File(s)	Change Summary
Multiline context & metadata logic src/multiline/flb_ml.c	Invoke flb_ml_register_context() earlier when stream_group->mp_sbuf.size == 0 (including the ENDSWITH path) and make metadata packing conditional on !truncated in addition to existing guards, preventing packing when content is truncated.
Multiline metadata regression tests tests/internal/multiline.c	Add tests for issue 10576: introduce metadata_result, flush_callback_metadata_check, append_log_with_metadata, test_issue_10576, and test_issue_truncation_10576; register both tests in TEST_LIST to assert per-record stream/file metadata presence and behavior across truncation boundaries.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐇 I hop through streams where fragments stay,
I plant the map when the first bytes play.
If truncation bites and lines divide,
I keep your tags on the continuing side.
A rabbit keeps metadata safe today.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'ml: ensure context is registered for REGEX type' is specific and directly related to the main functional change (context registration in multiline processing), though it uses technical terminology that may not convey the full impact (metadata preservation during flush/truncation).
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 79afb58 and 00f16ba.

📒 Files selected for processing (2)

• src/multiline/flb_ml.c
• tests/internal/multiline.c

🚧 Files skipped from review as they are similar to previous changes (1)

• src/multiline/flb_ml.c

🧰 Additional context used

🧠 Learnings (1)

📓 Common learnings

Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

🧬 Code graph analysis (1)

tests/internal/multiline.c (2)

src/multiline/flb_ml.c (1)

• flb_ml_append_object (764-863)

src/multiline/flb_ml_stream.c (1)

• flb_ml_stream_create (223-276)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)

• GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
• GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
• GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
• GitHub Check: PR - fuzzing test
• GitHub Check: pr-compile-centos-7
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
• GitHub Check: pr-compile-without-cxx (3.31.6)

🔇 Additional comments (5)

tests/internal/multiline.c (5)

1646-1663: LGTM: Well-designed helper structure for metadata tracking.

The metadata_result struct effectively tracks both counts (records with/without metadata) and per-record metadata values for verification. The fixed array sizes (10 entries) are adequate for the current tests (6 and 2 records respectively) and the code includes bounds checking before array access.

---

1671-1751: LGTM: Robust metadata verification callback.

The callback correctly:

• Unpacks msgpack records and verifies timestamps
• Safely extracts and copies metadata fields with bounds checking (if (res->current_record < 10))
• Uses proper string handling (strncpy with explicit length and null termination)
• Provides diagnostic output for debugging
• Tracks both successful and missing metadata cases

---

1756-1815: LGTM: Well-encapsulated test helper.

The append_log_with_metadata function cleanly synthesizes msgpack records with custom metadata (stream, log, file fields), properly manages msgpack resources, and correctly invokes the multiline processor. This makes the tests more readable and maintainable.

---

1827-1924: LGTM: Comprehensive test for issue 10576 metadata preservation.

This test effectively validates the fix by:

• Simulating slow log arrival via per-line flushing
• Using realistic regex patterns for timestamp detection
• Verifying all 6 records contain complete metadata (stream + file)
• Confirming no records are missing metadata

The test is well-documented and directly addresses the issue scenario.

---

1942-2067: LGTM: Excellent test for metadata isolation across truncation boundaries.

This test rigorously validates that:

• Truncation correctly flushes with the first group's metadata (stdout/app1.log)
• A new start_state after truncation establishes its own independent metadata (stderr/app2.log)
• No metadata "bleed-over" occurs between groups

The test uses a realistic scenario (buffer limit exceeded) and includes detailed verification with specific metadata value checks.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mirko-lazarevic]

This pull request should address the issue #10576

For the fluent-bit configuration example and steps how to reproduce the issue, navigate to #10576

Output after the fix:

Fluent Bit v4.2.1
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io

______ _                  _    ______ _ _             ___   _____
|  ___| |                | |   | ___ (_) |           /   | / __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| | `' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| |   / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |_./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)_____/

             Fluent Bit v4.2 – Direct Routes Ahead
         Celebrating 10 Years of Open, Fluent Innovation!

[2025/12/01 12:30:43.528267000] [ info] [fluent bit] version=4.2.1, commit=10ebd3a354, pid=6123
[2025/12/01 12:30:43.528771000] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/12/01 12:30:43.528993000] [ info] [simd    ] disabled
[2025/12/01 12:30:43.528998000] [ info] [cmetrics] version=1.0.5
[2025/12/01 12:30:43.529349000] [ info] [ctraces ] version=0.6.6
[2025/12/01 12:30:43.529578000] [ info] [input:tail:tail.0] initializing
[2025/12/01 12:30:43.529585000] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/12/01 12:30:43.530012000] [ info] [input:tail:tail.0] multiline core started
[2025/12/01 12:30:43.530308000] [ info] [input:tail:tail.0] thread instance initialized
[2025/12/01 12:30:43.530546000] [ info] [filter:multiline:ml-detect] created emitter: emitter_for_ml-detect
[2025/12/01 12:30:43.530591000] [ info] [input:emitter:emitter_for_ml-detect] initializing
[2025/12/01 12:30:43.530596000] [ info] [input:emitter:emitter_for_ml-detect] storage_strategy='memory' (memory only)
[2025/12/01 12:30:43.530916000] [ info] [output:stdout:stdout.0] worker #0 started
[2025/12/01 12:30:43.531683000] [ info] [http_server] listen iface=0.0.0.0 tcp_port=8081
[2025/12/01 12:30:43.531917000] [ info] [sp] stream processor started
[2025/12/01 12:30:43.532206000] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2


[2025/12/01 12:30:49.787352000] [ info] [filter:multiline:ml-detect] created new multiline stream for tail.0_kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log
[0] kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log: [[1764588649.296478000, {}], {"time"=>"2025-12-01T12:30:49.296478+01:00", "stream"=>"stdout", "_p"=>"F", "log"=>"Mon Dec  1 11:30:49 UTC 2025 Likely to fail", "file"=>"/Users/mila/dev/tmp/fluent-bit-10576-go-repro/output.log"}]
[0] kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log: [[1764588654.298018000, {}], {"time"=>"2025-12-01T12:30:54.298018+01:00", "stream"=>"stdout", "_p"=>"F", "log"=>"Mon Dec  1 11:30:54 UTC 2025 Likely to fail", "file"=>"/Users/mila/dev/tmp/fluent-bit-10576-go-repro/output.log"}]
[0] kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log: [[1764588659.299245000, {}], {"time"=>"2025-12-01T12:30:59.299245+01:00", "stream"=>"stdout", "_p"=>"F", "log"=>"2025-12-01T11:30:59+00:00 should be ok", "file"=>"/Users/mila/dev/tmp/fluent-bit-10576-go-repro/output.log"}]
[0] kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log: [[1764588667.512873000, {}], {"time"=>"2025-12-01T12:31:07.512873+01:00", "stream"=>"stdout", "_p"=>"F", "log"=>"Mon Dec  1 11:31:07 UTC 2025 Likely to fail", "file"=>"/Users/mila/dev/tmp/fluent-bit-10576-go-repro/output.log"}]
[0] kube.Users.mila.dev.tmp.fluent-bit-10576-go-repro.output.log: [[1764588672.513383999, {}], {"time"=>"2025-12-01T12:31:12.513384+01:00", "stream"=>"stdout", "_p"=>"F", "log"=>"Mon Dec  1 11:31:12 UTC 2025 Likely to fail", "file"=>"/Users/mila/dev/tmp/fluent-bit-10576-go-repro/output.log"}]

[patrick-stephens]

@mirko-lazarevic maybe tweak the commit slightly as having ml: in there is redundant and confusing.

Can you add some unit tests as well? I really like to see those as next time the code is refactored/updated it will prevent a similar problem.

[patrick-stephens]

The CIFuzz failure is down to something else so can be ignored: #11227

[mirko-lazarevic]

@patrick-stephens

@mirko-lazarevic maybe tweak the commit slightly as having ml: in there is redundant and confusing.

I saw exact the same commit message from one of the maintainers, that's why I did the same. Anyway, I removed ml:.

I'll see if I can add some unit tests, although my knowledge in this area is limited.

[mirko-lazarevic]

@mirko-lazarevic maybe tweak the commit slightly as having ml: in there is redundant and confusing.

Can you add some unit tests as well? I really like to see those as next time the code is refactored/updated it will prevent a similar problem.

@patrick-stephens Done

[cosmo0920]

Commit linter still complains one commit:
❌ Commit 01f3911 failed:
Subject prefix 'tests:' does not match files changed.
Expected one of: ml:, tests:
Commit prefix validation failed.
Error: Process completed with exit code 1.

[mirko-lazarevic]

Commit linter still complains one commit: ❌ Commit 01f3911 failed: Subject prefix 'tests:' does not match files changed. Expected one of: ml:, tests: Commit prefix validation failed. Error: Process completed with exit code 1.

@cosmo0920 I hope I did everything right this time. Sorry for the noise.

[cosmo0920]

Unfortunately, no.
We need to use the correct prefixes:

❌ Commit 2a27cbb failed:
Subject prefix 'ml:' does not match files changed.
Expected one of: ml:, tests:
Commit prefix validation failed.

[mirko-lazarevic]

@cosmo0920 I need your help then. In the commit 2a27cbb I edited two files flb_ml.c and /tests/internal/multiline.c. Does my commit message have to contain both prefixes, like this:

ml,tests: handle TRUNCATED return case

Thank you.

[patrick-stephens]

@cosmo0920 I need your help then. In the commit 2a27cbb I edited two files flb_ml.c and /tests/internal/multiline.c. Does my commit message have to contain both prefixes, like this:

ml,tests: handle TRUNCATED return case

Thank you.

The contribution guide typically requires you to have separate commits for different areas so one for ml and one for tests. I think this is why it is complaining.

https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

• One single commit must not include changes to files that are different from the component specified in the subject, e.g: If you are extending flb_utils.c file, the git patch should not touch any other file than flb_utils.c or flb_utils.h.

• One single commit must not include multiple prefixes to specify different areas being touched.

[mirko-lazarevic]

Thanks @patrick-stephens . I split the commit into two separate commits since it violates the contributing guidelines. How can I trigger "Check Commit Message" check ?