filter_wasm: Handle group metadata #11305
Conversation
Signed-off-by: Hiroshi Hatake <[email protected]>
WalkthroughThis change resolves OpenTelemetry metadata loss in the WASM filter by enabling group marker decoding in the log event decoder and implementing logic to preserve group start/end markers through raw byte emission, bypassing normal JSON/msgpack processing. A comprehensive end-to-end OTLP test validates group metadata preservation. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as OTLP Client
participant OTLPIn as OTLP Input
participant Decoder as Log Event Decoder
participant Filter as WASM Filter
participant Encoder as Output Encoder
participant Output as Output Plugin
Note over Decoder: Group marker decoding enabled
Client->>OTLPIn: POST /v1/logs (with group metadata)
OTLPIn->>Decoder: Decode log records
Decoder-->>Decoder: Enable group marker decoding
rect rgb(200, 220, 255)
Note over Decoder,Filter: New: Group Marker Handling Path
Decoder->>Decoder: Detect group start marker
Decoder->>Filter: Pass group marker to filter
Filter->>Encoder: Emit marker as raw bytes
Encoder->>Output: Preserve group boundary
end
rect rgb(220, 220, 220)
Note over Decoder,Filter: Existing: Normal Record Path
Decoder->>Filter: Decode normal log record
Filter->>Filter: Apply WASM logic
Filter->>Encoder: Emit processed record
Encoder->>Output: Send to output plugin
end
rect rgb(200, 220, 255)
Note over Decoder,Filter: New: Group Marker Handling Path
Decoder->>Decoder: Detect group end marker
Decoder->>Filter: Pass group marker to filter
Filter->>Encoder: Emit marker as raw bytes
Encoder->>Output: Preserve group boundary
end
Output->>Client: OTLP metadata and records preserved
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
7405c2c to
bb51057
Compare
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
bb51057 to
0bc837f
Compare
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
0bc837f to
45926d2
Compare
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
45926d2 to
8d56a96
Compare
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
8d56a96 to
5be26d7
Compare
Signed-off-by: Hiroshi Hatake <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-handle-group-metadata-on-filter_wasm
branch
from
5be26d7 to
d4bb848
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/runtime/filter_wasm.c (1)
87-101: Consider logging realloc failure.The realloc failure is silently ignored on line 94. While the code safely handles it by continuing with the old buffer, logging a warning would help diagnose issues during test failures.
🔎 Proposed enhancement
tmp = flb_realloc(mp_output, mp_output_size + size); if (tmp) { mp_output = tmp; memcpy(mp_output + mp_output_size, data, size); mp_output_size += size; } + else { + flb_warn("Failed to realloc msgpack output buffer"); + }
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
plugins/filter_wasm/filter_wasm.ctests/runtime/filter_wasm.c
🧰 Additional context used
🧠 Learnings (10)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-11-21T06:23:29.770Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:24:44.797Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-09-14T09:46:09.531Z
Learnt from: aminvakil
Repo: fluent/fluent-bit PR: 10844
File: conf/fluent-bit:13-15
Timestamp: 2025-09-14T09:46:09.531Z
Learning: For fluent-bit Debian packaging, /opt/fluent-bit/bin/ is the appropriate installation path since the package may be installed from non-official Debian sources, making /opt compliant with FHS for optional software packages.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
tests/runtime/filter_wasm.c
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.
Applied to files:
tests/runtime/filter_wasm.c
🔇 Additional comments (10)
plugins/filter_wasm/filter_wasm.c (4)
36-36: LGTM!Standard header inclusion for
int32_ttype used in the record_type variable.
59-59: LGTM!The
record_typevariable is properly declared to store the log event type (normal vs. group markers).
88-95: Good error handling for group marker enablement.The code properly enables group marker decoding and handles failure gracefully by cleaning up resources and returning
FLB_FILTER_NOTOUCH. The comment clearly documents the intent.
112-127: Excellent implementation of group marker preservation.The logic correctly identifies group start/end markers and emits them as raw bytes, bypassing the WASM filter processing. This ensures OpenTelemetry group boundaries are preserved intact. The safe default initialization of
record_typeand the use ofcontinueto skip normal processing are both well-designed.tests/runtime/filter_wasm.c (6)
22-22: LGTM!The added includes support the new HTTP client infrastructure and integer handling needed for the OTLP test.
Also applies to: 25-25
195-226: Well-designed msgpack navigation helpers.These utility functions properly handle NULL checks, type validation, and provide a clean API for navigating nested msgpack structures in the test assertions.
228-283: Excellent error handling in HTTP client setup.The
http_client_ctx_createfunction demonstrates thorough error handling with proper cleanup at each failure point, preventing resource leaks in the test infrastructure.
285-312: LGTM!The cleanup logic follows the correct destruction order (reverse of creation) and includes defensive NULL checks and pointer clearing.
910-911: LGTM!The test is properly registered with a descriptive name in the test suite.
692-901: Test thoroughly validates OpenTelemetry log event group preservation through WASM filter.The test demonstrates a complete end-to-end flow: OTLP v1/logs HTTP ingestion → WASM filter application → lib output capture with raw msgpack deserialization. It validates that group start/end markers and record metadata (service.name, scope.name) are properly preserved through the filter pipeline. The timestamp decoding logic correctly handles the msgpack ext type format with big-endian byte order for the 4 MSBs (seconds) as a uint32, and properly interprets the special group marker constants (FLB_LOG_EVENT_GROUP_START = -1, FLB_LOG_EVENT_GROUP_END = -2) defined in flb_log_event.h.
Currently, filter_wasm only handles ordinary metadata of logs. This should be sick for handling oltp style of group metadata.
This patch starts to handle oltp style of group metadata.
Closes #11302.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
$ curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"filter-service"}}]},"scopeLogs":[{"scope":{"name":"my.scope"},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"}}]}]}]}' http://0.0.0.0:4318/v1/logsleaks command result:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
Release Notes
✏️ Tip: You can customize this high-level summary in your review settings.