Skip to content

feat(stats)!: add telemetry for per-field cardinality limits - #2227

Open
Eldolfin wants to merge 15 commits into
oscarld/stats-per-key-cardinality-limitfrom
oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator
Open

feat(stats)!: add telemetry for per-field cardinality limits#2227
Eldolfin wants to merge 15 commits into
oscarld/stats-per-key-cardinality-limitfrom
oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator

Conversation

@Eldolfin

Copy link
Copy Markdown
Contributor

What does this PR do?

Send telemetry about per-field cardinality collapses. But unlike #2212, now using a single count metric with all the offending fields in the bucket instead of separate count metrics.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

@Eldolfin
Eldolfin changed the base branch from main to oscarld/stats-per-key-cardinality-limit July 10, 2026 15:15
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 10, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 97.25%
Overall Coverage: 74.74% (+0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 568df74 | Docs | Datadog PR Page | Give us feedback!

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 4476 documentation warning(s) found

📦 datadog-ipc - 1003 warning(s)

📦 libdd-data-pipeline-ffi - 1343 warning(s)

📦 libdd-data-pipeline - 1232 warning(s)

📦 libdd-trace-stats - 898 warning(s)


Updated: 2026-07-28 14:23:49 UTC | Commit: 3994fca | missing-docs job results

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 11 issue(s) found, showing only errors (advisories, bans, sources)

📦 datadog-ipc - 3 error(s)

Show output
error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:25:1
   │
25 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── datadog-ipc v0.1.0

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:66:1
   │
66 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── (dev) datadog-ipc v0.1.0
                     ├── libdd-common v5.1.0
                     │   ├── datadog-ipc v0.1.0 (*)
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │       └── datadog-ipc v0.1.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── datadog-ipc v0.1.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:205:1
    │
205 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v5.1.0
      │   ├── datadog-ipc v0.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │       └── datadog-ipc v0.1.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── datadog-ipc v0.1.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── datadog-ipc v0.1.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

advisories FAILED, bans ok, sources ok

📦 libdd-data-pipeline-ffi - 3 error(s)

Show output
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:78:1
   │
78 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-data-pipeline v7.0.0
                     │   │   │   └── libdd-data-pipeline-ffi v38.0.0
                     │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-data-pipeline v7.0.0 (*)
                     │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-common-ffi v38.0.0
                     │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-data-pipeline v7.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:226:1
    │
226 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-data-pipeline v7.0.0
      │   │   │   └── libdd-data-pipeline-ffi v38.0.0
      │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   │   ├── libdd-dogstatsd-client v4.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-data-pipeline v7.0.0 (*)
      │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-common-ffi v38.0.0
      │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-data-pipeline v7.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── libdd-data-pipeline v7.0.0 (*)
              ├── libdd-data-pipeline-ffi v38.0.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:296:1
    │
296 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v7.0.0
                  └── libdd-data-pipeline-ffi v38.0.0

advisories FAILED, bans ok, sources ok

📦 libdd-data-pipeline - 3 error(s)

Show output
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:75:1
   │
75 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-data-pipeline v7.0.0
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-data-pipeline v7.0.0 (*)
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-data-pipeline v7.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:218:1
    │
218 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-data-pipeline v7.0.0
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-dogstatsd-client v4.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-data-pipeline v7.0.0 (*)
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-data-pipeline v7.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── libdd-data-pipeline v7.0.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:287:1
    │
287 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v7.0.0

advisories FAILED, bans ok, sources ok

📦 libdd-trace-stats - 2 error(s)

Show output
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:66:1
   │
66 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:198:1
    │
198 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0
      │   │   └── libdd-trace-stats v6.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

advisories FAILED, bans ok, sources ok

Updated: 2026-07-28 14:25:04 UTC | Commit: 3994fca | dependency-check job results

@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator branch from 456c9bf to 295db59 Compare July 21, 2026 12:01
@pr-commenter

pr-commenter Bot commented Jul 21, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-07-28 14:34:03

Comparing candidate commit 568df74 in PR branch oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator with baseline commit 37ce354 in branch oscarld/stats-per-key-cardinality-limit.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 22 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:trace_buffer/4_senders/no_delay

  • 🟩 execution_time [-150.724µs; -122.983µs] or [-6.109%; -4.985%]
  • 🟩 throughput [+77557.384op/s; +95384.041op/s] or [+5.312%; +6.533%]

Benchmark execution time: 2026-07-28 14:30:38

Comparing candidate commit 568df74 in PR branch oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator with baseline commit 37ce354 in branch oscarld/stats-per-key-cardinality-limit.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 568df74 1785248457 oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
trace_buffer/1_senders/10us_delay execution_time 56.290ms 56.542ms ± 0.180ms 56.492ms ± 0.078ms 56.592ms 56.830ms 57.318ms 57.589ms 1.94% 2.509 9.259 0.32% 0.013ms 1 200
trace_buffer/1_senders/10us_delay throughput 15627.922op/s 15917.509op/s ± 50.249op/s 15931.477op/s ± 22.102op/s 15946.554op/s 15971.588op/s 15979.064op/s 15988.652op/s 0.36% -2.462 8.939 0.31% 3.553op/s 1 200
trace_buffer/1_senders/1us_delay execution_time 49.916ms 50.080ms ± 0.035ms 50.078ms ± 0.012ms 50.088ms 50.149ms 50.187ms 50.205ms 0.25% 0.350 3.307 0.07% 0.003ms 1 200
trace_buffer/1_senders/1us_delay throughput 17926.611op/s 17971.380op/s ± 12.710op/s 17972.107op/s ± 4.418op/s 17977.213op/s 17988.113op/s 17991.240op/s 18030.303op/s 0.32% -0.339 3.324 0.07% 0.899op/s 1 200
trace_buffer/1_senders/no_delay execution_time 337.815µs 345.821µs ± 5.019µs 345.250µs ± 1.470µs 346.827µs 350.967µs 361.335µs 396.476µs 14.84% 5.618 51.471 1.45% 0.355µs 1 200
trace_buffer/1_senders/no_delay throughput 2269996.579op/s 2603007.919op/s ± 35144.915op/s 2606809.458op/s ± 11150.003op/s 2616994.479op/s 2642655.158op/s 2662544.520op/s 2664182.797op/s 2.20% -4.750 40.114 1.35% 2485.121op/s 1 200
trace_buffer/2_senders/10us_delay execution_time 56.511ms 56.966ms ± 0.248ms 56.908ms ± 0.148ms 57.097ms 57.466ms 57.624ms 57.759ms 1.50% 0.905 0.328 0.43% 0.018ms 1 200
trace_buffer/2_senders/10us_delay throughput 31163.773op/s 31598.610op/s ± 136.795op/s 31630.029op/s ± 82.319op/s 31709.800op/s 31760.915op/s 31794.535op/s 31852.178op/s 0.70% -0.886 0.282 0.43% 9.673op/s 1 200
trace_buffer/2_senders/1us_delay execution_time 50.007ms 50.129ms ± 0.173ms 50.108ms ± 0.022ms 50.131ms 50.206ms 50.264ms 52.475ms 4.72% 12.597 168.063 0.34% 0.012ms 1 200
trace_buffer/2_senders/1us_delay throughput 34301.764op/s 35907.538op/s ± 118.759op/s 35922.311op/s ± 15.662op/s 35937.647op/s 35949.214op/s 35953.170op/s 35995.119op/s 0.20% -12.478 165.858 0.33% 8.398op/s 1 200
trace_buffer/2_senders/no_delay execution_time 842.488µs 900.748µs ± 20.226µs 901.934µs ± 12.887µs 913.387µs 931.067µs 940.371µs 1003.231µs 11.23% 0.429 2.621 2.24% 1.430µs 1 200
trace_buffer/2_senders/no_delay throughput 1794203.725op/s 1999335.212op/s ± 44605.796op/s 1995711.192op/s ± 28409.611op/s 2028083.549op/s 2068552.499op/s 2105020.581op/s 2136529.229op/s 7.06% -0.157 1.769 2.23% 3154.106op/s 1 200
trace_buffer/4_senders/10us_delay execution_time 57.154ms 57.454ms ± 0.161ms 57.433ms ± 0.102ms 57.549ms 57.740ms 57.867ms 58.030ms 1.04% 0.600 0.374 0.28% 0.011ms 1 200
trace_buffer/4_senders/10us_delay throughput 62036.888op/s 62659.256op/s ± 175.599op/s 62681.257op/s ± 111.358op/s 62780.423op/s 62920.879op/s 62980.738op/s 62987.192op/s 0.49% -0.583 0.338 0.28% 12.417op/s 1 200
trace_buffer/4_senders/1us_delay execution_time 50.086ms 50.168ms ± 0.035ms 50.161ms ± 0.018ms 50.182ms 50.237ms 50.269ms 50.317ms 0.31% 1.248 2.411 0.07% 0.002ms 1 200
trace_buffer/4_senders/1us_delay throughput 71545.844op/s 71758.822op/s ± 49.750op/s 71769.245op/s ± 25.147op/s 71791.795op/s 71819.125op/s 71835.680op/s 71877.034op/s 0.15% -1.243 2.390 0.07% 3.518op/s 1 200
trace_buffer/4_senders/no_delay execution_time 2.096ms 2.330ms ± 0.078ms 2.337ms ± 0.052ms 2.381ms 2.451ms 2.487ms 2.505ms 7.16% -0.198 -0.328 3.33% 0.006ms 1 200
trace_buffer/4_senders/no_delay throughput 1437217.116op/s 1546580.862op/s ± 52156.347op/s 1540150.110op/s ± 34903.989op/s 1584700.008op/s 1633698.517op/s 1674335.056op/s 1717706.097op/s 11.53% 0.366 -0.143 3.36% 3688.011op/s 1 200
trace_buffer/8_senders/10us_delay execution_time 57.623ms 57.936ms ± 0.227ms 57.867ms ± 0.132ms 58.079ms 58.383ms 58.508ms 58.536ms 1.16% 0.854 -0.272 0.39% 0.016ms 1 200
trace_buffer/8_senders/10us_delay throughput 123001.955op/s 124276.624op/s ± 484.389op/s 124423.337op/s ± 283.938op/s 124659.294op/s 124822.797op/s 124936.359op/s 124950.087op/s 0.42% -0.843 -0.296 0.39% 34.251op/s 1 200
trace_buffer/8_senders/1us_delay execution_time 50.201ms 50.291ms ± 0.062ms 50.273ms ± 0.027ms 50.309ms 50.418ms 50.492ms 50.607ms 0.66% 1.827 4.566 0.12% 0.004ms 1 200
trace_buffer/8_senders/1us_delay throughput 142272.251op/s 143167.768op/s ± 177.420op/s 143217.889op/s ± 78.036op/s 143289.217op/s 143350.422op/s 143406.565op/s 143423.001op/s 0.14% -1.815 4.497 0.12% 12.546op/s 1 200
trace_buffer/8_senders/no_delay execution_time 6.384ms 6.567ms ± 0.066ms 6.571ms ± 0.042ms 6.607ms 6.673ms 6.704ms 6.720ms 2.27% -0.159 -0.016 1.00% 0.005ms 1 200
trace_buffer/8_senders/no_delay throughput 1071478.584op/s 1096578.613op/s ± 10967.082op/s 1095763.562op/s ± 7085.029op/s 1104507.291op/s 1117194.424op/s 1122646.933op/s 1127762.195op/s 2.92% 0.218 0.011 1.00% 775.490op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
trace_buffer/1_senders/10us_delay execution_time [56.517ms; 56.567ms] or [-0.044%; +0.044%] None None None
trace_buffer/1_senders/10us_delay throughput [15910.545op/s; 15924.473op/s] or [-0.044%; +0.044%] None None None
trace_buffer/1_senders/1us_delay execution_time [50.075ms; 50.085ms] or [-0.010%; +0.010%] None None None
trace_buffer/1_senders/1us_delay throughput [17969.618op/s; 17973.141op/s] or [-0.010%; +0.010%] None None None
trace_buffer/1_senders/no_delay execution_time [345.126µs; 346.517µs] or [-0.201%; +0.201%] None None None
trace_buffer/1_senders/no_delay throughput [2598137.172op/s; 2607878.666op/s] or [-0.187%; +0.187%] None None None
trace_buffer/2_senders/10us_delay execution_time [56.931ms; 57.000ms] or [-0.060%; +0.060%] None None None
trace_buffer/2_senders/10us_delay throughput [31579.652op/s; 31617.569op/s] or [-0.060%; +0.060%] None None None
trace_buffer/2_senders/1us_delay execution_time [50.105ms; 50.153ms] or [-0.048%; +0.048%] None None None
trace_buffer/2_senders/1us_delay throughput [35891.080op/s; 35923.997op/s] or [-0.046%; +0.046%] None None None
trace_buffer/2_senders/no_delay execution_time [897.945µs; 903.551µs] or [-0.311%; +0.311%] None None None
trace_buffer/2_senders/no_delay throughput [1993153.278op/s; 2005517.146op/s] or [-0.309%; +0.309%] None None None
trace_buffer/4_senders/10us_delay execution_time [57.432ms; 57.476ms] or [-0.039%; +0.039%] None None None
trace_buffer/4_senders/10us_delay throughput [62634.919op/s; 62683.592op/s] or [-0.039%; +0.039%] None None None
trace_buffer/4_senders/1us_delay execution_time [50.163ms; 50.173ms] or [-0.010%; +0.010%] None None None
trace_buffer/4_senders/1us_delay throughput [71751.927op/s; 71765.717op/s] or [-0.010%; +0.010%] None None None
trace_buffer/4_senders/no_delay execution_time [2.320ms; 2.341ms] or [-0.463%; +0.463%] None None None
trace_buffer/4_senders/no_delay throughput [1539352.494op/s; 1553809.230op/s] or [-0.467%; +0.467%] None None None
trace_buffer/8_senders/10us_delay execution_time [57.905ms; 57.968ms] or [-0.054%; +0.054%] None None None
trace_buffer/8_senders/10us_delay throughput [124209.493op/s; 124343.756op/s] or [-0.054%; +0.054%] None None None
trace_buffer/8_senders/1us_delay execution_time [50.282ms; 50.299ms] or [-0.017%; +0.017%] None None None
trace_buffer/8_senders/1us_delay throughput [143143.179op/s; 143192.356op/s] or [-0.017%; +0.017%] None None None
trace_buffer/8_senders/no_delay execution_time [6.557ms; 6.576ms] or [-0.138%; +0.138%] None None None
trace_buffer/8_senders/no_delay throughput [1095058.681op/s; 1098098.545op/s] or [-0.139%; +0.139%] None None None

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 568df74 1785248457 oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 15.408µs 15.781µs ± 0.144µs 15.752µs ± 0.097µs 15.875µs 16.032µs 16.182µs 16.232µs 3.05% 0.723 0.381 0.91% 0.010µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [15.761µs; 15.801µs] or [-0.127%; +0.127%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 568df74 1785248457 oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.599ms 9.629ms ± 0.014ms 9.627ms ± 0.010ms 9.638ms 9.658ms 9.667ms 9.675ms 0.50% 0.738 0.287 0.15% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.627ms; 9.631ms] or [-0.021%; +0.021%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 568df74 1785248457 oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.073µs 1.103µs ± 0.012µs 1.100µs ± 0.008µs 1.114µs 1.122µs 1.127µs 1.131µs 2.75% 0.185 -0.622 1.08% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [1.102µs; 1.105µs] or [-0.151%; +0.151%] None None None

Baseline

Baseline benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 37ce354 1785248417 oscarld/stats-per-key-cardinality-limit
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
trace_buffer/1_senders/10us_delay execution_time 56.289ms 56.512ms ± 0.112ms 56.486ms ± 0.066ms 56.581ms 56.737ms 56.806ms 56.865ms 0.67% 0.633 0.289 0.20% 0.008ms 1 200
trace_buffer/1_senders/10us_delay throughput 15826.849op/s 15925.957op/s ± 31.498op/s 15933.193op/s ± 18.528op/s 15945.730op/s 15970.392op/s 15987.374op/s 15988.911op/s 0.35% -0.622 0.273 0.20% 2.227op/s 1 200
trace_buffer/1_senders/1us_delay execution_time 49.930ms 50.077ms ± 0.027ms 50.079ms ± 0.010ms 50.087ms 50.133ms 50.151ms 50.154ms 0.15% -0.370 5.013 0.05% 0.002ms 1 200
trace_buffer/1_senders/1us_delay throughput 17944.599op/s 17972.229op/s ± 9.676op/s 17971.535op/s ± 3.767op/s 17976.212op/s 17987.680op/s 17990.135op/s 18025.296op/s 0.30% 0.382 5.053 0.05% 0.684op/s 1 200
trace_buffer/1_senders/no_delay execution_time 335.899µs 343.430µs ± 4.421µs 343.230µs ± 1.685µs 344.840µs 347.606µs 351.813µs 391.089µs 13.94% 6.347 65.943 1.28% 0.313µs 1 200
trace_buffer/1_senders/no_delay throughput 2301269.251op/s 2621023.411op/s ± 31385.211op/s 2622145.003op/s ± 12806.703op/s 2636648.402op/s 2659955.546op/s 2671596.741op/s 2679376.207op/s 2.18% -5.355 52.338 1.19% 2219.270op/s 1 200
trace_buffer/2_senders/10us_delay execution_time 56.538ms 56.879ms ± 0.200ms 56.841ms ± 0.103ms 56.952ms 57.291ms 57.531ms 57.668ms 1.46% 1.369 2.196 0.35% 0.014ms 1 200
trace_buffer/2_senders/10us_delay throughput 31213.113op/s 31646.743op/s ± 110.923op/s 31667.406op/s ± 57.353op/s 31721.536op/s 31778.934op/s 31807.423op/s 31837.262op/s 0.54% -1.345 2.111 0.35% 7.843op/s 1 200
trace_buffer/2_senders/1us_delay execution_time 50.054ms 50.106ms ± 0.031ms 50.100ms ± 0.016ms 50.117ms 50.173ms 50.199ms 50.224ms 0.25% 1.286 1.700 0.06% 0.002ms 1 200
trace_buffer/2_senders/1us_delay throughput 35839.649op/s 35923.530op/s ± 22.071op/s 35928.365op/s ± 11.225op/s 35939.015op/s 35947.927op/s 35957.575op/s 35961.170op/s 0.09% -1.282 1.688 0.06% 1.561op/s 1 200
trace_buffer/2_senders/no_delay execution_time 871.131µs 925.232µs ± 15.659µs 926.891µs ± 8.654µs 935.016µs 947.326µs 956.375µs 969.747µs 4.62% -0.718 1.223 1.69% 1.107µs 1 200
trace_buffer/2_senders/no_delay throughput 1856153.809op/s 1946019.838op/s ± 33379.329op/s 1941976.034op/s ± 17973.310op/s 1961591.542op/s 2010722.756op/s 2044459.570op/s 2066278.300op/s 6.40% 0.856 1.432 1.71% 2360.275op/s 1 200
trace_buffer/4_senders/10us_delay execution_time 57.027ms 57.394ms ± 0.194ms 57.358ms ± 0.096ms 57.459ms 57.754ms 57.909ms 58.646ms 2.25% 2.021 8.311 0.34% 0.014ms 1 200
trace_buffer/4_senders/10us_delay throughput 61384.754op/s 62725.395op/s ± 210.721op/s 62764.038op/s ± 105.002op/s 62848.092op/s 62974.735op/s 63019.915op/s 63127.916op/s 0.58% -1.959 7.830 0.34% 14.900op/s 1 200
trace_buffer/4_senders/1us_delay execution_time 50.076ms 50.171ms ± 0.041ms 50.163ms ± 0.017ms 50.181ms 50.248ms 50.310ms 50.424ms 0.52% 2.064 7.948 0.08% 0.003ms 1 200
trace_buffer/4_senders/1us_delay throughput 71393.994op/s 71755.194op/s ± 58.674op/s 71766.629op/s ± 24.556op/s 71789.277op/s 71817.984op/s 71835.960op/s 71890.346op/s 0.17% -2.051 7.852 0.08% 4.149op/s 1 200
trace_buffer/4_senders/no_delay execution_time 2.264ms 2.467ms ± 0.063ms 2.472ms ± 0.036ms 2.508ms 2.550ms 2.586ms 2.743ms 10.95% -0.280 2.041 2.54% 0.004ms 1 200
trace_buffer/4_senders/no_delay throughput 1312359.673op/s 1460110.150op/s ± 37630.184op/s 1456064.225op/s ± 21282.572op/s 1480499.682op/s 1538917.194op/s 1564202.765op/s 1590213.674op/s 9.21% 0.565 1.833 2.57% 2660.856op/s 1 200
trace_buffer/8_senders/10us_delay execution_time 57.647ms 57.947ms ± 0.241ms 57.847ms ± 0.094ms 58.100ms 58.453ms 58.630ms 58.745ms 1.55% 1.194 0.571 0.42% 0.017ms 1 200
trace_buffer/8_senders/10us_delay throughput 122562.876op/s 124253.698op/s ± 515.228op/s 124467.060op/s ± 202.779op/s 124620.847op/s 124804.324op/s 124866.087op/s 124897.092op/s 0.35% -1.179 0.525 0.41% 36.432op/s 1 200
trace_buffer/8_senders/1us_delay execution_time 50.193ms 50.282ms ± 0.048ms 50.270ms ± 0.027ms 50.308ms 50.374ms 50.421ms 50.490ms 0.44% 1.066 1.552 0.09% 0.003ms 1 200
trace_buffer/8_senders/1us_delay throughput 142603.312op/s 143192.747op/s ± 136.052op/s 143226.954op/s ± 75.769op/s 143280.736op/s 143362.983op/s 143421.274op/s 143445.420op/s 0.15% -1.059 1.527 0.09% 9.620op/s 1 200
trace_buffer/8_senders/no_delay execution_time 5.534ms 6.484ms ± 0.176ms 6.502ms ± 0.092ms 6.597ms 6.678ms 6.707ms 6.750ms 3.81% -2.230 8.133 2.71% 0.012ms 1 200
trace_buffer/8_senders/no_delay throughput 1066638.750op/s 1111383.234op/s ± 32371.082op/s 1107266.866op/s ± 15644.856op/s 1120876.274op/s 1161285.097op/s 1270846.200op/s 1301027.143op/s 17.50% 2.696 11.183 2.91% 2288.981op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
trace_buffer/1_senders/10us_delay execution_time [56.496ms; 56.527ms] or [-0.027%; +0.027%] None None None
trace_buffer/1_senders/10us_delay throughput [15921.591op/s; 15930.322op/s] or [-0.027%; +0.027%] None None None
trace_buffer/1_senders/1us_delay execution_time [50.074ms; 50.081ms] or [-0.007%; +0.007%] None None None
trace_buffer/1_senders/1us_delay throughput [17970.888op/s; 17973.570op/s] or [-0.007%; +0.007%] None None None
trace_buffer/1_senders/no_delay execution_time [342.817µs; 344.043µs] or [-0.178%; +0.178%] None None None
trace_buffer/1_senders/no_delay throughput [2616673.723op/s; 2625373.100op/s] or [-0.166%; +0.166%] None None None
trace_buffer/2_senders/10us_delay execution_time [56.851ms; 56.906ms] or [-0.049%; +0.049%] None None None
trace_buffer/2_senders/10us_delay throughput [31631.370op/s; 31662.116op/s] or [-0.049%; +0.049%] None None None
trace_buffer/2_senders/1us_delay execution_time [50.102ms; 50.111ms] or [-0.009%; +0.009%] None None None
trace_buffer/2_senders/1us_delay throughput [35920.471op/s; 35926.589op/s] or [-0.009%; +0.009%] None None None
trace_buffer/2_senders/no_delay execution_time [923.062µs; 927.402µs] or [-0.235%; +0.235%] None None None
trace_buffer/2_senders/no_delay throughput [1941393.784op/s; 1950645.892op/s] or [-0.238%; +0.238%] None None None
trace_buffer/4_senders/10us_delay execution_time [57.367ms; 57.421ms] or [-0.047%; +0.047%] None None None
trace_buffer/4_senders/10us_delay throughput [62696.191op/s; 62754.599op/s] or [-0.047%; +0.047%] None None None
trace_buffer/4_senders/1us_delay execution_time [50.165ms; 50.176ms] or [-0.011%; +0.011%] None None None
trace_buffer/4_senders/1us_delay throughput [71747.063op/s; 71763.326op/s] or [-0.011%; +0.011%] None None None
trace_buffer/4_senders/no_delay execution_time [2.458ms; 2.476ms] or [-0.353%; +0.353%] None None None
trace_buffer/4_senders/no_delay throughput [1454894.968op/s; 1465325.331op/s] or [-0.357%; +0.357%] None None None
trace_buffer/8_senders/10us_delay execution_time [57.913ms; 57.980ms] or [-0.058%; +0.058%] None None None
trace_buffer/8_senders/10us_delay throughput [124182.292op/s; 124325.103op/s] or [-0.057%; +0.057%] None None None
trace_buffer/8_senders/1us_delay execution_time [50.275ms; 50.289ms] or [-0.013%; +0.013%] None None None
trace_buffer/8_senders/1us_delay throughput [143173.891op/s; 143211.602op/s] or [-0.013%; +0.013%] None None None
trace_buffer/8_senders/no_delay execution_time [6.459ms; 6.508ms] or [-0.377%; +0.377%] None None None
trace_buffer/8_senders/no_delay throughput [1106896.914op/s; 1115869.555op/s] or [-0.404%; +0.404%] None None None

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 37ce354 1785248417 oscarld/stats-per-key-cardinality-limit
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 15.381µs 15.827µs ± 0.200µs 15.788µs ± 0.106µs 15.928µs 16.191µs 16.397µs 16.497µs 4.49% 0.869 0.785 1.26% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [15.799µs; 15.855µs] or [-0.175%; +0.175%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 37ce354 1785248417 oscarld/stats-per-key-cardinality-limit
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.636ms 9.667ms ± 0.018ms 9.664ms ± 0.012ms 9.676ms 9.701ms 9.718ms 9.727ms 0.65% 0.912 0.565 0.18% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.665ms; 9.670ms] or [-0.026%; +0.026%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 37ce354 1785248417 oscarld/stats-per-key-cardinality-limit
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.063µs 1.075µs ± 0.006µs 1.074µs ± 0.004µs 1.079µs 1.086µs 1.090µs 1.096µs 2.05% 0.470 -0.172 0.59% 0.000µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [1.074µs; 1.076µs] or [-0.083%; +0.083%] None None None

@Eldolfin

Copy link
Copy Markdown
Contributor Author

@codex review

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.01 MB 8.01 MB +0% (+48 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 87.14 MB 87.27 MB +.15% (+134.98 KB) 🔍
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 98.41 MB 98.55 MB +.14% (+143.48 KB) 🔍
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.77 MB 10.78 MB +.04% (+4.53 KB) 🔍
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.00 MB 26.03 MB +.12% (+33.00 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 89.18 KB 89.60 KB +.47% (+430 B) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 187.93 MB 188.09 MB +.08% (+168.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 979.20 MB 979.99 MB +.08% (+810.72 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.47 MB 8.48 MB +.14% (+12.50 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 89.18 KB 89.60 KB +.47% (+430 B) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.05 MB 25.10 MB +.18% (+48.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 49.85 MB 49.92 MB +.14% (+71.79 KB) 🔍
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 22.64 MB 22.67 MB +.12% (+29.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 90.58 KB 91.00 KB +.46% (+434 B) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 192.63 MB 192.82 MB +.09% (+192.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 968.15 MB 969.15 MB +.10% (+1020.48 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.54 MB 6.55 MB +.19% (+13.00 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 90.58 KB 91.00 KB +.46% (+434 B) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 26.91 MB 26.95 MB +.14% (+40.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 47.44 MB 47.51 MB +.14% (+71.35 KB) 🔍
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 77.74 MB 77.85 MB +.14% (+116.99 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.91 MB 8.93 MB +.17% (+16.06 KB) 🔍
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.24 MB 93.36 MB +.12% (+124.10 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.85 MB 10.87 MB +.17% (+19.28 KB) 🔍

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: e1430135ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Eldolfin Eldolfin changed the title feat(stats)!: add telemetry for per-field cardinality limits V2 feat(stats)!: add telemetry for per-field cardinality limits Jul 21, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 21, 2026

Copy link
Copy Markdown

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

test_collapsed_spans_telemetry dropped its HTTP mock before StatsExporter::send completed. The test now retains the mock for its full lifetime in libdd-trace-stats/src/stats_exporter.rs.

Commit fix to this PR


View in Datadog | Reviewed commit e47093e · Any feedback? Reach out in #deveng-pr-agent

@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator branch 2 times, most recently from bf05f1a to e648bf3 Compare July 22, 2026 11:05
@Eldolfin
Eldolfin marked this pull request as ready for review July 22, 2026 11:17
@Eldolfin
Eldolfin requested review from a team as code owners July 22, 2026 11:17
@Eldolfin

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59adf71d1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libdd-trace-stats/src/span_concentrator/cardinality_limit_telemetry.rs Outdated

#[derive(Debug, Clone, Default, Copy)]
// Note: slot 0 is a counter for non_collapsed spans. It's not used for emitting telemetry
pub struct CollapsedFieldsMetrics(pub(super) [usize; COLLAPSED_FIELD_METRIC_SIZE]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if I understand correctly, you don't count each collapsed field individually, but you count sets of collapsed fields instead. I'm curious about the reason behind it. I would naively expect that counting each field individually would be enough.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes from the cardinality spec. Since we apply tags to increment of a global count we want the global count to be the number of collapsed spans.

This is easy to do if you increment the count when you collapse you just apply the correct tags to the increment. However this means adding a reference to the telemetry/statd client to the SpanConcentrator which is currently "pure computation logic" making it harder to reuse in serverless/sidecar.

This was found as a way to allow the StatsExporter to emit the metric by giving it for each increment the offending fields. However this arguably not much better than adding a telemetry/dogstatsd handle to the concentrator.

Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/cardinality_limit_telemetry.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/cardinality_limit_telemetry.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/mod.rs Outdated
Comment thread datadog-ipc/src/shm_stats.rs

#[derive(Debug, Clone, Default, Copy)]
// Note: slot 0 is a counter for non_collapsed spans. It's not used for emitting telemetry
pub struct CollapsedFieldsMetrics(pub(super) [usize; COLLAPSED_FIELD_METRIC_SIZE]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes from the cardinality spec. Since we apply tags to increment of a global count we want the global count to be the number of collapsed spans.

This is easy to do if you increment the count when you collapse you just apply the correct tags to the increment. However this means adding a reference to the telemetry/statd client to the SpanConcentrator which is currently "pure computation logic" making it harder to reuse in serverless/sidecar.

This was found as a way to allow the StatsExporter to emit the metric by giving it for each increment the offending fields. However this arguably not much better than adding a telemetry/dogstatsd handle to the concentrator.

@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit-telemetry-in-concentrator branch from e4264ec to 568df74 Compare July 28, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants