feat(profiling): add USDT semaphores to skip slow path around heap profile sampler by scottgerring · Pull Request #2266 · DataDog/libdatadog

scottgerring · 2026-07-23T08:09:37Z

When no profiler is attached, we should use USDT semaphores to skip the slow path. This was a helpful suggestion from the profiling SIG. Some DOE numbers without profiler attached:

1 KiB allocations against 512 KiB sample size

At on avg. 1/512 requests sampled, its all in the noise floor:

variant	binary	USDT semaphore addr	time (5M allocs)	vs none	allocs/sec
none (bare System allocator)	no sampler linked	n/a	813 ms	—	6.15M
alloc, no semaphore (crates.io v1)	sampler linked	`0x0` (none)	802 ms	−1.4%	6.23M
alloc, fast-path semaphore (`sgg/usdt-semaphore`)	sampler linked	`0x782ac8`	836 ms	+2.8%	5.98M

64 KiB allocations with 64 KiB mean sample size

When we dial up the sample rate, we do see some cost emerge, but not heaps. The majority is really the allocations themselves:

variant	time	vs none
none	405 ms	—
alloc, no semaphore (100% slow path)	418 ms	+3.2%
alloc, fast-path semaphore	400 ms	−1.2%

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

…tached

github-actions · 2026-07-23T08:17:11Z

📚 Documentation Check Results

⚠️ 94 documentation warning(s) found

📦 `libdd-profiling-heap-allocator` - 47 warning(s)

📦 `libdd-profiling-heap-sampler` - 47 warning(s)

Updated: 2026-07-23 14:31:11 UTC | Commit: 690791d | missing-docs job results

github-actions · 2026-07-23T08:18:38Z

🔒 Cargo Deny Results

⚠️ 1 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-profiling-heap-allocator` - 1 error(s)

Show output

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:21:1
   │
21 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     └── (dev) libdd-profiling-heap-allocator v1.0.0

advisories FAILED, bans ok, sources ok

📦 `libdd-profiling-heap-sampler` - ✅ No issues

Updated: 2026-07-23 14:32:58 UTC | Commit: 690791d | dependency-check job results

datadog-datadog-prod-us1 · 2026-07-23T08:23:22Z

⚠️ Warnings

🚦 5 Pipeline jobs failed

Test | Test x86_64-unknown-linux-gnu on centOS7 docker image

Test | cargo test #ubuntu-latest

Verify libdd-profiling-heap-sampler bindings | Verify libdd-profiling-heap-sampler generated bindings are in sync

View all 5 failed jobs.

🧪 2 Tests failed in 1 job

Test | cross-centos7

allocator::tests::lazy_init_populates_tls_on_first_alloc from libdd-profiling-heap-allocator

thread &#39;allocator::tests::lazy_init_populates_tls_on_first_alloc&#39; panicked at libdd-profiling-heap-allocator/src/allocator.rs:175:10

thread &#39;allocator::tests::lazy_init_populates_tls_on_first_alloc&#39; panicked at libdd-profiling-heap-allocator/src/allocator.rs:175:10:
called \`Result::unwrap()\` on an \`Err\` value: Any { .. }

tests::requested_initializes_tls_on_first_use from libdd-profiling-heap-sampler

thread &#39;tests::requested_initializes_tls_on_first_use&#39; panicked at libdd-profiling-heap-sampler/src/lib.rs:223:10

thread &#39;tests::requested_initializes_tls_on_first_use&#39; panicked at libdd-profiling-heap-sampler/src/lib.rs:223:10:
called \`Result::unwrap()\` on an \`Err\` value: Any { .. }

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5e9cf0b | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-07-23T08:32:12Z

Benchmarks

Comparison

Candidate

Candidate benchmark details

Baseline

Baseline benchmark details

dd-octo-sts · 2026-07-23T08:50:47Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	88.18 MB	88.18 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.13 MB	8.13 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	99.46 MB	99.46 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.93 MB	10.93 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.59 MB	26.59 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.42 MB	191.39 MB	--.01% (-32.00 KB) 💪
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	1.04 GB	1.04 GB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.66 MB	8.66 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.36 MB	25.36 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	50.56 MB	50.56 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	23.16 MB	23.16 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	196.09 MB	196.07 MB	--.01% (-24.00 KB) 💪
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	1.03 GB	1.03 GB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.68 MB	6.68 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	27.26 MB	27.26 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	48.13 MB	48.13 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	78.65 MB	78.65 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.06 MB	9.06 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	94.14 MB	94.14 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.02 MB	11.02 MB	0% (0 B) 👌

feat(heap-profiling): add USDT semaphores to skip slow path when unat…

e746345

…tached

scottgerring force-pushed the sgg/usdt-semaphore branch from 6f5c645 to e746345 Compare July 23, 2026 08:14

scottgerring force-pushed the sgg/usdt-semaphore branch from de1b18e to e8420ed Compare July 23, 2026 12:03

feat(heap-profiling): move semaphore check into fast path

5e9cf0b

scottgerring force-pushed the sgg/usdt-semaphore branch from e8420ed to 5e9cf0b Compare July 23, 2026 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(profiling): add USDT semaphores to skip slow path around heap profile sampler#2266

feat(profiling): add USDT semaphores to skip slow path around heap profile sampler#2266
scottgerring wants to merge 2 commits into
mainfrom
sgg/usdt-semaphore

scottgerring commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 23, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

pr-commenter Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

scottgerring commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-profiling-heap-allocator - 47 warning(s)

📦 libdd-profiling-heap-sampler - 47 warning(s)

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-profiling-heap-allocator - 1 error(s)

📦 libdd-profiling-heap-sampler - ✅ No issues

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 23, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Candidate

Baseline

Uh oh!

dd-octo-sts Bot commented Jul 23, 2026

Artifact Size Benchmark Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

scottgerring commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

📦 `libdd-profiling-heap-allocator` - 47 warning(s)

📦 `libdd-profiling-heap-sampler` - 47 warning(s)

github-actions Bot commented Jul 23, 2026 •

edited

Loading

📦 `libdd-profiling-heap-allocator` - 1 error(s)

📦 `libdd-profiling-heap-sampler` - ✅ No issues

datadog-datadog-prod-us1 Bot commented Jul 23, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

pr-commenter Bot commented Jul 23, 2026 •

edited

Loading