feat(tracing): Enable Sentry span streaming mode by phacops · Pull Request #8245 · getsentry/snuba

phacops · 2026-07-30T22:08:17Z

Why

Snuba runs the Sentry Python SDK in transaction mode: spans accumulate in memory and ship as one transaction event when the root span ends. Stream mode (trace_lifecycle="stream") sends them in batches as they finish, which gets us:

No 1000-span cap per transaction.
Flat memory in long-running consumers.
Spans survive crashes — a process killed mid-trace keeps whatever already flushed, instead of losing the whole transaction.

Why it's one big change

Stream mode disables the legacy tracing API, and it fails silently. start_span(), start_transaction() and update_current_span() become no-ops; get_current_span(), scope.span and scope.transaction return None. No exception, no warning at runtime. So every call site has to move to sentry_sdk.traces in the same commit or tracing just goes dark.

The translation: op → sentry.op attribute, description → span name, set_data/set_tag on a span → set_attribute. StreamedSpan genuinely has no op, description, set_data, set_tag, or start_child, so mypy catches most of the sweep.

Tags vs. attributes

Worth a close look in review. Verified empirically against 2.66.1: scope tags and scope attributes reach disjoint destinations — tags land on error events only, attributes on spans only. Since stream mode emits no transaction event, query-path tags had to become attributes to stay queryable on trace data.

But the consumer and error-reporting paths (cli/consumer.py, lw_deletions_consumer.py, consumers/consumer.py, clickhouse/http.py) have no active span at all — no WSGI middleware, no root segment. Converting those would have sent the data nowhere, so they keep set_tag, with a comment explaining why.

Beyond the mechanical translation

Span-name cardinality. Several spans passed raw SQL as description, which becomes the span name — the grouping key — under stream mode. Unbounded cardinality. Now stable names with the SQL in db.query.text, matching what the SDK itself does.
Latent AttributeError. metrics.incr/metrics.timing don't exist in sentry-sdk 2.66.1, so SentryMetricsBackend would raise on any deploy with DOGSTATSD_SOCKET_PATH set. Mapped to count/distribution.
Three pre-existing mypy errors in ClickhouseConnectPool. Previously unreported because the mypy hook only checks changed files; touching that file surfaced them, so they're fixed here.
On-demand profiler deleted. It drives Transaction._profile, which has no stream-mode equivalent. Removed the module, its ondemand_profiler_hostnames option, and the docs section.

Verification

mypy . — clean across all 1079 files (was 3 errors on master).
1986 tests pass with SDK deprecation warnings escalated to errors, which is the real check for missed call sites. Note -p no:warnings in pyproject.toml makes -W error::... inert, so this needed an addopts override — confirmed with a deliberate canary that the guard actually fires before trusting the green.
End-to-end against a capturing transport: a real Flask request emits an http.server segment with the migrated attributes; child spans nest correctly.
Querylog trace_id verified at sample rate 0 (the default) — NoOpStreamedSpan inherits the propagation context's id, so it stays a real 32-hex value. The RPC path now reads it from the propagation context directly, which would otherwise have silently started writing "".

Deploy note

SENTRY_TRACE_SAMPLE_RATE is 0 in every checked-in settings file, so this emits nothing until a deployment sets it (configured in ops, not here). One behavior change to be aware of: the CLI-init transaction used sampled=True to bypass sampling, and stream mode has no equivalent — it now obeys the sample rate like every other span. The snuba_init_time metric is unaffected.

🤖 Generated with Claude Code

Turn on `trace_lifecycle="stream"` so spans are sent to Sentry in batches as they finish, instead of buffering a whole trace in memory and shipping it as a single transaction event. This removes the 1000-span-per-transaction cap, keeps memory flat in long-running consumers, and preserves spans that already finished when a process is killed mid-trace (e.g. an OOM-kill). Stream mode disables the legacy tracing API, and it fails silently: `start_span()`, `start_transaction()` and `update_current_span()` become no-ops, and `get_current_span()` / `scope.span` / `scope.transaction` return None. Every call site therefore has to move to `sentry_sdk.traces` in the same change or tracing goes dark with no error. `op` becomes the `sentry.op` attribute, `description` becomes the span name, and `set_data`/`set_tag` on a span become `set_attribute`. Scope tags and scope attributes reach disjoint destinations: tags land on error events only, attributes on spans only. Query-path tags are converted to attributes so they stay queryable on trace data; the consumer and error-reporting paths keep `set_tag`, since they have no active span and would otherwise lose the data entirely. Three things beyond the mechanical translation: - Several spans passed raw SQL as `description`, which becomes the span name -- the grouping key -- under stream mode. Give them stable names and move the SQL to a `db.query.text` attribute. - `metrics.incr`/`metrics.timing` no longer exist in sentry-sdk 2.66.1, so SentryMetricsBackend was a latent AttributeError. Map them to `count` and `distribution`. - Fix three pre-existing Mapping-vs-dict type errors in ClickhouseConnectPool. They were previously unreported because the mypy pre-commit hook only checks changed files, and this change touches that file. Drop the on-demand profiler: it drives `Transaction._profile`, which has no stream-mode equivalent. Note that `SENTRY_TRACE_SAMPLE_RATE` is 0 in every checked-in settings file, so this emits nothing until a deployment sets it. The CLI-init transaction previously forced `sampled=True`; stream mode has no equivalent, so it now obeys the sample rate like every other span. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5cc1bea. Configure here.}

pbhandari · 2026-07-30T23:29:03Z

+    if not params:
+        return None
+    return dict(params) if isinstance(params, Mapping) else params


I'm not sure I understand the logic behind the else params. It seems to go against what the docstring suggests is the intended behaviour of this function.

pbhandari · 2026-07-30T23:32:01Z

+                if query_id is not None:
+                    span.set_attribute("query_id", query_id)


Suggested change

if query_id is not None:

span.set_attribute("query_id", query_id)

span.set_attribute("query_id", query_id or "unknown-query-id")

or even go back to span.set_attribute("query_id", str(query_id)) so that when looking it's more obvious that query_id is missing.

pbhandari · 2026-07-30T23:34:46Z

+                name=f"INSERT INTO {table}",
+                attributes={
+                    "sentry.op": "db.clickhouse",
+                    sentry_sdk.consts.SPANDATA.DB_SYSTEM: "clickhouse",


Do we also want a sentry_sdk.consts.SPANDATA.DB_QUERY_TEXT: query,?

No particular reason for me to want it, just want to make sure the exclusion was deliberate.

pbhandari · 2026-07-30T23:37:47Z

    for validator_func in VALIDATORS:
        description = getattr(validator_func, "__name__", "custom")
-        with sentry_sdk.start_span(op="validator", description=description):
+        with traces.start_span(name=description, attributes={"sentry.op": "validator"}):


can we pull out the sentry.op into a constant? I feel like I'm seeing it in way too many places.

pbhandari · 2026-07-30T23:44:55Z

+if TYPE_CHECKING:
+    from sentry_sdk._types import Attributes
+


Is this a bug in the sentry_sdk? I feel like reaching into a private file to get the types for an input into start_span isn't the desired flow we want.

(obvs non-blocking)

I don't think we expose this in a public file yet, but definitely makes sense to.
I've opened an issue: getsentry/sentry-python#6970

pbhandari · 2026-07-30T23:51:21Z

+        # sentry_sdk must be >=2.62 for `sentry_sdk.traces` (span streaming API).
+        # Keep the floor in sync with pyproject.toml.
+        additional_dependencies: [ 'fastjsonschema', 'pyyaml', 'sentry_sdk>=2.66.1' ]


Suggested change

# sentry_sdk must be >=2.62 for `sentry_sdk.traces` (span streaming API).

# Keep the floor in sync with pyproject.toml.

additional_dependencies: [ 'fastjsonschema', 'pyyaml', 'sentry_sdk>=2.66.1' ]

# sentry_sdk must be >=2.62 for `sentry_sdk.traces` (span streaming API).

additional_dependencies: [ 'fastjsonschema', 'pyyaml', 'sentry_sdk>=2.66.1' ]

this is almost definitely going to go out of sync with pyproject. no point in leaving that in the comment.

alexander-alderman-webb

Looks good to me overall.
The two bot comments look valid.

The main possibly disruptive change is calling Scope.set_attribute() where there was previously Scope.set_tag(), which includes edits of sentry_sdk.set_tag() -> sentry_sdk.set_attribute().

Keep in mind that attributes on the scope are applied to logs, metrics, and (streamed) spans, whereas tags are applied to exceptions. Completely replacing set_tag() with set_attribute() results in the data not showing up on exceptions while the scope is active.

alexander-alderman-webb · 2026-07-31T11:27:35Z

+def _attributes(tags: Tags | None) -> dict[str, Any] | None:
+    """The SDK takes an invariant ``dict``; ``Tags`` is a ``Mapping``."""
+    return dict(tags) if tags is not None else None


FYI you can clean this up (change type definitions in SentryMetricsBackend).
Just surfacing this, not a big deal or blocking.

Dual-write scope tags and attributes on API error-enrichment paths so values still land on Sentry issues during the tags→attributes transition. Centralize the stream-mode op key as SENTRY_OP, harden querylog trace_id lookup, clarify clickhouse-connect param narrowing, and always set insert query_id on spans.

phacops requested review from a team as code owners July 30, 2026 22:08

cursor Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread snuba/web/views.py Outdated

sentry Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread snuba/web/rpc/storage_routing/routing_strategies/storage_routing.py Outdated

pbhandari reviewed Jul 30, 2026

View reviewed changes

cleptric requested a review from alexander-alderman-webb July 31, 2026 11:08

alexander-alderman-webb mentioned this pull request Jul 31, 2026

Expose Attributes type definition in public place getsentry/sentry-python#6970

Open

alexander-alderman-webb reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(tracing): Enable Sentry span streaming mode - #8245

feat(tracing): Enable Sentry span streaming mode#8245
phacops wants to merge 2 commits into
masterfrom
sentry-span-streaming-snuba

phacops commented Jul 30, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

alexander-alderman-webb Jul 31, 2026

Uh oh!

pbhandari Jul 30, 2026

Uh oh!

alexander-alderman-webb left a comment

Uh oh!

alexander-alderman-webb Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if query_id is not None:
		span.set_attribute("query_id", query_id)

	if query_id is not None:
	span.set_attribute("query_id", query_id)
	span.set_attribute("query_id", query_id or "unknown-query-id")

Uh oh!

Uh oh!

Conversation

phacops commented Jul 30, 2026

Why

Why it's one big change

Tags vs. attributes

Beyond the mechanical translation

Verification

Deploy note

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexander-alderman-webb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants