feat: cache template render counting

[hachall]

No description provided.

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	8e2d483
Status:	✅  Deploy successful!
Preview URL:	https://2350c79f.control-layer.pages.dev
Branch Preview URL:	https://feat-cache-render-counting.control-layer.pages.dev

View logs

[Copilot]

Pull request overview

This PR adds an “exact counting” path for prompt-cache pricing by using tokenizer-svc /v1/render to count chat-template-rendered tokens (including template versioning), and introduces a drift metric comparing those counts to engine-reported usage.prompt_tokens.

Changes:

• Add tokenizer-svc /v1/render client types and error handling, including template-version support from /v1/models.
• Extend prompt parsing to generate structured prefix specs for every cache marker, enabling exact per-breakpoint counting.
• Add render-drift metrics + config flag (cache.render_counting) and plumb render totals into usage injection for observability.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
dwctl/src/prompt_cache/tokenizer.rs	Adds /v1/render request/response types and client method; exposes template_version via /v1/models.
dwctl/src/prompt_cache/stats.rs	Extends cache stats with optional render_total for drift measurement.
dwctl/src/prompt_cache/parse.rs	Builds PrefixSpec/WirePrefix mappings per marker, including telemetry-strip ordinal handling and new unit tests.
dwctl/src/prompt_cache/layer.rs	Updates cache-layer tests to account for new classifier constructor parameter.
dwctl/src/prompt_cache/inject.rs	Injects drift metrics into usage rewriting using render_total and model label.
dwctl/src/prompt_cache/classifier.rs	Adds exact-counting mode, folds template version into index scope, and implements render-count flow + fallbacks/backfills.
dwctl/src/lib.rs	Registers custom histogram buckets for render-drift metric.
dwctl/src/config.rs	Introduces cache.render_counting config option with defaults and rollout guidance.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (3)

dwctl/src/prompt_cache/tokenizer.rs:190

• TokenizerClient::render computes the size bucket by serializing messages/tools to String (and does it twice). This allocates and re-serializes potentially large JSON payloads on the hot path just for metrics.

        let start = std::time::Instant::now();
        let size = cache_metrics::tokenize_size_bucket(messages.to_string().len() + tools.map(|t| t.to_string().len()).unwrap_or(0));

dwctl/src/prompt_cache/classifier.rs:416

• render_counts clones the entire messages JSON array (v.get("messages").cloned()), which can be large. The value can be borrowed from v (or a local empty default) across the async /v1/render call to avoid a potentially expensive deep clone.

        let messages = v.get("messages").cloned().unwrap_or(serde_json::Value::Array(Vec::new()));
        let tools = v.get("tools");

dwctl/src/prompt_cache/inject.rs:236

• The drift metric path allocates a new String for the model label and then clones it for the histogram. This adds per-request allocations on the response hot path; the metrics macros accept &str label values, so this can be allocation-free. Also drift.unsigned_abs() is computed twice.

        let drift = render_total as i64 - prompt as i64;
        let model_label = model.unwrap_or("unknown").to_string();
        metrics::histogram!("dwctl_cache_render_drift_tokens", "model" => model_label.clone()).record(drift as f64);
        // Small constant seams are expected; alarm only on >1% AND >20 tokens, per-model
        // so a single drifting template is visible amid healthy traffic. A zero prompt