Skip to content

feat(libsy): render the judge payload as one condensed block under a char budget #349

Description

@gburachas

Problem

With recent_turn_window set, the classifier judge receives the windowed conversation as a list of messages. Two things follow from that shape, and both hurt routing quality:

  • The payload is unbounded in characters. The window is counted in turns, and turn size varies by orders of magnitude — a window of 4 is a few hundred characters of chat or tens of thousands of characters of tool output. Turn count is a poor proxy for payload size, so the judge call's cost and latency swing wildly for the same configuration.
  • A long tool result can bury the task. The judge is being asked to assess the user's request; handing it 30 KB of pasted logs with the request somewhere above tends to produce a verdict about the logs.

Proposal

judge_char_budget on the followup table: render the judge's view as one condensed block capped at that many characters, instead of a raw message list.

followup = { refresh = "on_user_turn", recent_turn_window = 6, judge_char_budget = 6000 }

Condensation, when the budget binds:

  • the opening task anchor and the newest turn are protected — those are what the judge is actually assessing;
  • older window turns are shed first;
  • individual oversized messages are clipped rather than dropped, with an explicit truncation marker, so the judge can see that content was removed rather than silently inferring a shorter conversation;
  • tool results collapse to a placeholder, since their bulk is rarely what determines the tier.

Unset (the default) leaves the existing message-list behaviour untouched.

Why a character budget rather than a token budget

Tokens would be the more principled unit, but counting them requires a tokenizer for the judge model, which the route does not have and which would differ per provider. Characters are a stable, provider-independent proxy, and the failure mode of being slightly wrong is a marginally smaller or larger payload rather than a rejected request.

Note on shedding order

The implementation sheds oldest-first. An earlier draft shed by priority (newest turn > task anchor > framing > rest) and there is a reasonable argument for it; oldest-first was chosen because it keeps the judge's view of the conversation contiguous, which matters when the rubric reasons about how the conversation progressed. If benchmarks favour the other order it is a natural follow-up as a policy knob rather than a change of default.

Depends on the followup table in #347. Rebased onto current main and green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions