Repository navigation
Add read-aloud work mode and hold-to-ask - #225
Merged
Merged
Conversation
With read-aloud on, Settings → Advanced → Experimental gains a "Work mode" switch. Turned on, reads play at 2× by default (1× to 3× in a picker) and leave out code, file paths, links, email addresses and long numbers. Off by default. - Speed is the playback synchronizer's rate, with the renderer's time-domain pitch algorithm holding the voice's pitch, so nothing changes on the wire. PCMSchedule scales its lead by the rate to keep the same real headroom, and drain() bounds its wait in wall-clock time. - Skipping goes through a new ReadAloudLLM: one chat-completions call to the LLM Gateway (qwen3.5-4b-32k-fast, temperature 0) that returns the selection word for word minus the noisy spans. It's best-effort: a failure, a reply cut off at max_tokens, or output too long to be a rewrite falls back to reading the selection verbatim. A stop during the rewrite plays nothing. The Settings footer says the selection goes to the gateway while skipping is on. - ReadAloudWorkModeStore owns the three settings and snaps a stored speed to the picker's choices. - check-invariants.sh: the client-side LLM ban exempts exactly one file, ReadAloudLLM.swift, and now also catches the gateway's hostname. AGENTS.md and the guardrails skill say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With read-aloud on, holding the trigger over selected text now records a
spoken request ("summarize this"), and the answer is read aloud. A tap
over a selection still reads it, and so does a hold with nothing said,
which is how a hold-only trigger still reads at all. A "Hold to ask about
the selection" switch under read-aloud turns it off. It defaults to on.
- DictationKeyGate reports .latch when a release latches a recording on,
the one moment a tap is told apart from a hold.
- DictationSession gains press(handingBack:) (Command.pressHandingBack):
the same dictation, whose transcript ends the run as the new terminal
phase .handedBack(text) instead of being pasted or added to the context
ring. The overlay and menu bar treat it as idle.
- SelectionSpeechRouting gains three states (selected, asking,
transcribing) and a 350 ms hold timer. The mic opens only once the key
has been held that long, so a tap never pays for a recording. The router
also resets the gate when a later press is buffered behind the selection
read, so the press after it isn't swallowed.
- ReadAloudLLM.answer sends the selection and the request in separate
tags, and SelectionSpeaker reads the reply at work mode's rate. An answer
has no fallback.
- A submitted cancel that lands while a dictation is transcribing is now
spent there. Left set, it cancelled the next press during its mic
bring-up, which a press during an ask's transcription would hit.
- Settings: the switch, how-to text that follows it, and a footer note
that asking sends the selection and the request to the LLM Gateway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings Work mode was one switch gating two settings. Drop the switch: with read-aloud on, Settings shows the Speed picker and the "Skip code, links, and long numbers" toggle directly, each independent. - Speed defaults to 1x (was 2x under work mode) and stays adjustable 1x-3x; the rate plumbing in StreamingPCMPlayer/PCMSchedule is unchanged. - Skipping defaults off: on its own it's an opt-in, since it sends the selection to the LLM Gateway before reading it. - ReadAloudWorkModeStore becomes ReadAloudStyleStore; the keys become ReadAloudSpeed and ReadAloudSkipsJargon, and ReadAloudWorkMode goes (unreleased, so no migration). - Docs and the check-invariants comment say "skip-jargon" instead of "work mode". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from a code review of the ⌘C fallback for apps whose selection Accessibility can't see: - The press still waits for the copy (starting the recording first and cancelling on a hit could cancel the previous dictation's pipeline and opens the mic on every read), but the no-selection wait drops from 250 ms to 100 ms. - Pasteboard snapshot, polling, and restore run on contextQueue via a shared DictationSession.offPool, never the cooperative pool: a snapshot can make another app produce promised data synchronously. - The copy waits for a string, not just a change count, so an app that clears the clipboard before writing isn't read as "nothing selected". - Copied text gets the AX path's cleanup (visibleTextOrNil, clip), so a zero-width-only copy isn't spoken. Cleanups: the change-count-gated restore lives once on SystemClipboard (restore(_:ifChangeCountIs:)), shared by the paste path and the copy; the trust check moves out of focusedElementForReading so each capture checks once; finishHint and stopHint derive from one gesture phrase; the fixed TTS Flush/Terminate commands are encoded once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mode # Conflicts: # AGENTS.md # Sources/BlurtEngine/FocusCapture/FocusCapture.swift # Sources/BlurtEngine/Injection/SystemClipboard.swift # Sources/BlurtEngine/Pipeline/DictationSession.swift # Sources/BlurtEngine/TTS/SelectionCopy.swift # Sources/BlurtEngine/TTS/SelectionSpeaker.swift
alexkroman
enabled auto-merge (squash)
October 2, 2026 22:21
- KeyboardModel.perform switched over DictationKeyGate.Action without the .latch case hold-to-ask added, which broke the iOS build once main's iPhone app arrived. A latch keeps the recording going, so the keyboard ignores it, as it ignored the .none the gate returned before. - SelectionCopy read the change count and the string in two steps, so a write between them paired the string with a stale count and the restore skipped, leaving the copy on the clipboard (CI caught it in "a clear before the write isn't mistaken for an empty copy"). It now re-reads the count and polls again if it moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Two additions to the experimental read-aloud feature, one commit each. Please merge with Rebase and merge rather than squash, so the two commits stay separate on
main: either can then be reverted or cherry-picked on its own, and each gets its own line in the release notes.Work mode (
e43754a). A switch under read-aloud that plays reads faster (2× by default, 1× to 3× in a picker) and can leave out code, file paths, links, email addresses and long numbers. Speed is the playback rate with pitch held, so nothing changes on the wire. Skipping is one LLM Gateway call that rewrites the selection for listening, and any failure falls back to reading the selection verbatim.Hold to ask (
1897b5c). Holding the trigger over a selection records a spoken request ("summarize this"), and the answer is read aloud. A tap still reads the selection. This one touches the dictation path: the key gate reports a new.latchaction,press(handingBack:)ends a dictation in a new.handedBack(text)phase instead of a paste, and the read-aloud router gains three states and a 350 ms hold timer. The commit also fixes a submitted cancel that outlived a transcribing dictation and cancelled the next press.Both features can send the selection to the LLM Gateway, and the Settings footer says so whenever a setting that does is on.
ReadAloudLLM.swiftis now the one file exempt from the client-side LLM ban incheck-invariants.sh, andAGENTS.mdand the guardrails skill record the exception.SelectionAskStoredefaults to on, and that needs a decision before merge. Anyone who already turned on read-aloud would find, after updating, that holding the key over a selection asks a question instead of reading it, and that their selection and request go to the gateway, without having opted in to either. Flipping the default to off is a one-line change.How it was tested
scripts/check.shpasses on each commit separately, run locally on macOS:1897b5c(the PR head): every check, with 766 tests in 117 suites passing plain and under the thread and address sanitizers.e43754a(work mode alone): every check, with 731 tests in 114 suites passing the same three ways. Periphery ran against an isolated index for this one, because its shared build cache in~/Library/Cachesstill held records from a second checkout and flagged a file this commit doesn't have.The UI suite and leak scan only run on CI.
scripts/check.shpasses (or CI will, if I'm not on a Mac)deliberately removed
🤖 Generated with Claude Code