You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BUG SelectiveTextConverter token stages convert text that was never selected #2892
Chaining SelectiveTextConverter stages with TokenSelectionStrategy and preserve_tokens=True ends up converting text that was never selected.
convert_tokens_async always strips the ⟪⟫ markers, so the token branch in convert_async checks self._start_token not in result.output_text and wraps the whole output in one pair. The next stage then treats the entire prompt as selected:
stage 1 (base64, second half): tell me how ⟪dG8=⟫ ⟪ZG8=⟫ ⟪aXQ=⟫
stage 2 (rot13, token strategy): ⟪tell me how qT8= MT8= nKD=⟫
stage 3 (base64, token strategy): ⟪dGVsbCBtZSBob3cgcVQ4PSBNVDg9IG5LRD0=⟫
"tell me how" was never selected but gets base64-encoded in stage 3. This is the chaining pattern the TokenSelectionStrategy docstring describes, and the GUI stage editor from #2729 makes it easy to hit.
Fix: let convert_tokens_async keep the markers around each converted region, and have the selective converter use that when preserve_tokens is set.
I checked current main before making changes and found this is already covered: test_three_stage_selective_chain_preserves_original_regions_async now exercises the exact Base64 → ROT13 → Base64 chain and preserves only the selected regions. I am stepping back rather than submitting a duplicate fix.
Activity
nightcityblade commentedon Oct 7, 2026
Hi, I'd like to work on this. I'll submit a PR shortly.
nightcityblade commentedon Oct 7, 2026
I checked current main before making changes and found this is already covered: test_three_stage_selective_chain_preserves_original_regions_async now exercises the exact Base64 → ROT13 → Base64 chain and preserves only the selected regions. I am stepping back rather than submitting a duplicate fix.
romanlutz commentedon Oct 14, 2026
#3002 should cover this.