Skip to content

BUG SelectiveTextConverter token stages convert text that was never selected #2892

Closed

Description

Chaining SelectiveTextConverter stages with TokenSelectionStrategy and preserve_tokens=True ends up converting text that was never selected.

convert_tokens_async always strips the ⟪⟫ markers, so the token branch in convert_async checks self._start_token not in result.output_text and wraps the whole output in one pair. The next stage then treats the entire prompt as selected:

stage 1 (base64, second half): tell me how ⟪dG8=⟫ ⟪ZG8=⟫ ⟪aXQ=⟫
stage 2 (rot13, token strategy): ⟪tell me how qT8= MT8= nKD=⟫
stage 3 (base64, token strategy): ⟪dGVsbCBtZSBob3cgcVQ4PSBNVDg9IG5LRD0=⟫

"tell me how" was never selected but gets base64-encoded in stage 3. This is the chaining pattern the TokenSelectionStrategy docstring describes, and the GUI stage editor from #2729 makes it easy to hit.

Fix: let convert_tokens_async keep the markers around each converted region, and have the selective converter use that when preserve_tokens is set.

Activity

  1. nightcityblade commented on Oct 7, 2026

    @nightcityblade

    Hi, I'd like to work on this. I'll submit a PR shortly.

  2. nightcityblade commented on Oct 7, 2026

    @nightcityblade

    I checked current main before making changes and found this is already covered: test_three_stage_selective_chain_preserves_original_regions_async now exercises the exact Base64 → ROT13 → Base64 chain and preserves only the selected regions. I am stepping back rather than submitting a duplicate fix.

  3. romanlutz commented on Oct 14, 2026

    @romanlutz
    Contributor

    #3002 should cover this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions