FEAT: Support nested and multiple conversion markers

[Richard Lundeen (richlundeen)]

Description

Support nested and multiple marked text regions across direct converter calls, attack pipelines, API previews, and GUI editing. Each stage transforms all innermost regions and leaves surrounding text unchanged. Consuming stages remove the selected pairs; outer pairs remain for later stages. Once no markers remain, later converters retain the existing whole-value behavior.

For Translate to French -> Base64 -> ROT13, three layers keep the instruction prefix unchanged through all three stages:

Decode this recursively: ⟪⟪⟪Hello⟫⟫⟫ and ⟪⟪⟪Goodbye⟫⟫⟫

Add keep_tokens=False to convert_tokens_async. Direct callers can retain each selected pair with True; unmarked text results are wrapped, while non-text outputs remain unchanged. SelectiveTextConverter uses the same shared retention logic through preserve_tokens. Native token wrappers that select the same region retain one boundary pair rather than adding duplicate layers. Explicit input nesting and separate selections remain intact.

Preserve selective-subclass convert_async overrides, including subclasses that call super(). Resolve nested selections using the active per-call marker pair, not constructor defaults. A per-call text callback reuses the shared parser without storing mutable call state on converter instances. Seeded conversion and concurrent calls with different markers are covered.

Keep older custom convert_tokens_async overrides callable without new arguments when preservation is disabled. Unsupported preservation raises before invoking the override instead of bypassing it or wrapping unselected text. Request and response APIs accept custom markers, and the normalizer rejects empty markers at construction. Longer ASCII examples document the risk of marker collisions in target replies.

The GUI supports selections inside existing regions or around complete regions, rejects partial boundary crossings, and keeps all marker characters visible. Its selection button retains the Unicode defaults. Preconverted pieces are sent unchanged without rerunning request converters.

Tests and Documentation

Rewrite the paired selective-conversion guide as a credential-free, executable first-read tutorial. It covers marked text, direct keep_tokens calls, preserving chains, explicit nesting, programmatic selection, attack wiring, custom markers, wrapper composition, and compatibility limits. Include an executed mixed-marker example. All 26 paired cell sources match, and eleven cells retain their executed outputs.

Add regressions for top-level selective subclasses, super() delegation, inherited selection, custom token overrides, subclass random scopes, active-marker composition, and concurrent mixed-marker calls. Existing tests cover multiple regions, empty/multiline input, generated marker text, media outputs, API persistence, and single-/multi-turn attacks.

Validation from the repository root:

• $env:UV_NO_SYNC='1'; uv run --no-sync pre-commit run --all-files — passed all 18 hooks after notebook metadata and path normalization.
• uv run --no-sync pytest tests\unit\converter\test_converter.py tests\unit\converter\test_selective_text_converter.py tests\unit\common\test_random_context.py tests\unit\backend\test_converter_service.py::TestPreviewConversion tests\unit\backend\test_message_send_service.py::TestNormalizerPersistence tests\unit\executor\attack\single_turn\test_prompt_sending.py::test_attack_marker_pipelines_cover_history_request_response_async tests\unit\executor\attack\multi_turn\test_multi_prompt_sending.py::test_attack_marker_pipelines_cover_multiple_messages_async tests\unit\prompt_normalizer\test_prompt_normalizer.py -q — 402 passed.
• uv run --no-sync jupytext --to ipynb --execute --set-kernel python3 doc\code\converters\6_selectively_converting.py — passed; all code examples executed locally without LLM credentials and outputs are retained.
• git diff --check — passed.

The affected GUI suites passed earlier (69 tests), along with the frontend type check. Full unit and integration suites were not run locally.