Skip to content

FIX: route three stragglers through the converter random context - #2726

Open
Vishnu Rajeev (VishnuR23) wants to merge 1 commit into
microsoft:mainfrom
VishnuR23:fix/finish-converter-random-migration
Open

Vishnu Rajeev (VishnuR23) wants to merge 1 commit into
microsoft:mainfrom
VishnuR23:fix/finish-converter-random-migration

Conversation

@VishnuR23

Copy link
Copy Markdown
Contributor

Description

#2472 ("make converter randomness reproducible and composable", merged Aug 25) moved 20 converter modules onto pyrit.common.random_context, so that initialize_pyrit_async(seed=...) makes a run reproducible.

Three converters merged in the two days after it and were written against the pre-migration pattern, so the sweep missed them:

converter merged what it does instead
BijectionConverter (#2301) Aug 26 random.Random(seed) at construction
PuzzledConverter (#2256) Aug 26 random.Random(self._seed) per conversion
CharNoiseConverter (#2277) Aug 27 random.random() / random.choice() on the global module

They are the only three left in pyrit/converter/ that draw randomness outside the context — after this change, that grep is empty.

The effect is that a configured root seed silently does not reach them. Running each three times under configure_random_seed(seed=42) on main:

CharNoiseConverter   ugd quicj!cqnwo!gox jvnpr nvfr shf l`{y!coh
                     uhe!puick brnvo epx!jumqr!ovdr the!k`zx dpg
                     uhf!qvjbj asnxn!enx ktlqs!nufr!thd mazy enf     -> 3 different outputs

LetterBijection      {'a': 'n', 'b': 'p', 'c': 'g', 'd': 'v', ...}
                     {'a': 'v', 'b': 'a', 'c': 'i', 'd': 'r', ...}
                     {'a': 'e', 'b': 'i', 'c': 't', 'd': 'd', ...}   -> 3 different mappings

PuzzledConverter behaves the same way: its existing test_output_is_reproducible_with_same_seed only covers an explicit seed=42, which random.Random(42) satisfies by accident, so the root-seed path was never exercised.

CharNoiseConverter has a second problem: because it draws from the global random module, it also advances process-wide RNG state on every conversion. That is the isolation invariant #2397 established, and it regressed for this one converter.

Changes

Each converter now draws from its own stream:

  • CharNoiseConverter._noise -> self._get_random_generator(stream="character-noise") (the random import is now unused and removed).
  • PuzzledConverter -> self._get_random_generator(stream="puzzle"). The explicit seed argument keeps working: the base class already routes self._seed through _get_random_seed_override().
  • BijectionConverter.__init__ -> get_random_generator(namespace=..., stream="mapping", seed=seed). The mapping is drawn at construction time, outside any conversion execution, so namespace and seed are passed explicitly — the same shape ImagePromptStyleConverter already uses for its construction-time draw. It also now stores self._seed, per the contract documented on Converter.

No behavior change when no root seed is configured: all three still produce fresh randomness per call. Explicit per-converter seeds keep taking precedence over the root.

Tests and Documentation

  • CharNoiseConverter added to _stochastic_converter_cases() in tests/unit/converter/test_seeded_converter_determinism.py, which is the existing matrix from ENH: make converter randomness reproducible and composable #2472. It picks up both parametrized contracts for free — repeatability under a root seed, and non-disturbance of the global RNG.
  • test_output_is_reproducible_under_configured_root_seed added to test_puzzled_converter.py, covering the root-seed path the existing explicit-seed test misses. It lives in that file so it inherits the autouse fixture that avoids a spaCy dependency.
  • Four tests added to test_bijection_converter.py pinning the contract in both directions: reproducible under a root seed, varies across different root seeds, explicit seed overrides the root, and still unseeded when no root is configured.

Four of these fail on main and pass with this change (verified by reverting only the three source files and re-running: 4 failed, 119 passed). The one that may be non-obvious is test_initialized_seed_does_not_disturb_global_rng[character noise] — it fails on main precisely because CharNoiseConverter consumes the global RNG.

Verification:

  • pytest tests/unit/converter/test_seeded_converter_determinism.py test_puzzled_converter.py test_bijection_converter.py -> 123 passed
  • pytest -n 4 --dist=loadfile tests/unit -> 18518 passed, 11 skipped
  • pre-commit run --files <changed> -> all hooks pass, including ruff and ty

Docstrings updated where they described the old behavior (CharNoiseConverter's "each call draws fresh randomness" and PuzzledConverter's seed argument). No other documentation changes — this brings three stragglers in line with the contract #2472 already documents.

🤖 Generated with Claude Code

microsoft#2472 moved 20 converter modules onto pyrit.common.random_context so that
initialize_pyrit_async(seed=...) makes a run reproducible. Three converters
merged in the two days after it and were written against the pre-migration
pattern, so the sweep missed them:

  BijectionConverter (microsoft#2301, Aug 26) - random.Random(seed) at construction
  PuzzledConverter   (microsoft#2256, Aug 26) - random.Random(self._seed) per call
  CharNoiseConverter (microsoft#2277, Aug 27) - random.random()/choice() on the global module

A configured root seed silently did not reach any of them, so each produced
different output on every run under the same seed. CharNoiseConverter also
advanced process-wide RNG state, regressing the isolation invariant from microsoft#2397.

Each now draws from its own named stream. Explicit per-converter seeds keep
taking precedence, and behavior is unchanged when no root seed is configured.
After this change no converter draws randomness outside the context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
return self._create_identifier(params={"noise_probability": self.noise_probability})

def _noise(self, text: str) -> str:
rng = self._get_random_generator(stream="character-noise")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a configured root seed, this makes the built-in best_of_n technique send the same prompt on every attempt. PromptSendingAttack retries the same input, and the character-swap and capitalization steps are already deterministic. Making the final noise step deterministic removes the remaining variation.

I reproduced this with seed=42 through the actual retry loop and normalizer: the base commit sends 20 distinct prompts, while this branch sends one identical prompt 20 times.

Could we give retries a stable, attempt-specific random scope at the attack level? Different attempts should generate different variants, while rerunning with the same root seed should reproduce the complete sequence. This would preserve both behaviors without putting retry logic in the converter.

Comment on lines +60 to +64
rng = get_random_generator(
namespace=f"{type(self).__module__}.{type(self).__qualname__}",
stream="mapping",
seed=seed,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Construction can happen inside another converter's convert_async, where a random execution is already active. Without an owner, this call returns the same cached generator for instances with the same type and seed. The second instance continues the first instance's RNG sequence instead of independently honoring its seed.

For example, this assertion passes on the base commit but fails here, even without a configured root seed:

with random_execution(namespace="composite"):
    first = LetterBijectionConverter(seed=7)
    second = LetterBijectionConverter(seed=7)
    assert first.mapping == second.mapping

Could we give each construction a private generator derived from get_random_seed(...), rather than sharing the invocation's cached generator, and cover construction inside an active conversion?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants