FIX: route three stragglers through the converter random context - #2726
Vishnu Rajeev (VishnuR23) wants to merge 1 commit into
Conversation
microsoft#2472 moved 20 converter modules onto pyrit.common.random_context so that initialize_pyrit_async(seed=...) makes a run reproducible. Three converters merged in the two days after it and were written against the pre-migration pattern, so the sweep missed them: BijectionConverter (microsoft#2301, Aug 26) - random.Random(seed) at construction PuzzledConverter (microsoft#2256, Aug 26) - random.Random(self._seed) per call CharNoiseConverter (microsoft#2277, Aug 27) - random.random()/choice() on the global module A configured root seed silently did not reach any of them, so each produced different output on every run under the same seed. CharNoiseConverter also advanced process-wide RNG state, regressing the isolation invariant from microsoft#2397. Each now draws from its own named stream. Explicit per-converter seeds keep taking precedence, and behavior is unchanged when no root seed is configured. After this change no converter draws randomness outside the context. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| return self._create_identifier(params={"noise_probability": self.noise_probability}) | ||
|
|
||
| def _noise(self, text: str) -> str: | ||
| rng = self._get_random_generator(stream="character-noise") |
There was a problem hiding this comment.
With a configured root seed, this makes the built-in best_of_n technique send the same prompt on every attempt. PromptSendingAttack retries the same input, and the character-swap and capitalization steps are already deterministic. Making the final noise step deterministic removes the remaining variation.
I reproduced this with seed=42 through the actual retry loop and normalizer: the base commit sends 20 distinct prompts, while this branch sends one identical prompt 20 times.
Could we give retries a stable, attempt-specific random scope at the attack level? Different attempts should generate different variants, while rerunning with the same root seed should reproduce the complete sequence. This would preserve both behaviors without putting retry logic in the converter.
| rng = get_random_generator( | ||
| namespace=f"{type(self).__module__}.{type(self).__qualname__}", | ||
| stream="mapping", | ||
| seed=seed, | ||
| ) |
There was a problem hiding this comment.
Construction can happen inside another converter's convert_async, where a random execution is already active. Without an owner, this call returns the same cached generator for instances with the same type and seed. The second instance continues the first instance's RNG sequence instead of independently honoring its seed.
For example, this assertion passes on the base commit but fails here, even without a configured root seed:
with random_execution(namespace="composite"):
first = LetterBijectionConverter(seed=7)
second = LetterBijectionConverter(seed=7)
assert first.mapping == second.mappingCould we give each construction a private generator derived from get_random_seed(...), rather than sharing the invocation's cached generator, and cover construction inside an active conversion?
Description
#2472 ("make converter randomness reproducible and composable", merged Aug 25) moved 20 converter modules onto
pyrit.common.random_context, so thatinitialize_pyrit_async(seed=...)makes a run reproducible.Three converters merged in the two days after it and were written against the pre-migration pattern, so the sweep missed them:
BijectionConverter(#2301)random.Random(seed)at constructionPuzzledConverter(#2256)random.Random(self._seed)per conversionCharNoiseConverter(#2277)random.random()/random.choice()on the global moduleThey are the only three left in
pyrit/converter/that draw randomness outside the context — after this change, that grep is empty.The effect is that a configured root seed silently does not reach them. Running each three times under
configure_random_seed(seed=42)onmain:PuzzledConverterbehaves the same way: its existingtest_output_is_reproducible_with_same_seedonly covers an explicitseed=42, whichrandom.Random(42)satisfies by accident, so the root-seed path was never exercised.CharNoiseConverterhas a second problem: because it draws from the globalrandommodule, it also advances process-wide RNG state on every conversion. That is the isolation invariant #2397 established, and it regressed for this one converter.Changes
Each converter now draws from its own stream:
CharNoiseConverter._noise->self._get_random_generator(stream="character-noise")(therandomimport is now unused and removed).PuzzledConverter->self._get_random_generator(stream="puzzle"). The explicitseedargument keeps working: the base class already routesself._seedthrough_get_random_seed_override().BijectionConverter.__init__->get_random_generator(namespace=..., stream="mapping", seed=seed). The mapping is drawn at construction time, outside any conversion execution, so namespace and seed are passed explicitly — the same shapeImagePromptStyleConverteralready uses for its construction-time draw. It also now storesself._seed, per the contract documented onConverter.No behavior change when no root seed is configured: all three still produce fresh randomness per call. Explicit per-converter seeds keep taking precedence over the root.
Tests and Documentation
CharNoiseConverteradded to_stochastic_converter_cases()intests/unit/converter/test_seeded_converter_determinism.py, which is the existing matrix from ENH: make converter randomness reproducible and composable #2472. It picks up both parametrized contracts for free — repeatability under a root seed, and non-disturbance of the global RNG.test_output_is_reproducible_under_configured_root_seedadded totest_puzzled_converter.py, covering the root-seed path the existing explicit-seed test misses. It lives in that file so it inherits the autouse fixture that avoids a spaCy dependency.test_bijection_converter.pypinning the contract in both directions: reproducible under a root seed, varies across different root seeds, explicit seed overrides the root, and still unseeded when no root is configured.Four of these fail on
mainand pass with this change (verified by reverting only the three source files and re-running:4 failed, 119 passed). The one that may be non-obvious istest_initialized_seed_does_not_disturb_global_rng[character noise]— it fails onmainprecisely becauseCharNoiseConverterconsumes the global RNG.Verification:
pytest tests/unit/converter/test_seeded_converter_determinism.py test_puzzled_converter.py test_bijection_converter.py-> 123 passedpytest -n 4 --dist=loadfile tests/unit-> 18518 passed, 11 skippedpre-commit run --files <changed>-> all hooks pass, includingruffandtyDocstrings updated where they described the old behavior (
CharNoiseConverter's "each call draws fresh randomness" andPuzzledConverter'sseedargument). No other documentation changes — this brings three stragglers in line with the contract #2472 already documents.🤖 Generated with Claude Code