Skip to content

FIX Handle attack parameter construction failures - #2344

Open
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-attack-parameter-failures
Open

FIX Handle attack parameter construction failures#2344
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-attack-parameter-failures

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Parameter construction for multi-objective attacks previously used fail-fast gathering, even when return_partial_on_failure=True. A simulated-conversation build could already perform model calls and persist memory state before a sibling failed, after which no successfully built objective executed and Scenario retry regenerated that work.

This change makes parameter construction follow the executor's existing failure contract:

  • Partial mode executes successfully built objectives and reports build and execution failures in original input order.
  • Strict mode settles parameter builds, suppresses outer attack execution when any build fails, and raises the first input-ordered build error.
  • Cancellation and other fatal BaseException values continue to propagate.
  • Completed result indices retain their original seed-group positions so Scenario retry skips persisted successes and avoids duplicate materialization.

Tests and Documentation

Added deterministic event-controlled tests for out-of-order build failures, mixed build/execution failures, strict suppression, cancellation propagation, original input-index alignment, and real Strategy-to-AtomicAttack-to-Scenario-to-CentralMemory retry linkage.

Validation:

  • Focused contract tests: 14 passed.
  • Executor, parameter, simulated-conversation, Scenario retry/partial-result, and attack-result memory tests: 214 passed.
  • Ruff check and format check passed for all changed files.
  • ty passed for all changed files.

Documentation was not changed because this fixes the existing return_partial_on_failure contract without adding a new public API. JupyText was not applicable.

Settle parameter construction failures deterministically, preserve original input alignment, and execute successful builds in partial mode so scenario resume can avoid duplicate materialization.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2b3468bc-0737-461b-bb82-18dad97d400e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants