Skip to content

Stabilize Playwright approval assertion #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
justsml merged 2 commits into from
Jul 13, 2026
Merged

Stabilize Playwright approval assertion #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
95a26ae
stabilize Playwright approval assertion
justsml Jul 13, 2026
5b4f317
serialize shared Playwright app state
justsml Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion playwright.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@ import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
testDir: "./tests/playwright",
fullyParallel: true,
// Specs share one app process plus mocked API and lab state, so concurrent
// workers can consume each other's routes and make otherwise valid tests
// nondeterministic in CI.
fullyParallel: false,
workers: 1,
retries: process.env.CI ? 2 : 0,
reporter: process.env.CI ? [["github"], ["html", { open: "never" }]] : "list",
use: {
Expand Down
2 changes: 1 addition & 1 deletion tests/playwright/docker-lab-idor-flow.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,13 +483,13 @@
await page.getByLabel("Initial security research prompt").fill(`Assess the IDOR Docker lab at ${labTarget}`);
await page.getByLabel("Create workspace and start thread").click();

await expect(page.getByRole("region", { name: "Investigation Plan" }).first()).toBeVisible();

Check failure on line 486 in tests/playwright/docker-lab-idor-flow.spec.ts

View workflow job for this annotation

GitHub Actions / Test

[chromium] › tests/playwright/docker-lab-idor-flow.spec.ts:478:3 › Docker lab IDOR browser eval flow › shows approval rejection and approval-gated tool evidence through the browser 

  1) [chromium] › tests/playwright/docker-lab-idor-flow.spec.ts:478:3 › Docker lab IDOR browser eval flow › shows approval rejection and approval-gated tool evidence through the browser 

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: expect(locator).toBeVisible() failed

    Locator: getByRole('region', { name: 'Investigation Plan' }).first()
    Expected: visible
    Timeout: 5000ms
    Error: element(s) not found

    Call log:
      - Expect "toBeVisible" with timeout 5000ms
      - waiting for getByRole('region', { name: 'Investigation Plan' }).first()


      484 |     await page.getByLabel("Create workspace and start thread").click();
      485 |
    > 486 |     await expect(page.getByRole("region", { name: "Investigation Plan" }).first()).toBeVisible();
          |                                                                                    ^
      487 |     await expect(page.getByText("Gate direct report-object checks").first()).toBeVisible();
      488 |     await expect(page.getByText("Approval-gated validation").first()).toBeVisible();
      489 |     await expect(page.getByRole("button", { name: "Reject" }).first()).toBeVisible();
        at /home/runner/work/ExploitHunter.app/ExploitHunter.app/tests/playwright/docker-lab-idor-flow.spec.ts:486:84
await expect(page.getByText("Gate direct report-object checks").first()).toBeVisible();
await expect(page.getByText("Approval-gated validation").first()).toBeVisible();
await expect(page.getByRole("button", { name: "Reject" }).first()).toBeVisible();

await page.getByRole("button", { name: "Reject" }).first().click();
await expect(page.locator("body")).toContainText(/request\s*rejected/i);
await expect(page.locator("body")).toContainText(/(?:request\s*rejected|user\s*rejected\s*request)/i);
await expect(page.getByText(/will not run the direct report-object check/i).first()).toBeVisible();
expect(state.phase).toBe("rejected");
expect(
Expand Down
Loading