Skip to content

[Bug]: playwright-test-generator.agent.md contains invalid TypeScript in its example (sync { page } =>) #42743

Description

Version

1.63.0 (both playwright and @playwright/test)

Steps to reproduce

  1. npx playwright init-agents --loop=claude (any --loop value; the file is copied verbatim from the package)
  2. Open the generated playwright-test-generator.agent.md, or read it directly at node_modules/playwright/lib/agents/playwright-test-generator.agent.md
  3. Look at the <example-generation> block, under "Following file is generated:"

Actual

// spec: specs/plan.md
// seed: tests/seed.spec.ts

test.describe('Adding New Todos', () => {
  test('Add Valid Todo', async { page } => {
    // 1. Click in the "What needs to be done?" input field
    await page.click(...);

    ...
  });
});

async { page } => is a syntax error — the destructured parameter is missing its parentheses.

Expected

test('Add Valid Todo', async ({ page }) => {

Why it matters

The block is presented to the agent as the reference output shape under "Following file is generated". It is the one concrete example of a generated spec in the file, so it is exactly the thing a model pattern-matches when producing a new test. An agent reproducing the shape emits code that does not parse, and the failure surfaces later as a confusing TypeScript error in generated output rather than as a problem with the prompt.

It also appears to be the only TypeScript in these agent files that is not valid, so a lint or type-check pass over the shipped lib/agents/*.md fenced blocks would likely catch this one and nothing else.

Happy to send a PR if a one-line fix is welcome.


Secondary, lower confidence — playwright-test-healer.agent.md

Not a syntax bug, and arguably working as intended, but raising it in the same place since it is the same set of files:

  • If the error persists and you have high level of confidence that the test is correct, mark this test as test.fixme()
    so that it is skipped during the execution.

The precondition — the agent is confident the test is correct and the failure persists — is the case most likely to be a genuine application defect. test.fixme() is unconditional, so acting on this removes the journey from the suite and the pipeline goes green while the bug the test just caught is still there.

Escalating (leave it failing, report it) seems the safer default for that specific branch, with disabling reserved for a human decision. I appreciate this may be deliberate for an autonomous healing loop where a red suite blocks everything — if so, a sentence acknowledging the trade-off would help teams whose own standards forbid unconditional skips, which is how we hit it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions