AI-Generated Playwright Tests: How to Make Them Reliable

To make an AI-generated Playwright script trustworthy for QA, define the expected outcome before generation, scope every action to the right record and prove the assertions reject a controlled defect. Run the reviewed code with isolated data and retain the first failure in CI. A passing script earns its place in the suite by detecting the behavior that matters.

An AI-generated E2E test is browser-test code or a test plan produced with model assistance. Its reliability comes from the test contract and execution environment, not the fact that a model authored it. A repeatable green result is still weak evidence if the test never checks whether the right order was saved.

Why can generated E2E tests pass while the feature is broken?

A generated test can encode the wrong expectation, observe a superficial signal or accidentally depend on state left by another run. Those problems have different remedies. Calling all of them “flaky” can hide a test that consistently passes for the wrong reason.

The official Playwright Assertions guide explains how web assertions wait for a specified condition. Choose that condition from the requirement: a visible success message alone does not establish that a record survives a fresh read.

Use this table to classify the problem before editing the test. The examples are possible failure modes, not measured failure frequencies.

A timing failure needs synchronization. A weak assertion needs a stronger oracle: the evidence that decides correctness. A missing requirement needs a human decision. Increasing retries cannot resolve those last two problems.

What should you give the generator before it writes code?

Give the assistant a small acceptance contract with a controlled start and an observable finish. Specify the record, account role, input data, permitted actions and cleanup. Include at least one behavior that must remain unchanged.

For example, an owned order fixture could require: create an order for one seeded product at the fixture's known price; after submission, its ID, quantity and total match the input; a fresh application read returns that same order; no second order is created. These are example requirements, not observations of a live store.

A focused generation request can look like this:

Draft a Playwright Test for our seeded order fixture.
Start with a new cart and a permitted test account.
Scope the product by its unique test identifier.
Verify quantity and total against the seeded values.
Submit once, then verify the order through a fresh UI read
and the authorized read API.
Do not change expected values to match the application.
Stop and ask for the missing contract if a field or API is unclear.

Playwright's test-agent documentation separates planning, test generation and healing roles. That division is useful when organizing the work, but the application owner still needs to approve what constitutes a correct result.

Do not give a generator production credentials just to make a test convenient. Use an authorized test environment and define what a failed setup should report. A test that cannot establish its starting state should fail setup rather than inventing another journey.

How do you review the first generated test?

Review the generated diff as application code. Inspect every assumption the script adds: guessed selectors, expected prices, hidden setup, permissive fallbacks and assertions that merely check an element exists. Ask whether each assertion could pass if the feature were broken.

The following changes are review priorities. They improve the test contract, but they still need execution against your own fixture.

Playwright's best practices emphasize user-facing behavior, resilient locators and isolated tests. Apply those ideas to the actual page, rather than adding a role locator that does not match the application's accessible markup.

A weak illustrative test might end here:

await page.getByRole("button", { name: "Place order" }).click();
await expect(page.getByText("Order placed")).toBeVisible();

That checks a message. To check the order, use a stable identity, known expected inputs and a fresh read. Keep display assertions too when they are part of the requirement, but do not let them replace persistence checks.

What did the generated TodoMVC draft actually do?

The October 12, 2026 recording uses the public Playwright TodoMVC demo. Its task is explicit: create one uniquely named todo, mark that exact todo completed, reload, then check that both the exact text and completed state remain. This exercises browser-local persistence rather than an order API or backend database. Select any image to inspect it at full resolution.

The controller reports that the unchanged first draft ran once with zero retries and verified exact text, the checked checkbox and completed styling after reload. The retained browser shows a completed todo. The Playwright report below provides native test-run evidence: one test passed. The test source and trace are referenced in the recording but have not been supplied for independent assertion review.

Which locators survive a generated test review?

A stable locator identifies the intended control within the intended business record. Start with the page's actual accessible markup: a role and accessible name for a button, an associated label for an input, or an explicit test ID agreed with developers. A role locator can still select the wrong record if several rows have a Save button.

The following example assumes an owned fixture exposes order QA-104 throughdata-testid="order-QA-104". Adapt that contract to your application. The count assertion rejects a missing or duplicated target before the action.

const order = page.getByTestId('order-QA-104');
await expect(order).toHaveCount(1);
await expect(order.getByTestId('order-id')).toHaveText('QA-104');
await order.getByRole('button', { name: 'Save', exact: true }).click();
await expect(order.getByRole('status')).toHaveText('Saved');

Avoid generated CSS chains and unexplained .first() or .nth()selection. An indexed match needs an explicit ordering requirement. For an iframe, enter the expected frame with frameLocator() before locating its control. Follow the official locator guide for frame and shadow-DOM boundaries. Playwright locators resolve the current DOM for each action, which helps with re-renders; it cannot infer a business identity your selector omitted.

Put repeated page interactions in small page objects or fixtures when that removes real duplication. Keep expected amounts and business assertions visible to reviewers rather than burying them in a helper that reports success. ThePlaywright locator reference covers the wider selector catalogue; this review is about which target the generated code is allowed to act on.

What does a reviewed persistence check look like?

A stronger check compares the final record with values known before the action. The example below assumes an owned fixture with test-only seed and cleanup endpoints, a cart URL tied to the seed, and an order ID exposed through a test identifier. Those endpoints and UI contracts must be implemented or adapted before this test can run.

The seed creates one product, a private cart and an authorized fixture session. baseURL and authentication must be configured so browser and request fixtures have access to the same environment. The snippet is illustrative; it is not a report of an executed test.

import { randomUUID } from "node:crypto";
import { test, expect } from "@playwright/test";

test("the submitted order is stored with the expected amount", async ({
  page, request,
}) => {
  const runId = randomUUID();
  let failed = false;
  try {
    const seed = await request.post("/api/test/seed-order", {
      data: { runId, quantity: 1, totalCents: 7900 },
    });
    expect(seed.ok()).toBeTruthy();
    await page.goto(`/test/carts/${runId}`);
    await expect(page.getByTestId("cart-total")).toHaveText("$79.00");
    await page.getByRole("button", {
      name: "Place order", exact: true,
    }).click();

    const idField = page.getByTestId("order-id");
    await expect(idField).toHaveText(/\S+/);
    const orderId = (await idField.innerText()).trim();
    await page.reload();
    await expect(page.getByTestId("order-id")).toHaveText(orderId);

    await expect.poll(async () => {
      const response = await request.get(
        `/api/orders/${encodeURIComponent(orderId)}`,
      );
      if (!response.ok()) return null;
      const order = await response.json();
      return {
        id: order.id, runId: order.runId,
        quantity: order.quantity, totalCents: order.totalCents,
      };
    }, { timeout: 10000 }).toEqual({
      id: orderId, runId, quantity: 1, totalCents: 7900,
    });
  } catch (error) {
    failed = true;
    throw error;
  } finally {
    try {
      const cleanup = await request.delete(
        `/api/test/seed-order/${encodeURIComponent(runId)}`,
      );
      expect(cleanup.ok()).toBeTruthy();
    } catch (cleanupError) {
      if (!failed) throw cleanupError;
      console.error("Cleanup also failed", cleanupError);
    }
  }
});

The Playwright assertions reference documents retrying assertions and expect.poll. Bounded polling lets a known asynchronous write settle without silently accepting the wrong record. Adapt the timeout to your system's contract and retain the final failed observation.

This test checks identity and amount; it does not yet establish the absence of a duplicate order. If that is part of your contract, add a read scoped to runId and assert that exactly one order exists. Keep fixture cleanup failures visible without losing the original failure evidence.

How do you prove the test catches a regression?

Run a passing control and a deliberately broken case against a controlled environment. A test that fails during login in both cases has not demonstrated that its order assertion catches the injected defect.

The table below is an acceptance protocol, not test results. For each mutation, record the exact failure and show that restoring the behavior restores the expected outcome.

Use only failures you can inject safely in your own test environment. Name the exact requirement each mutation exercises. Count an inconclusive run separately from a meaningful pass or detected regression.

Keep the generator prompt, reviewed code and mutation evidence together. A before/after comparison should say what changed and what was actually observed. Do not claim improved reliability from a code diff alone.

When should a generated test retry?

Assertion waiting and test retries solve different problems. An awaitedexpect(locator).toHaveText() rechecks one condition within its timeout. A configured test retry reruns the test and can repeat its external actions. Prefer a specific readiness assertion over waitForTimeout(); actionability alone does not prove that an application finished saving.

// A one-time observation can race an asynchronous update.
expect(await page.getByRole('status').isVisible()).toBe(true);

// This waits for the required text within a bounded assertion timeout.
await expect(page.getByRole('status')).toHaveText('Saved');

Begin qualification with retries: 0. Once the test is understood, choose any suite-level retry policy deliberately and keep the original failed attempt. Playwright'sretry documentation distinguishes tests that pass initially, pass after retry and keep failing. A recovered retry should remain visible as flaky in the report. It is not evidence that the underlying cause was fixed.

Inspect the trace before increasing a timeout. Classify the failure as application behavior, test code, setup or infrastructure, and assign an owner. If a broken test is quarantined, record the issue and expiry date and keep a separate run exposing it. Review every new skip, caught exception and weakened assertion. Re-running an uncertain purchase or submission requires outcome reconciliation and an approved test environment.

What must stay stable when the test moves into CI?

Pin dependencies, install the matching browser binaries and use the same configuration you qualified locally. Define how the environment becomes ready, how test accounts are provisioned and who owns its seeded data. A visible homepage is not proof that all backend dependencies are ready.

Playwright browser contexts isolate browser state between tests. They do not reset a shared backend database. Use unique run identities or isolated environments for server-side data, and make cleanup part of the fixture contract.

The official CI guide covers running Playwright in CI. Start qualification with retries disabled so failures remain visible. If your production suite later uses retries, report first-attempt failures and recovered attempts separately; a final green build should not erase the instability you are investigating.

Retain traces and failure artifacts with access appropriate to their contents. The Trace Viewer guide explains how to inspect actions, DOM snapshots and network activity. Compare those observations with the intended contract before blaming the model, locator or application.

Give each worker its own mutable data

A new browser context does not isolate a shared backend account or database. Create a unique run namespace, seed known records and clean up only those records. Parallel workers that mutate account state should use separate accounts or tenants. Freeze locale, timezone and other inputs that affect expected output. Mock third-party services in a focused test, and retain separate integration coverage for the real dependency.

Use a dedicated test identity for authenticated CI. Theauthentication guide warns that saved authentication state can contain sensitive credentials. Keep it out of Git, restrict report access and refresh it through the approved setup. A permission failure should not silently fall back to another account.

Make the CI command and failure artifacts explicit

This configuration is a starting policy for a Node Playwright Test project, not a measured optimum. Adjust the origin and deadlines to your environment. It uses one CI worker and no whole-test retries so qualification failures stay visible.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  forbidOnly: !!process.env.CI,
  retries: 0,
  workers: process.env.CI ? 1 : undefined,
  timeout: 30_000,
  globalTimeout: 10 * 60_000,
  expect: { timeout: 5_000 },
  reporter: [
    ['list'],
    ['html', { open: 'never' }],
    ['junit', { outputFile: 'test-results/junit.xml' }],
  ],
  use: {
    baseURL: process.env.E2E_BASE_URL ?? 'http://127.0.0.1:4173',
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
});

npm ci
npx tsc --noEmit
npx playwright install --with-deps chromium
npx playwright test --project=chromium

Commit the dependency lockfile and use the project's declared Node version. Provision the application, readiness check and seed before the last command; this configuration does not start a server. Playwright transforms TypeScript without a full type check, so runTypeScript checking separately. Keep the CI job deadline above Playwright's global timeout so the runner can finish its report.

Upload playwright-report/ and test-results/ from an always-run artifact step, including when the test command fails. Preserve the failing exit status. Connect JUnit to your CI test-results view and retain the trace for diagnosis. Gate the merge on the reviewed test job, not whether an HTML report was produced. Set artifact retention and access to account for screenshots, DOM content and network data.

How should an agent help without weakening the release gate?

Let the agent investigate failures and propose edits. Require review when an edit changes expected values, drops an assertion, expands permitted actions or substitutes another business record. A locator repair and a changed acceptance criterion are different decisions.

ego(lite)'s visible Space workflow can help a coding agent reproduce the relevant flow with a person checking the target and outcome. Use that inspection to clarify the test plan and repair the code. Run the reviewed test again through the controlled cases before promoting it into CI.

This article covers the initial path from generation to qualification. Long-term suite maintenance, changing requirements and locator conventions deserve their own processes. Start with one important flow whose initial state and final outcome you can independently inspect.

Use visible inspection when a real session needs review

ego(lite) can help investigate an authorized signed-in flow while a person watches or takes control. Its quick startand Space documentation describe reusing the browser login environment and inspecting the task in its own workspace. Use that path to resolve a target or session question, then rerun the reviewed Playwright code in its isolated QA environment.

With ego(lite) installed and connected, run ego-browser skill to read the API matching the installed runtime. Give the connected coding agent a bounded task such as the prompt below, replacing the origin and record with your approved staging fixture. The ego-browser custom API is separate from Playwright's Page; pasting Playwright methods into that runtime is not an integration step.

Use the installed ego-browser skill and one named Space: QA order review.
Open our approved staging origin and inspect seeded order QA-104.
Confirm the intended account and record before any action.
Read only: do not submit, delete, purchase or alter unrelated records.
Take a fresh Snapshot after navigation or a re-render.
If login, MFA or a security challenge needs me, hand control back and pause.
Resume only after my explicit handback and recheck the current page.
Report observed fields and the evidence that supports them.
Propose the smallest Playwright diff; do not weaken assertions or auto-commit it.

A reviewer should approve the business expectation, data setup and changed test diff. Require fresh evidence for a changed expected value, removed assertion, new skip or broader permission. Page content and operation results may reach the connected AI provider; follow the privacy policy and redact sensitive artifacts. This inspection path supplies observations, while the reviewed assertions decide whether the build meets the requirement.

How did the agent check the same task in ego(lite)?

The coding agent opened TodoMVC in a named ego(lite) Space and was asked to create and complete one unique todo without modifying unrelated existing items. The overview shows the named task running alongside another Space. It also records setup discovery for an older installed command; that issue is not hidden by the final result.

The final capture shows the result table beside the live task, including Agent is in control, Take over and Stop. It reports that the same unique row was created, completed and still matched after reload. Other existing todos remain visible. This is a separate browser inspection of the business requirement, not execution of the generated Playwright test.

Which checks are reported in the result?

The ego(lite) result identifies the todo as Ego persistence check 2026-10-12T09:18:45.895Z. Its four checks are transcribed below so the essential evidence does not depend on reading small screenshot text.

These normal-run captures do not show a reviewed test diff or a suppressed-storage-write experiment. They therefore do not establish that the generated draft misses a fault, that a revision detects one, or that ego(lite) improved its reliability. The next qualification step is to challenge the preserved draft and reviewed version under the same deliberate fault, retain actual failures, and accept the result if both already detect it.

What does qualification leave untested?

Qualification establishes only what the chosen scenarios exercise. The example order contract does not cover performance, accessibility, authorization boundaries or every supported browser. Select those checks from the application's requirements and risk, rather than treating one qualified flow as complete coverage.

The code fragments require an application fixture and have not been used here to measure a generator's reliability. Keep your executed controls, mutations and environment details with the reviewed test. Only those observations can support claims about the behavior of your suite.

FAQ

Are AI-generated E2E tests inherently flaky?

The authoring method alone does not determine flakiness. Examine synchronization, isolation, environment assumptions and assertions before assigning a cause.

Is a test that passes ten times reliable?

Repeated passes show behavior under those runs' conditions. They do not demonstrate that the test can detect a regression it never encounters.

Can AI generate the expected values too?

Use values from the approved requirement or controlled seed. An expectation inferred from the same application output can make an incorrect result pass.

Does a fresh browser context reset test data?

A browser context isolates browser state. Shared server-side records need their own seed, namespace or reset mechanism.

Should a healed test commit itself?

Review the proposed diff and rerun its qualification cases. Give special scrutiny to changes in assertions and expected outcomes.

What if there is no read API for verification?

Check the exact record through a fresh UI navigation or reload and document what that evidence can establish. Do not claim a backend check you cannot perform.

Related reading

Read Playwright locators for target selection and Claude Code browser testing for an interactive testing workflow. For browser execution and review, see computer use agents and their controls. Pick one generated test and write the defect it must catch before adding more tests.