Post-Mortem: Surviving AI-Generated Playwright Tests in Production

The promise of AI-assisted testing was speed. The reality, after six months of deploying AI-generated Playwright tests into a high-traffic production environment, is a nuanced landscape of high velocity and significant maintenance friction. While Large Language Models (LLMs) can generate a Playwright test script in seconds, the resulting code often lacks the defensive engineering, semantic stability, and architectural understanding required for reliable Continuous Integration (CI) and continuous deployment pipelines.

This analysis examines the outcomes of a longitudinal study where a team replaced 40% of their end-to-end (E2E) regression suite with AI-generated scripts. The findings reveal that while AI excels at structural scaffolding, it consistently fails at handling state, flakiness mitigation, and complex user flows without human intervention. The "survival rate" of these tests—defined as the percentage of tests that remained stable and green without manual refactoring over six months—was initially low, but improved dramatically after implementing strict prompt engineering guardrails and post-generation linting.

The Experiment: Setting the Baseline

The objective was to accelerate the creation of regression tests for a complex SaaS dashboard. The baseline was a legacy Cypress suite that was slow, flaky, and difficult to read. The team integrated a workflow where product managers and developers would describe test scenarios in natural language, and an AI agent would generate the corresponding Playwright code in TypeScript.

The generated code was pushed to a branch, run through a local pre-commit hook for basic syntax checking, and then merged into the main CI pipeline. No manual code review of the test logic was performed for the first two months, simulating a "maximum automation" scenario.

What Survived: The High-Velocity Patterns

Certain test patterns survived the transition with minimal maintenance. The most successful AI-generated tests were "Happy Path" scenarios involving linear interactions with stable UI elements.

Linear, Stateless Interactions

AI models are highly probabilistic engines trained on vast datasets of code. When the task is "fill form, click submit, verify success message," the output is consistent. In our analysis, 85% of these simple tests passed their first execution in CI. The generated code typically utilized Playwright's auto-waiting mechanisms correctly, avoiding the common "race condition" errors that plague manual test writing.

// Typical AI-Generated Success Pattern
import { test, expect } from '@playwright/test';

test('user can create a new project', async ({ page }) => {
  await page.goto('/dashboard');
  
  // AI correctly identified the stable button testid
  await page.getByTestId('new-project-btn').click();
  
  await page.getByLabel('Project Name').fill('Test Project');
  await page.getByRole('button', { name: 'Create' }).click();
  
  // Verification is usually accurate for simple text
  await expect(page.getByText('Project created successfully')).toBeVisible();
});

In this example, the AI correctly prioritized getByTestId and getByRole over fragile CSS selectors. This semantic locatability is a key reason these tests survived; the locators were resilient to minor UI styling changes.

Code Quality and Structure

Surprisingly, the code generated by modern LLMs adhered to Playwright's best practices better than some human-written tests from years prior. The AI consistently imported the correct modules, used async/await properly, and utilized the expect library for assertions. It did not hallucinate non-existent Playwright methods. The syntax was clean, readable, and strictly typed (TypeScript). For developers, this meant that the time spent writing boilerplate was reduced to zero, allowing them to focus on the test logic and assertions.

What Broke: The Failure Modes

Where the AI-based workflow fractured was in complexity, state management, and flakiness mitigation. The failure modes were not random; they followed predictable patterns related to the limitations of LLM context windows and their lack of understanding of application architecture.

Flakiness and Race Conditions

In Month 2, 30% of the generated tests exhibited flakiness. The AI lacks awareness of network latency, server-side rendering delays, or database transaction locks.

A common failure involved waiting for dynamic content. The AI would generate:

await page.click('#save-button');
expect(page.locator('.success-message')).toBeVisible(); // Flaky

The AI failed to include a specific web-first assertion that polls for visibility, or a manual wait for a network response. Instead of a hard wait (page.waitForTimeout), which it could generate but rarely did effectively for complex async flows, it assumed the DOM update was synchronous. This caused CI pipelines to run, fail, and pass on re-run, eroding team confidence in the test suite.

Handling Authentication and State

E2E tests require a logged-in user context. While Playwright supports storageState and test.beforeAll, the AI frequently got this wrong. It often tried to automate the login flow inside every test, which is slow and prone to MFA (Multi-Factor Authentication) lockouts. Alternatively, it would reference environment variables for session tokens that did not exist in the CI environment.

The AI lacked the "global context" of the application. It did not know that the API returns a 401 on stale tokens or that the backend invalidates sessions after 24 hours. These stateful interactions required human intervention to harden the setup and teardown logic.

Fragmented Complex Flows

When asked to test multi-step workflows (e.g., "Onboard a user, assign them a role, and verify they can access the restricted dashboard"), the AI often fragmented the logic. It would write the code correctly for step one, but for step two, it would hallucinate selectors for UI elements that only appear after a specific server-side validation. The result was code that looked logical but failed at runtime because the preconditions for the next step were not met or verified.

The Engineering Pivot: Hardening the Workflow

Recognizing these failure modes, the team pivoted from "blind generation" to "assisted hardening." This change in strategy is the most critical takeaway for organizations adopting AI for testing.

1. Post-Generation Linting

We implemented a CI step that runs eslint and tsc on the generated files. While the AI rarely made syntax errors, it occasionally produced invalid TypeScript. More importantly, we added a custom ESLint rule to ban page.waitForTimeout. If the AI generated a hard wait, the pipeline would fail, forcing the developer to review and replace it with a web-first assertion. This single rule reduced flakiness by 40% in the subsequent month.

2. Prompt Engineering for Defensive Coding

We stopped asking the AI to "write a test." Instead, we used a system prompt that enforced specific constraints:

"You are a senior test engineer. Write a Playwright test in TypeScript. Do NOT use hard waits. Use web-first assertions. Assume the app has network latency. If the user must be logged in, assume context.storageState is already set; do not automate the login form unless explicitly asked. Use getByRole and getByLabel exclusively."

This context shift dramatically improved the survival rate of stateful tests. By removing the need for the AI to solve the authentication problem, it could focus on the interaction logic.

3. The "Human-in-the-Loop" Review

The most significant drop in breakage occurred when we mandated a human review for any test involving more than three distinct user actions. The AI is a code generator, not a test architect. It does not understand the business risk. A test that verifies a button turns green is easy to generate; a test that verifies a button turns green only if the database transaction succeeded is hard to generate correctly. Humans were needed to add the negative test cases and the edge-case assertions that the AI consistently omitted.

Quantitative Results: The 6-Month Data

The data from the six-month period provides a clear picture of the trade-offs.

The AI-first approach (Months 1-2) actually increased maintenance overhead. Developers were spending more time debugging AI-generated tests than writing them. However, the

AI-assisted approach (Months 3-6) reversed the trend. Once we established strict guardrails and human-in-the-loop review processes, the maintenance burden dropped significantly. The key wasn't replacing developers with AI, but rather shifting developers from writing boilerplate assertions to curating test logic and verifying business intent.

Phase 2: Establishing Guardrails and "AI Hygiene"

To combat the fragility of LLM-generated tests, we implemented a three-layer defense system.

1. The Prompt Engineering Protocol

We stopped allowing developers to prompt directly against the raw application state. Instead, we created a structured context window that included:

• Recent DOM Snapshots: The last 5 successful test runs for the target page.
• Selector Registry: A centralized JSON file mapping semantic UI elements (e.g., #login-button) to their stable Playwright locators.
• Failure History: A log of the last 10 failures for that specific test file, allowing the AI to understand why previous tests broke.

Example prompt template stored in our internal prompts/test-generation.md:

# Role
You are a senior QA engineer specializing in Playwright.

# Context
- Target Page: {{page_url}}
- Current Test Goal: {{user_story}}
- Stable Selectors: {{selector_registry}}
- Recent Failures: {{failure_logs}}

# Constraints
1. Never use CSS class selectors unless they are data-testid driven.
2. Always wrap flaky network assertions in `expect.poll`.
3. Use `page.getByRole` over `page.locator` where possible.
4. Output only valid TypeScript. Do not include explanations.

# Task
Generate a Playwright test for: {{user_story}}

2. The "Human Review" Gate

No AI-generated test could merge into the main branch without a human sign-off. This wasn't about code style; it was about semantic validation. The AI often generated tests that passed technically but failed logically (e.g., verifying a button is visible when it's actually hidden behind a modal).

We created a lightweight CLI tool, ai-test-audit, that parsed generated tests and flagged high-risk patterns:

// ai-test-audit/analyzer.js
import { glob } from 'glob';

const bannedPatterns = [
  /locator\('class=/g,      // Fragile CSS classes
  /waitForTimeout\(/g,      // Arbitrary waiting
  /input\('[a-z-]+/'g       // Raw input by name without context
];

export function auditTestFile(filePath) {
  const content = require('fs').readFileSync(filePath, 'utf8');
  let riskScore = 0;

  bannedPatterns.forEach(pattern => {
    if (pattern.test(content)) {
      riskScore += 10;
      console.warn(`[WARNING] Found fragile pattern in ${filePath}: ${pattern}`);
    }
  });

  // Check for missing auto-waiting
  if (!/page\.goto|page\.click/.test(content)) {
    riskScore += 5;
  }

  return riskScore;
}

Any test scoring above 15 was rejected by our CI pipeline, forcing developers to refine the prompt or manually fix the output.

3. Self-Healing Selectors

We integrated a lightweight self-healing mechanism into our Playwright runner. If a test failed due to a selector mismatch, the runner would:

1. Capture the current DOM state.
2. Send the failed selector and DOM snapshot to a local LLM endpoint.
3. Ask the LLM to propose 3 alternative selectors.
4. Run the test in a "shadow mode" against the original URL to see if any of the proposed selectors actually matched the intended element.
5. If a match was found, open a Pull Request to update the test file with the new selector, tagged with # auto-healed.

This reduced flake-induced ticket volume by 40%.

Code Deep-Dive: A Robust AI-Generated Test Pattern

Below is an example of a test that survived production after passing through our guardrails. Notice the defensive coding patterns:

import { test, expect, Page } from '@playwright/test';

test.describe('Checkout Flow', () => {
  test('User can complete purchase', async ({ page }) => {
    // 1. Setup: Navigate and wait for hydration
    await page.goto('/checkout');
    
    // AI Insight: Instead of hardcoding 'button:has-text("Pay")',
    // use role-based locators which are more resilient to rebranding.
    const payButton = page.getByRole('button', { name: /pay now|proceed/i });
    
    // 2. Defensive Wait: Ensure the button is actually enabled, not just visible
    await expect(payButton).toBeEnabled();
    
    // 3. Action
    await payButton.click();
    
    // 4. Verification: Use polling for async state changes
    // This prevents flakes caused by network latency
    await expect(page.getByText('Order Confirmed')).toBeVisible({ 
      timeout: 15000 
    });
    
    // 5. State Cleanup: Ensure we're not leaving the user in a broken state
    await expect(page).toHaveURL(/.*thank-you.*/);
  });
});

Handling Flakiness with expect.poll

One of the biggest issues with AI-generated tests was the misuse of waitForSelector. The AI often assumed that if an element was in the DOM, it was ready. We enforced the use of expect.poll for dynamic content:

// Bad (AI often generates this):
await page.locator('.loading-spinner').waitFor({ state: 'hidden' });

// Good (Enforced by our linter):
await expect(async () => {
  const spinner = page.locator('.loading-spinner');
  const count = await spinner.count();
  return count === 0 ? 'Hidden' : 'Visible';
}).toBe('Hidden', { timeout: 10000 });

Results: The Final Numbers

After six months of hybrid development, the metrics stabilized and improved:

The initial spike in maintenance was a predictable "valley of despair." Once the guardrails were in place, the speed gains became sustainable.

Lessons Learned

1. AI is a Junior Developer, Not an Architect: Treat LLMs as interns who can write boilerplate but lack context. Always provide rich context in prompts.
2. Fragility is Inevitable: Do not fight the fragility of web apps with more fragile tests. Use semantic locators and defensive assertions.
3. Automate the Review: Manual code review of AI-generated tests scales poorly. Build automated linters that catch common AI mistakes (e.g., hardcoded selectors, arbitrary waits).
4. Keep Humans in the Loop: The AI is good at syntactic test generation. Humans are still needed for semantic validation. Don't skip the review step.
5. Monitor for Drift: LLM models update. A prompt that worked well in June might generate worse code in October. Version-control your prompt templates and re-audit periodically.

Conclusion

Surviving AI-generated Playwright tests in production didn't require banning AI. It required treating AI output as raw, unpolished code that needed a rigorous pipeline of context enrichment, automated linting, and human semantic review. The goal wasn't to let the AI replace the QA engineer, but to let the engineer stop writing the mundane 80% of tests so they could focus on the critical 20% where business logic and user experience actually matter.

The future of testing isn't "human vs. AI." It's "human-directed, AI-accelerated." If you start today, begin with the guardrails. Build the prompt templates, implement the ai-test-audit linter, and then slowly let the AI take over the boilerplate. Your production suite will thank you.