AI has made the first hour of test automation dramatically cheaper.

Describe a workflow, paste a requirement, or point an agent at an application, and it can produce a respectable first draft. The code may include page objects, fixtures, assertions, and comments. It looks like progress because it is progress.

But creation was never the full cost.

The expensive part begins after the test joins the repository.

Now the team must decide whether the assertions are meaningful, whether the selectors are durable, whether the data is safe, whether the test fails for the right reasons, and who will fix it six months later when the interface changes.

AI lowers the cost of producing tests. It can also increase the number of tests you are responsible for.

That is the trade-off teams need to understand.

Generated code can fail before generated tests even run

AI-generated frontend changes frequently work in a local environment and fail in CI. The problem is not always the quality of the generated code. It is the missing context around it.

The CI environment may use a different Node version, package lock state, browser build, environment variable set, feature flag, operating system, or resource limit.

This analysis of why AI-generated frontend changes fail in CI even when local runs pass highlights an important principle: AI tends to optimise for the context it can see.

If the model sees the component but not the pipeline, it can create a locally valid solution that violates an invisible constraint.

The same thing happens with generated tests. A model may assume:

  • a clean database;
  • one browser worker;
  • stable test IDs;
  • immediate API responses;
  • permission to create arbitrary users;
  • access to secrets that do not exist in CI;
  • a desktop viewport;
  • English copy.

A useful generation prompt should include operational constraints, not only the user journey.

Tell the model where the test runs, how data is created, what can execute in parallel, which selectors are preferred, and what evidence should be captured on failure.

The prompt is part of the test architecture.

Claude can write Playwright. That is not the decision

A modern coding model can generate Playwright tests quickly. It can also refactor helpers, interpret error logs, and suggest alternate selectors.

The interesting question is not whether it can write the code. It can.

The better question is what changes in maintenance, review, and signal quality after generation becomes abundant.

The article When Claude Writes Your Playwright Tests describes the shift well. Reviewers need to evaluate more than syntax.

A generated test can be perfectly valid TypeScript and still be a poor test because it:

  • asserts incidental copy;
  • uses a selector that matches multiple elements;
  • hides an unreliable action behind retries;
  • duplicates coverage that already exists;
  • skips negative behaviour;
  • passes after an optimistic update but before server confirmation;
  • creates data it never cleans up.

Code review for generated tests should begin with intent:

  1. What product risk is this test reducing?
  2. What failure would it catch?
  3. Why is the browser the right layer?
  4. Does it fail when the product is broken?
  5. Will the team understand the failure report?

Only then should the reviewer care about formatting and helper reuse.

The test count can become a vanity metric

When generation is expensive, teams are selective. When generation becomes almost free, restraint becomes more important.

It is easy to produce 200 tests from requirements, tickets, and recorded sessions. The number looks impressive. Then the product changes, 47 tests fail, and nobody knows which failures matter.

This framework on what to measure before trusting a Playwright test suite generated with Claude recommends measuring the suite as an operating system rather than a code output.

Useful metrics include:

  • percentage of failures that represent real defects;
  • median diagnosis time;
  • maintenance hours per month;
  • repeated failures from the same cause;
  • coverage of critical workflows;
  • tests that have never failed meaningfully;
  • retries required for a green pipeline;
  • reviewer effort per generated test.

That final metric matters. A model can produce code in seconds, but a senior engineer may spend 20 minutes proving that the code is safe and useful.

AI does not eliminate review cost. It moves the work.

AI testing platforms also contain human work

A platform may advertise AI-generated tests, self-healing, natural-language instructions, or autonomous maintenance. Those capabilities can reduce effort. They can also introduce new operating tasks:

  • reviewing generated steps;
  • validating prompts;
  • inspecting traces;
  • approving suggested fixes;
  • resolving ambiguous assertions;
  • monitoring model usage;
  • handling false repairs.

The article on estimating the real cost of an AI testing platform when prompt reviews, traces, and human approvals add up offers a healthier cost model.

Do not compare only subscription price with an open-source licence price.

Compare the full workflow:

Total cost = platform cost
           + test creation time
           + review time
           + maintenance time
           + infrastructure
           + failure diagnosis
           + training and adoption

Enter fullscreen mode Exit fullscreen mode

The same equation should be used for a home-grown framework.

“Free” software can be expensive when specialised engineers spend a significant part of every sprint maintaining it. A paid platform can also be expensive when it requires constant human supervision.

The correct unit is not licence cost. It is cost per trustworthy result.

Build versus buy is mostly a staffing decision

The comparison between Endtest and hand-rolled Playwright for fast-changing frontends frames the decision around ownership.

A custom Playwright framework can provide excellent control. Your team can define fixtures, abstractions, reporters, environment management, and CI behaviour exactly as needed.

But somebody owns all of it.

That ownership includes:

  • browser and dependency upgrades;
  • authentication helpers;
  • test-data utilities;
  • parallel execution rules;
  • retry strategy;
  • screenshots, videos, traces, and logs;
  • integrations with email, SMS, and storage;
  • onboarding documentation;
  • support for less technical contributors.

A managed system such as Endtest trades some framework-level freedom for a smaller internal maintenance surface and broader accessibility.

Neither model is automatically better.

The mistake is selecting a code-first framework because the initial proof of concept was easy, then discovering that the team unintentionally created an internal product.

A useful question is:

Do we want to build tests, or do we also want to build and operate a testing platform?

Some organisations should answer yes. Many should not.

Reliability is different for streaming AI applications

AI applications introduce behaviours that traditional CRUD tests do not encounter as often:

  • streamed text arrives incrementally;
  • the UI changes while the response is still in progress;
  • intermediate state may be incomplete but valid;
  • model outputs vary;
  • cancellations and retries matter;
  • network latency changes the rendering sequence.

A brittle test may wait for exact text, assume a fixed number of updates, or interact before the stream is complete.

The guide to benchmarking browser test reliability for AI apps that stream responses and update state incrementally recommends testing the state machine rather than the precise output.

For example, verify that:

  • a response begins within an acceptable period;
  • the loading or streaming state is visible;
  • content grows rather than being replaced incorrectly;
  • controls are enabled or disabled at the right times;
  • cancellation stops the stream;
  • the final state is persisted correctly;
  • an error offers a recovery action.

For non-deterministic content, assert structure and constraints instead of a complete sentence.

You may check that the response is non-empty, contains required facts, follows a schema, avoids prohibited content, or receives an acceptable evaluation score.

The more variable the product, the more deliberate the oracle must be.

Stable coverage does not require a heavy framework

Some teams need deep framework control. Others mainly need dependable coverage of business-critical journeys without dedicating several engineers to test infrastructure.

This practical review of Endtest for teams that need stable coverage without a heavy framework represents the second group.

The value proposition is not that code is bad. It is that infrastructure work has an opportunity cost.

A team that avoids building its own reporting, test editor, execution grid, collaboration layer, and integrations can spend that time improving product coverage.

The trade-off is accepting the platform's model and limits. That is why a proof of concept should test the awkward workflows, not only login.

Try:

  • multi-step authentication;
  • file uploads and downloads;
  • dynamic tables;
  • cross-domain navigation;
  • email and SMS verification;
  • failure diagnosis;
  • role-based collaboration;
  • CI execution under realistic parallelism.

Any tool looks good when the demo is “open page, click button.”

Human approval should focus on risk

AI-generated tests should not require a committee meeting. But they should receive review proportional to their impact.

A low-risk visual check for an internal page may need a quick inspection. A payment, access-control, deletion, or regulatory workflow deserves deeper review.

A lightweight approval checklist can ask:

  • Is the workflow worth testing at the browser layer?
  • Are secrets and personal data handled safely?
  • Can the test run in parallel?
  • Are assertions tied to user outcomes?
  • Does failure produce enough evidence?
  • Is cleanup defined?
  • Who owns future maintenance?

This prevents the two common extremes: blindly accepting generated code or making AI assistance so bureaucratic that nobody uses it.

The goal is not perfect generation.

The goal is controlled leverage.

The new bottleneck is judgement

For years, the bottleneck in browser automation was writing the code.

AI is removing part of that bottleneck.

What remains is judgement:

  • choosing the right risks;
  • deciding the right test layer;
  • defining stable assertions;
  • controlling test data;
  • evaluating failures;
  • deleting low-value coverage;
  • choosing which infrastructure to own.

That is good news. These are more valuable problems than remembering the syntax for a locator.

But they are not automatic.

AI can write the test.

Your team still owns whether anyone should trust it.