Mudit Singh is the Co-Founder and Head of Growth at TestMu AI, an AI-native unified enterprise test execution cloud platform.
Developers are shipping tests generated by AI coding assistants faster than most teams can review them. Some are brittle. Suites may become flaky. Senior engineers could lose time maintaining them. Dashboards may stay green while customers find defects.
The common conclusion is that AI-generated tests are the problem. Some may say they’re shallow, tied to selectors that break after a refactor and often blind to the intent behind a workflow.
That criticism is not entirely wrong, but I believe it is aimed at the wrong layer.
Many teams are asking regression tests to prove product quality, a job they were never designed to do. AI-generated code did not create that weakness. It increased the rate of change until the weakness became impossible to ignore.
What Test Suites Were Actually Built For
Automated tests were meant to answer a narrow question: After a change, did something that used to work stop working?
Over time, organizations expanded the meaning of a passing suite. Green dashboards started to stand in for product quality. But a passing suite proves only that the behaviors represented in those tests still behave as expected. It does not prove that the right behaviors were tested or that the implementation matches the requirement.
AI-assisted development makes that distinction harder to ignore. In Stack Overflow’s 2025 survey, 66% of developers said their biggest frustration with AI tools was output that was “almost right, but not quite,” while 45% said debugging AI-generated code took more time.
“Almost right” is exactly where regression thinking struggles. The code may compile. It may look reasonable in review. And if the implementation and test came from the same prompt or mistaken assumption, the test can validate the same misunderstanding.
That is not a test-generation problem. It is a verification problem.
Testing And Verification Are Different Disciplines
Testing is what you do to a build before you ship. It involves predefined checks that tell you whether you broke something you already knew about.
Verification is what you do to the running product. You observe what the software actually does, from the outside, the way a user would experience it, and you judge it against the requirement rather than against the code or the prompt that produced it. That can happen before a pull request opens or in production.
Teams have conflated the two for 20 years because there was no affordable way to verify, so they wrote more tests and hoped coverage was enough. AI-generated code broke that arrangement. Change now moves too fast for a pre-deploy testing culture to certify what ships.
Why AI-Generated Tests Are Actually Fine
Once regression testing and product verification are treated separately, AI-generated tests can become easier to evaluate.
They are cheap regression coverage. Let coding assistants generate unit tests, integration checks and browser tests where the goal is to protect known behavior. Review whether they assert the requirement and are worth keeping.
If a low-value test breaks repeatedly after harmless UI changes, regenerating or deleting it may be cheaper than assigning a senior engineer to preserve it forever.
The higher-signal work belongs in a separate verification layer. That is where teams should validate critical workflows on the running product: a checkout that renders differently in Safari, a multistep journey that fails on the third screen, an accessibility issue unit tests never see or a mobile flow that behaves differently on a real device.
Modern AI-native testing platforms can hand a coding agent a real browser and return a pass or fail result with evidence before the pull request opens. Whichever tool you use, the check has to start from the requirement rather than the code, or it cannot fail where the code and its self-generated test agree.
The Real Failure
If your quality strategy is failing, the cause is older than AI coding assistants. They only raised the rate of change until an old gap could no longer be explained away.
Google’s testing team published its flaky test numbers in 2016, showing that almost 16% of its tests exhibited some flakiness, with new flaky tests appearing at roughly the same rate existing ones were being fixed.
Blaming today’s maintenance burden entirely on AI-written tests risks missing the older structural problem: quality programs that rely too heavily on pass rates, test counts and automation coverage as proof that the product works.
How To Put This Into Practice
Not sure where to get started? Run these in order. The first one pays for the rest.
1. Audit escaped defects first. Sort last quarter’s customer-reported bugs by whether a regression test could have caught them. That table is your budget argument.
2. Give verification to the owner and a dashboard line. Report test pass rate as regression health. Report validated user workflows separately, with a name against the number.
3. Stop paying senior engineers to repair brittle tests. Cap review of an AI-generated test at one question: Does it assert what the ticket asked for? When a test breaks on cosmetic changes more than on real regressions, delete it and regenerate it.
4. Verify your critical workflows from the outside. Sign-up, login, checkout and billing: These should all be validated on real browsers and devices against the written requirement, on every merge.
5. Move verification in front of the pull request. Give developers and coding agents a real browser to prove what they built and make the evidence part of the review.
The Bottom Line
Stop trying to fix your AI-generated tests. They are doing the job they were built for, and that job is supposed to be simple. Fix the strategy above them: Separate testing from verification and leverage product verification as a discipline with its own layer beside the suite.
Google was stuck at the same flaky test insertion and fix rate in 2016. A generation of coding assistants later, a team still trying to repair its way out of that arithmetic is repairing the wrong thing. The theory of quality behind the tests is what needs fixing.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?