AI Coding Agents Are Changing Testing. But What Do They Really Cost?

When AI Writes The Code And The Test

Here is the shift that matters, and it is specific to this moment. For the first time, the same kind of intelligence can write the software and write the tests that verify it.

Developers have always written tests for their own code, and the discipline has always leaned on a second person, a reviewer or a QA engineer, to catch what the author missed. What is new is a pipeline where code and tests come out of one pass, from one reading of the ticket.

But a test written by the same intelligence that wrote the code inherits its blind spots. The agents work well and improve every quarter. The weak point is the structure around them, a loop that generates and validates its own work with nothing independent in the middle.

This isn’t a token-cost problem, and it isn’t a story about agents being unreliable. The individual pieces work well. The cost lives in the structure: You have built a loop that generates and validates its own work with no independent party in the middle, and you are paying for confidence that was never actually earned.

Why The Invoice Is The Wrong Number

The bill for a missing independent check is always deferred, and it is always larger for having waited. It does not arrive as a line item. It arrives as an incident: a regression every generated test waved through or a failure mode no agent was adversarial enough to imagine, because it was reasoning from the same assumptions that produced the bug.

There is a second cost that accrues the whole time. The CTO’s three engineers were not an accident. When a team senses that a green pipeline no longer guarantees what it used to, they do the rational thing and start verifying it themselves.

The efficiency the agents were brought in to deliver leaks back out through human rechecking, and the organization pays twice: once for the agents, and once for the people making sure the agents were right.

Three Places The Hidden Cost Accumulates

Pay careful attention to these areas:​

1. Triage Overhead: Non-deterministic runs produce a kind of failure that did not exist before, neither a real bug nor a broken test, but an artifact of the agent’s own variability. This is what the fintech CTO was really paying for. Telling these apart is slow, manual work, and it consumes the time the agents were supposed to give back.

2. Invisible Maintenance: There is a comforting assumption that AI-generated tests maintain themselves, and the better systems do, up to a point. But an agent that adapts to a changed interface can just as easily adapt to a broken one and keep passing. The maintenance cost did not disappear. It changed shape, from visibly updating scripts to invisibly auditing whether your agents still test what you think they test. Invisible work is the most expensive kind because no one budgets for it until it fails.

3. Eroded Trust: This is the one that compounds. Every result an engineer cannot immediately explain chips away at their confidence in the system, until they stop believing a green build on its own. Once that happens, every downstream efficiency the agent promised is spent back on proving it right.

What The Teams That Win Are Doing Differently

They put the agent at authoring time, not runtime. The agent explores, reasons and proposes, and what it hands back is an artifact: test code, in the repo, in a framework the team owns. The intelligence happens once, under supervision, and the execution afterward is boring and identical every time.

Variability is worth a great deal when an agent is generating options and far too costly when it is issuing verdicts, so the winning teams spend it on the first and refuse to spend it on the second.

They make agent output reviewable. Because the agent produces real test code rather than a black-box decision, an engineer can read it, question it and correct it before it ever runs. The agent’s reasoning becomes something the team can inspect, not something they have to take on faith.

They keep a deterministic floor. Once a test is authored and reviewed, it runs the same way every time, so a green result means what it always has. The creativity stays at the top of the pipeline, where it helps and never leaks into the layer whose entire job is to give you a straight answer.​

The Real Price Of The Agentic Era

AI coding agents are going to write most of the world’s tests within a few years. I run a company betting on exactly that, so read my enthusiasm accordingly. But the teams that get there first won’t be the ones who found the cheapest tokens. They’ll be the ones who noticed early that they were buying certainty, not compute, and who refused to let a system that changes its mind be the thing that tells them the software is ready.

The invoice was never the cost. The cost is what your team stopped believing.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?