The test that can't disagree

Of all the disciplines AI authorship touches, testing is the one that looks like it should come out ahead. Writing tests was the chore — the part everyone agreed mattered and nobody wanted to do, the tax you paid after the interesting work was finished. A tool that writes the tests for you, fast, in volume, to a coverage number that only climbs, reads like the rare case where the machine takes the drudgery and leaves the judgment untouched. The suite gets bigger. The bar, on every visible measure, gets higher.

It is the discipline whose ground quietly fell out instead, for a reason that was never written down because it was never in danger. A test’s power was never in the assertion it made or the coverage it bought. It was in the independence between the test and the code it checked — two derivations of the same intent, from different sources, by different acts, so that agreement between them carried information. A test is a second opinion, and a second opinion is worth exactly the independence of whoever holds it. When one source writes both the code and the test that vouches for it, the agreement stops meaning anything. The suite is still green. Green now certifies that the code does what the code does.

Testing ran on a single premise nothing in the toolchain ever enforced: the test came from somewhere other than the code. That one fact bought three things the discipline was built around and never had to name, because nothing had ever threatened them at once. An oracle — an independent definition of what correct meant. A design pressure — a force that pushed back on the code and reshaped it before it set. And a compounding record — an accreting, executable history of every intent the team had committed to defend. Co-generation removes the independence in one motion, and all three go hollow while still looking exactly as they did the day before.

1. The collapsed oracle

Every test needs an oracle: something outside the code under test that decides what the right answer is. In a hand-written test the oracle is a person, and it lives in the most boring part of the file — the expected value. assertEqual(total, 42) carries a decision. Someone determined that correct meant 42 and wrote it down, and the writing-down is the whole event. The test passes when the code meets the decision and fails when the code diverges from it, and the failure is information because the decision came from a different place than the code did. Two sources, asked the same question, gave different answers. One of them is wrong, and now you know to look.

When the model writes the assertions, the 42 is no longer a decision. It is a transcription — of what the code produced, or of what the same guess that wrote the code supposes the code should produce. The oracle moved inside the system under test. The expected value used to be the one part of a test guaranteed to originate with a person; it is now the part most likely to be filled in automatically, because it is the most tedious to type and the easiest for a generator to supply by running the code once and freezing whatever fell out. The result certifies that the code does what the author of the code thought it should do, where the author and the certifier are the same author. This is the collapsed oracle: not a test that checks the wrong thing, but a test with nothing outside the code left to check against. A green result confirms the work agrees with itself, and self-agreement was never what anyone wanted a test to confirm. The check launders a guess into the costume of a commitment, and the costume is convincing because it is, character for character, the costume a real commitment wears.

2. The ratifying test

Verification was always only half of what a test did. The underrated half was generative: writing the test made the code better. Test-driven development was the explicit version, but the effect did not depend on the ceremony. Naming the cases, enumerating the edge inputs, stating the contract the function had to honor — the act of writing the test interrogated the code, and the interrogation changed it. A test written from the intent, before or independently of the implementation, is an adversary to the code. It encodes what the code must do and dares the code to fail. Half the bugs a good test suite prevents were never caught by it; they were designed out by the person who, in writing the test, noticed the case the implementation was about to forget.

A co-generated test cannot push back, because it was derived from the code that has to pass it. It arrives pre-shaped to the implementation’s exact contour. The arrow that used to run from test to code — the test specifying, the code conforming — reverses: now the code is written first and the test is fitted around it. Whatever the implementation happens to do becomes, by construction, what the test expects. The edge case the code forgot is the edge case the test also forgets, because the test learned the edges from the code and there was nowhere else for it to learn them. This is the ratifying test: it does not examine the work, it notarizes it. The discipline kept the half that produces a green light and lost the half that improved the design — and only the kept half is counted. Coverage, test count, suite size all measure ratification. The design pressure a test used to exert has no counter anywhere, so its disappearance registers as nothing changing at all. The suite grows while the force that justified writing it is quietly subtracted.

3. The fossilized guess

A test suite’s deepest value is that it compounds. It accretes — each defended intent written down once and enforced forever — so that years on, the suite is a permanent executable record of everything the team decided the system must never stop doing. That record is what makes a codebase changeable. You can rewrite any part of it and the suite tells you the instant you have broken a commitment, which means the commitments, and not the current code, are the thing being preserved. The regression net catches error — divergence from intent — because intent is precisely what was encoded in it.

A co-generated suite accretes plausibility instead of intent. What builds up over time is not a record of decisions but a cast of whatever the code first happened to do, frozen at the moment of generation and trusted thereafter. The team comes to treat the cast as the definition: the tests are green, the tests describe the system, leave them alone. Now the suite constrains the code to keep doing what was first guessed — including the parts that were guessed wrong. Refactor, and tests “break” that were never defending a real commitment; the team is forced to choose between preserving a behavior nobody ever intended in order to keep the suite green, or deleting the one artifact that looked like coverage. This is the fossilized guess, and its signature is a suite that resists every change with equal force. A net that flags all change indiscriminately is not protecting intent. It is protecting the past — its own first draft of the past, mistakes and accidents included — and calling the resistance safety. The thing that used to make code safe to change becomes the thing that makes it unsafe to touch.

Closing

The green suite was always a proxy. It stood in for a sentence no one said out loud: an independent mind checked this against an intent it had derived on its own. The proxy was honest for decades because the independence was guaranteed by cost — building a real test took a second, separate act of understanding, and that expense was the very thing that made the second opinion second. The price was not the friction in the discipline; it was the discipline. Generation removed the price, and with it the independence, and left the proxy standing exactly where it stood: fuller, greener, larger than it has ever been, and emptied of the single property it existed to represent.

The teams that see this stop counting tests and start asking a narrower thing of each one: does this encode a commitment a person made that a generator could not have produced by reading the code? The tests that pass that question are verification, and there are fewer of them than the coverage number implies. Everything else is documentation of the present — sometimes useful as a description of what the system does today, never evidence that today is what anyone wanted. The two read identically on the dashboard, and the dashboard is the wrong instrument for telling them apart, because the difference is in where the expected value came from and a coverage report does not record provenance.

So the suite keeps passing, and the passing keeps getting easier to produce, and both facts are true at once and point in opposite directions. The dangerous test is not the one that fails. A failure is the suite doing its job — two sources disagreed, and the disagreement found you. The dangerous test is the one that has never once failed for a reason its author did not already know, because that is what a real check and an elaborate mirror both look like, right up until the day the difference is the only thing that would have mattered.