Cheap code needs a verification budget

When capable models get radically cheaper, organizations can generate more candidate code for less money.

Intent still has to be clarified. Output reviewed. Tests run. Regressions investigated. Production behavior observed.

The supply curve for candidates falls. The scarce resource moves to trustworthy acceptance.

The useful cost equation is: accepted-change cost equals inference, steering, retries, review, CI, rework, and expected failure cost.

Most model comparisons stop at the first term. That makes a verbose cheap model look economical even if it consumes more senior attention.

A costly model can win if it produces fewer dangerous near-misses. Token price matters — but only inside the whole system.

DORA's 2025 research found AI adoption associated with higher delivery throughput and higher delivery instability at the same time.

Evidence:DORA findingsDeveloper survey

In Stack Overflow's 2025 survey, 66 percent of respondents named almost-correct AI answers as a frustration. 45 percent said debugging AI-generated code took more time.

Those sources are observational and self-reported — not proof that AI caused every failure. Together they identify the mechanism a local test should measure: more output can arrive with more uncertainty attached.

A large Cursor adoption study found a short-lived increase in development velocity alongside persistent increases in static-analysis warnings and code complexity.

Evidence:Cursor quality study

Its repository-level design cannot show that every warning became a defect. It still undercuts PR volume as a complete productivity measure.

The easy response is to replace source review with specifications and automated tests.

That is directionally right and still insufficient.

OpenAI audited a difficult 138-task subset of SWE-bench Verified and identified material specification or test issues in 59.4 percent of it.

Evidence:Benchmark audit

Verification therefore needs layers.

Deterministic checks should cover invariants that can be formalized. Review should focus on intent, risk, and evidence the checks cannot express.

Staged rollout and telemetry should test assumptions that only production can reveal.

Every escaped failure should become either a new check, a sharper brief, or an explicit accepted risk. Proof is a maintained system, not a final gate.

Start by measuring reviewer minutes per accepted change, CI minutes per accepted change, first-pass acceptance, reverts, escaped defects, and rework within 30 days.

Segment by task risk and workflow: autocomplete, human-led assistance, delegated scoped work, and autonomous delivery inside verified boundaries.

A single organization-wide AI productivity number hides the exact task classes where cheap generation is creating expensive review.

That may mean faster deterministic tests, representative evaluation tasks, review tooling, observability, or time for senior engineers to encode recurring comments as constraints.

The budget should follow measured queue pressure, not a fixed percentage.

If generation capacity doubles and review capacity does not, work in progress expands even when every individual demo looks faster.

Give several models the same 20 real tasks and count total cost through acceptance: API spend, attempts, human steering and review, CI consumption, first-pass acceptance, and defects found after approval.

Blind the maintainer reviewing the output. Repeat trials for nondeterministic tasks and separate low-risk mechanical work from high-blast-radius changes.

The theory predicts that the cheapest model per token will often not be the cheapest per accepted change — and that verification effort will become a larger share of cost as generation prices fall.

It fails where outputs are deterministically checkable and model spend remains dominant after rework is included.

That is a good boundary: automate aggressively where proof is cheap, and slow down where the organization is still building the verifier.