AI in Engineering: Identical Gates, Honest Metrics, and No Vanity Numbers

If you only take one rule from how a disciplined team uses AI in engineering, take this one: AI-assisted code passes the exact same gates as human-written code, plus a disclosure of which tools were used. There is no relaxed review because “AI wrote it,” no skipping the automated checks, no separate track. The engineer who signs the change owns it. Everything else, the tooling, the metrics, the honest caveats, follows from that single principle.

Part 1 covered the operating model a board cares about: exit ramps, capped spend, and a gated program. This part is about the codebase.

The approved toolset, and what AI is allowed to do#

A standardized, centrally managed set beats a free-for-all. That means a primary AI coding assistant on a team plan with central billing, administration, and usage reporting, and an established, widely-adopted AI code-review platform that independently reviews every meaningful change. Optional inline completion tools are allowed by preference. Anything else requires a documented request and review, and free or personal AI accounts are not permitted for company work, because they take your code outside the agreements that protect it.

Under human direction, AI helps with drafting and reworking code, writing tests, investigating bugs, drafting design documents and tickets, generating documentation, and reviewing changes. What it does not do is ship anything unsupervised or make a customer-affecting decision on its own.

How AI-assisted code review works in practice#

“Identical gates” is easy to assert and easy to fake. Here is what it looks like as a real change moves through the pipeline.

Figure 1: The same gates run whether or not AI was involved. AI adds a draft at the front; it removes nothing from the pipeline.

A change is AI-drafted under direction, then passes automated linting and strict type-checking, the full test suite (on the order of a thousand-plus tests), a domain-specific regression check on a small canary corpus, and the independent automated reviewer, which on a typical change raises a couple of findings that a human triages, some applied and some rejected after verification against the real code, before a human review and sign-off and a human pushing the change. Security-sensitive areas, anything touching authentication, secrets, build and deployment, or infrastructure, require senior human review regardless of AI involvement. Automated quality and security checks cannot be bypassed to push AI output through; the underlying issue has to be fixed.

Why “percentage of code written by AI” is a vanity metric#

The question every executive has been trained to ask is “what percentage of your code is written by AI?” The right answer is to refuse the question, and to gently encourage skepticism of any vendor who quotes a number.

Lines of code is a vanity measure. It rewards verbosity, double-counts rewritten work, and tells you nothing about quality or risk. Worse, once a human edits an AI draft, “AI-written versus human-written” is no longer a meaningful line to draw. The published percentages from large vendors are marketing numbers with undisclosed methods. Most new code today is drafted with AI assistance and then revised, reviewed, and signed by a human who is accountable for it, and every line that reaches the codebase has passed human review and the same automated gates as any other code. That is the honest description. A percentage is not.

Measure what actually matters instead: how fast you ship, how thoroughly you review, and whether defects escape to production.

Why AI should make code review deeper, not optional#

This is the most underrated effect of an independent automated reviewer on every change, and it is measurable from your own baseline.

Figure 2: A common baseline is that fewer than one in ten non-trivial changes receive a substantive human review comment. A reasonable 90-day target is 40 percent or more.

Before such a program, a common baseline is that fewer than one in ten non-trivial changes get a substantive human review comment (in one measured case, under 7 percent). A reasonable 90-day target is 40 percent or more. AI raises review depth because the automated reviewer surfaces findings a busy human would skim past, and because the team deliberately spends some of the speed gain on reviewing more carefully. That is the trade you want.

There is a reason this matters more than it looks. An independent check compounds with the first pass multiplicatively, not additively. If an automated step is wrong on some fraction of changes and a human reviewer independently misses that same fraction, the residual error is the product of the two, not their average. A 10 percent automated error rate behind a reviewer who catches 90 percent of mistakes leaves on the order of 1 percent, an order of magnitude better than either step alone, and better real rates push it lower still. The catch is the word independent: this only holds if the human review is a genuine second look, not a rubber stamp of the automated pass. Keep the two failing independently and the residual error, and the cost of the errors that escape, drop by orders of magnitude. That is what human-in-the-loop buys, and it is why review depth is a number worth raising rather than a tax to minimize.

DORA-class metrics that survive a board meeting#

Capture a baseline before scaling AI. A workable baseline set, all drawn from your own history, is throughput (merged changes per engineer per week), cycle time at the median broken out by change size, the substantive-review rate, and the revert rate. Set explicit improvement targets against it, for example cycle time down 15 percent on normal-sized changes and 20 percent on large ones, throughput up 15 to 20 percent, and review depth raised to 40 percent or more. Report the actual movement at the 30-day checkpoint and the 90-day gate, and do not claim a gain before the data exists.

For defect rates, resist the urge to fake a comparison. Early on you will not have enough attributed data to compare AI-assisted and human-written defects fairly, because nearly every change involves some AI assistance. The question that actually matters is whether AI use degrades quality overall, and that is answered at the 90-day gate by requiring that the rate of defects escaping to production, and error monitoring, do not get worse as AI use scales. When you benchmark, use well-established, DORA-class engineering metrics (throughput, lead and cycle time, change-failure rate, and time to recovery) against your own measured baseline, not a vendor’s marketing figure.

Where AI slows teams down#

A document with no caveats is not credible. There are three honest categories where AI creates rework, each with a mitigation already in place.

First, confident-but-wrong suggestions. AI assistants, including automated reviewers, sometimes produce a wrong answer stated confidently. The rule is that every AI finding is verified against the real code before it is acted on, and automated findings are advisory, never auto-applied.

Second, review overhead. AI output that looks finished still has to be reviewed in proportion to its volume. Unreviewed AI output is negative productivity, which is exactly why you spend some of the speed gain on deeper review rather than banking all of it.

Third, going fast in the wrong direction. AI can iterate quickly down a dead end on environment-specific quirks. The working rule is to measure before concluding, and to stop and escalate rather than brute-force a fix.

How intellectual property is protected when AI writes code#

Vendor terms bar training on your code or data, the automated reviewer works under its own data agreement and on the change only, and code stays in private repositories with no free or personal AI account allowed to touch it. On licensing, do not bulk-import AI-suggested code of unknown origin; everything enters through human-reviewed submissions, and any output suspected of reproducing third-party copyrighted code is a reportable incident. Because developer tools are limited to internal data, customer data and intellectual property are never in the exposure surface to begin with.

The honest list#

The most credible thing an engineering organization can publish about its AI program is a short list of what it does not yet claim. Do not quote a percentage of code written by AI; it is a vanity metric. Do not present an AI-versus-human defect comparison you cannot support; establish the baseline and answer the real question, whether quality regresses, at the 90-day review. Do not state a productivity gain as a number before the baseline data exists. Each of these has a date or a mechanism attached. None of them is a surprise being hidden; they are the parts of an honestly run program that take a measurement period to answer.

Run AI in engineering this way and the answer to “what would break if we removed AI tomorrow?” is: nothing, functionally. The product keeps running, the code is ordinary reviewed code, and development returns to its previously measured pace. The exposure is speed and review depth, not whether the software works, and it is fully reversible. That reversibility is not a weakness of the approach. It is the whole point.

Red Black Tree helps engineering and product organizations adopt AI under governance: identical gates for AI and human work, outcome metrics instead of vanity numbers, and a defensible quality posture for any board, partner, or acquirer review. Get in touch for the full operating model.

Frequently asked questions

What percentage of code is written by AI?

A number worth refusing. Lines of code reward verbosity and say nothing about quality or risk, and once a human edits an AI draft the AI-versus-human line is meaningless. Measure shipping speed, review depth, and escaped defects instead.

Should AI-generated code be reviewed differently than human code?

No. It passes identical gates plus a disclosure of which tools were used. Security-sensitive areas such as authentication, secrets, and deployment require senior human review regardless of AI involvement.

Which metrics measure AI-assisted engineering?

DORA-class metrics against your own measured baseline: throughput, cycle time, change-failure rate, and time to recovery, plus the substantive-review rate and defects escaping to production, reported at 30-day and 90-day gates.

Where does AI slow engineering teams down?

Three known ways: confident-but-wrong suggestions, review overhead proportional to output volume, and fast iteration down dead ends. The mitigations: verify every finding against real code, spend part of the speed gain on deeper review, and measure before concluding.