Build a fair AI-assisted versus human-only delivery comparison

The comparison everyone wants and nobody designs well

Someone on the leadership team asks: "Are the teams using AI coding tools actually delivering better?" The analyst pulls two groups -- teams with AI tooling enabled and teams without -- computes average cycle time and defect rate, and presents a slide. The AI-assisted group looks faster. The room nods. Nobody asks whether the AI-assisted group was already working on simpler tasks, in newer repositories, with more senior engineers, during a lighter on-call rotation.

That comparison is not wrong in the way a calculation error is wrong. It is wrong in the way a courtroom verdict is wrong when the jury only heard one side. The numbers are accurate. The design is not.

This guide is for the analyst, engineering manager, or platform lead who needs to build a comparison that withstands scrutiny -- one that controls what can be controlled, measures multiple dimensions simultaneously, and labels the remaining uncertainty instead of hiding it. The goal is a reusable cohort design, not a one-time dashboard.

Quick-start path

If you need a working comparison in the next two weeks, follow this sequence:

1. Define authorship categories using the classification worksheet below. Do not guess from code style; retain an "unknown" bucket.
2. Select a matching window of four to eight weeks where both cohorts were active.
3. Match changes by repository, task type (bug fix, feature, refactor), and scope (lines changed or files touched).
4. Measure three things together: throughput (merged changes per engineer-week), review burden (review rounds and reviewer time), and follow-up defects (patches or reverts within 30 days).
5. Run the comparison at two levels: aggregate across all matched pairs, and within each repository separately.
6. Audit self-report accuracy by spot-checking a sample of PRs against available tool telemetry (see the reporting-bias audit procedure below).
7. Write down what the comparison can and cannot claim, using the limitations checklist at the end.

Each step is detailed in its own section below. Skip ahead to the cohort worksheet if you already have your categories defined.

Define authorship categories honestly

The first failure in most comparisons is misclassification. A pull request is labeled "AI-assisted" because the author had Copilot enabled, even though the actual code was hand-written. Or a change is labeled "human-only" because the author did not mention AI, even though they pasted generated code from a chat session.

You need categories that reflect what you can actually observe, not what you wish you could observe.

Recommended classification scheme

Why the "unknown" bucket matters. If you force every change into AI-assisted or human-only, you will either inflate one category with guesses or quietly drop data. Both distort the comparison. Keep "unknown" as a first-class category. Report its size. If it exceeds 30% of your total changes, your classification mechanism needs work before the comparison is meaningful.

Do not classify based on code style, comment density, or commit message patterns. These are unreliable proxies that introduce a new confound: you would be selecting for style rather than authorship method.

Adapting to different team contexts

• Small team (fewer than 10 engineers): Self-report at PR creation is lightweight enough. Add a required dropdown field to your PR template.
• Large org with multiple tools: Distinguish between tools if you can (e.g., inline completion vs. agent-driven implementation). If you cannot distinguish reliably, group them all as "AI-assisted" and note the heterogeneity.
• Teams mid-rollout: You will have a natural before/after boundary. Use it, but be aware that time-based confounds (seasonal workload, hiring waves) will overlap with the rollout date.

Auditing self-report accuracy

Self-report is a necessary starting point, but trusting it blindly is a design flaw you can partially detect. Here is a concrete procedure:

1. Sample. Pull a random sample of 20 PRs from each cohort (AI-assisted and human-only) per quarter.
2. Cross-reference tool telemetry. If your AI tool logs session activity (file opens, suggestion accepts, chat queries), check whether the sampled PRs show tool activity that matches the self-reported label. A PR tagged "human-only" where telemetry shows 15 accepted suggestions during the same time window and touching the same files is a candidate mislabel.
3. Interview spot-checks. For 5 of the 20 sampled PRs in each cohort, ask the author: "Walk me through how you wrote this change." Frame it as process research, not compliance auditing. Note whether their description is consistent with the tag.
4. Compute a disagreement rate. Count the PRs where telemetry or interview contradicts the self-reported label, divided by total sampled PRs. This is your estimated misclassification rate.

Why this matters beyond listing "reporting bias" as a confound. Most guides (including many internal analysis docs) name reporting bias and move on. The audit procedure above gives you a number -- the disagreement rate -- that you can report alongside your cohort results. A finding of "+20% throughput for AI-assisted PRs" hits differently when accompanied by "our estimated misclassification rate is 8%" versus "we have not checked." It converts an abstract limitation into a quantified one.

If your AI tools do not provide telemetry you can cross-reference, the interview spot-check alone still produces useful signal. Five conversations per cohort per quarter is roughly two hours of work.

Match changes to reduce confounding

Raw group averages are almost always misleading. The AI-assisted group may disproportionately contain greenfield features in modern repositories. The human-only group may contain legacy bug fixes in poorly-tested codebases. Comparing their averages tells you about the mix of work, not about the tool.

Matching dimensions

Match each AI-assisted change to a human-only change (or small group of changes) along these dimensions, in order of importance:

1. Repository. Code complexity, test coverage, CI speed, and review norms vary enormously across repositories. Cross-repository comparisons are almost never fair.
2. Task type. Bug fix, feature, refactor, infrastructure/config. A bug fix in a legacy service and a greenfield feature in a new microservice are not comparable units.
3. Scope. Use a rough size bracket: small (fewer than 50 lines changed), medium (50-250), large (250+). Exact line counts are noisy, but order-of-magnitude matching removes the worst distortions.
4. Time window. Changes should fall within the same two- to four-week window to control for seasonal effects, on-call load, and org-wide disruptions.

Selection bias you cannot match away

Even with careful matching, several biases remain:

• Self-selection. Engineers who adopt AI tools early may be systematically different from those who do not -- more senior, more curious, more comfortable with new workflows. You are comparing the combination of engineer and tool, not the tool alone.
• Task selection. Engineers may choose to use AI tools on tasks where they expect the tool to help, and avoid it on tasks where they expect friction. The AI-assisted cohort may be enriched for "AI-friendly" tasks.
• Reporting bias. Engineers who had a good experience with AI may be more likely to tag their PR as AI-assisted. Those who discarded AI output and rewrote it may tag it as human-only. Use the audit procedure above to estimate the magnitude of this problem rather than treating it as a blanket caveat.

None of these can be eliminated by matching. They require a randomized or phased rollout design, discussed in the limitations section.

Measure three things together with consistent denominators

A comparison that only measures throughput will miss quality problems. A comparison that only measures defects will miss the throughput gains that justify accepting slightly higher defect rates. You need at least three dimensions measured with consistent denominators.

Metric definitions

Why consistent denominators matter. If you measure throughput per engineer but defects per line of code, a team that ships larger changes will look artificially defect-prone. Pick one denominator per metric and hold it constant across cohorts.

Collecting the data

For throughput and review burden, your version control and code review platform already have the raw events. For follow-up defects, you need a way to trace a fix back to the change that introduced the problem. If your team does not already link patches to originating PRs, start that practice now -- it is the hardest data to reconstruct after the fact.

Examining engineering delivery and code quality across both cohorts requires that you instrument both groups identically. If the AI-assisted group has richer telemetry because the AI tool logs more, normalize to the data you have for both groups.

Hypothetical worked example: when aggregate and within-repository results disagree

The following scenario is entirely hypothetical. All numbers are illustrative.

Setup

An analyst at a mid-size company (hypothetically, 60 engineers across 8 repositories) runs a cohort comparison over an 8-week window. She classifies 410 merged PRs: 180 AI-assisted, 190 human-only, 40 unknown. She discards the unknown bucket from the comparison but reports its size. The unknown rate is 40/410 = 9.8%, well below the 30% threshold.

She runs the self-report audit: 20 sampled PRs from each cohort, cross-referenced against tool telemetry and 5 spot-check interviews per cohort. She finds 2 disagreements out of 40 sampled PRs -- a 5% estimated misclassification rate. She proceeds with confidence in the labels but reports the rate.

She matches on repository, task type, and scope bracket, producing 140 matched pairs.

Aggregate results

At the aggregate level, AI-assisted changes are faster to produce but require more review rounds and generate more follow-up defects. The throughput gain is partially offset by downstream costs.

Within-repository results

Within payments-service and internal-tools, the differences are small and likely within noise. Within user-api, AI-assisted throughput is meaningfully higher, but so is the defect rate. Within data-pipeline, AI-assisted defect rates are substantially elevated.

How the aggregate misleads

The aggregate +26% throughput figure looks like a clean win tempered by quality costs. But the within-repository view reveals that user-api contributes a disproportionate share of the AI-assisted matched pairs (hypothetically, 55 of 140), because that repository had the highest AI adoption rate. Its large throughput gap pulls the aggregate upward. Meanwhile, data-pipeline contributes fewer pairs (hypothetically, 20 of 140) but has the worst defect differential. If you acted on the aggregate alone, you might expand AI tooling everywhere. The within-repository view suggests expanding in internal-tools (modest gain, negligible quality cost) and investigating data-pipeline (elevated defects warrant a closer look at what kinds of changes are failing).

This is Simpson's paradox applied to your delivery data: the aggregate trend can reverse or exaggerate when you break it down by the grouping variable that matters most (repository).

What the analyst reports

The analyst's summary:

• AI-assisted delivery shows a hypothetical throughput advantage that varies by repository, from negligible to substantial.
• Review burden increases across the board, suggesting reviewers are doing more work per AI-assisted PR.
• Follow-up defects are elevated in repositories with complex business logic (data-pipeline) and less elevated in repositories with simpler, more CRUD-like patterns (internal-tools).
• Self-report audit found a 5% estimated misclassification rate (2/40 sampled PRs), which is unlikely to account for the observed differences but is reported for transparency.
• The comparison cannot establish that AI tooling caused these differences. Self-selection of engineers and tasks is uncontrolled. A randomized rollout within data-pipeline would be needed to test the causal claim.

This last point is the most important sentence in the report, and the one most likely to be cut from the executive slide. Protect it.

Cohort comparison worksheet (copyable)

Copy this worksheet into your own document. Fill in the bracketed fields. The hypothetical examples show the format.

COHORT COMPARISON WORKSHEET
===========================

Study period: [start date] to [end date]
Example: 2025-03-01 to 2025-04-25

Classification method: [describe how authorship is determined]
Example: Required PR dropdown with three options (AI-assisted, Human-only, Prefer not to say)

Total changes in period: [number]
  AI-assisted: [number]  ([percentage]%)
  Human-only:  [number]  ([percentage]%)
  Unknown:     [number]  ([percentage]%)

If Unknown exceeds 30%, STOP. Fix classification before proceeding.

SELF-REPORT AUDIT
-----------------
Sample size:        [number per cohort]
Telemetry cross-ref disagreements: [number]
Interview spot-check disagreements: [number]
Estimated misclassification rate:   [number / total sampled]%

If misclassification rate exceeds 25%, STOP. Redesign reporting mechanism.

MATCHING CRITERIA
-----------------
Dimension 1 - Repository:   [list repositories included]
Dimension 2 - Task type:    [bug fix / feature / refactor / infra]
Dimension 3 - Scope:        [small < 50 LOC / medium 50-250 / large 250+]
Dimension 4 - Time window:  [same 2-4 week sub-period]

Matched pairs produced: [number]
Unmatched AI-assisted changes excluded: [number]
Unmatched human-only changes excluded:  [number]

AGGREGATE RESULTS
-----------------
                          AI-assisted    Human-only    Difference
Throughput (PRs/eng-wk):  [value]        [value]       [+/-  %]
Review rounds per PR:     [value]        [value]       [+/-  %]
Follow-up defects/100 PR: [value]        [value]       [+/-  %]

WITHIN-REPOSITORY RESULTS
-------------------------
Repository: [name]
  Matched pairs: [number]
  Throughput:    AI [value] vs Human [value]
  Review rounds: AI [value] vs Human [value]
  Defect rate:   AI [value] vs Human [value]

(Repeat for each repository with >= 10 matched pairs)

SIMPSON'S PARADOX CHECK
-----------------------
Does any within-repository result reverse the direction of the aggregate?
  [ ] Yes -- report within-repo as primary, note the reversal explicitly
  [ ] No  -- aggregate and within-repo are directionally consistent

UNCONTROLLED CONFOUNDS
----------------------
[ ] Engineer self-selection (who adopts AI tools)
[ ] Task self-selection (which tasks get AI assistance)
[ ] Reporting bias -- estimated misclassification rate: [value]%
[ ] Other: [describe]

CLAIM BOUNDARY
--------------
This comparison can support: [describe observational findings]
This comparison CANNOT support: [describe causal claims it cannot make]
To establish causation, we would need: [describe required design]

Usage instructions: Fill this worksheet once per comparison period. Share it with stakeholders alongside any summary slides. The "Self-Report Audit" section quantifies your confidence in the labels. The "Simpson's Paradox Check" forces you to compare aggregate and within-repository directions before presenting either. The "Claim Boundary" section at the bottom is not optional -- it is the part that prevents your observational data from being cited as proof of causal impact.

Comparing realistic analysis approaches

Recommendation for most teams: Start with the matched cohort approach. If the results are ambiguous or high-stakes decisions depend on them, invest in a phased rollout within one or two repositories. The phased rollout is the only design that can support a causal claim, and even it has practical limitations.

Failure symptoms, corrective actions, and escalation

Explicit limitations

State these in every report. They are not caveats to bury in an appendix; they are load-bearing walls of your analysis.

1. Observational, not experimental. The matched cohort design reduces confounding but does not eliminate it. Any difference you observe could be driven by unmeasured factors (engineer experience, task difficulty beyond what the type label captures, team morale).
2. Self-report classification is imperfect, but now quantified. The audit procedure gives you a misclassification rate estimate. Report it. A low rate does not guarantee accuracy -- it means accuracy is plausible given your sample. A high rate means the comparison's foundation is unstable.
3. Follow-up defect attribution is approximate. A defect discovered 25 days after merge may have been latent in a dependency, not in the change itself. Your 30-day window is a convention, not a physical law.
4. Small samples within repositories. If a repository contributes only 12 matched pairs, the within-repository comparison is unreliable. Report sample sizes alongside every number.
5. Throughput is not value. A team that merges more PRs is not necessarily delivering more customer value. Throughput measures activity, not impact. Connecting delivery metrics to product signals and planning is a separate, harder problem.
6. The comparison ages. AI tools change rapidly. A comparison run in Q1 may not apply to Q3 if the tooling, prompts, or team practices have evolved.

Ongoing review routine

Run this comparison on a regular cadence, not as a one-time study. A quarterly cycle works for most teams.

Every quarter:

• Re-pull the matched cohort data for the latest period.
• Run the self-report audit: 20 PRs per cohort, telemetry cross-reference, 5 spot-check interviews. Record the disagreement rate and compare it to the previous quarter.
• Check the unknown classification rate. If it has crept above 20%, intervene.
• Compare the new period's results to the previous period's. Look for trend changes, not just levels.
• Run the Simpson's paradox check. If the aggregate and within-repository directions have diverged since last quarter, investigate which repository's mix shifted.
• Update the claim boundary section if the design has changed (e.g., you added a new matching dimension or switched from self-report to tool-detected classification).
• Present within-repository results first. Add the aggregate only if someone asks.

Every six months:

• Reassess whether the authorship categories still make sense. If most engineers now use AI tools for most work, the "human-only" cohort may be too small or too unusual to be a useful comparison group.
• Evaluate whether a phased rollout is feasible for the repositories where the observational results are most ambiguous.
• Review whether follow-up defect tracing is consistently practiced. Audit a sample of 10 recent hotfixes to confirm they link back to the originating change.

FeatureFactory can help teams examine engineering delivery and code quality, including AI-assisted work, which supports the data collection side of this routine. The analysis design and claim boundaries remain your responsibility.

Next-action checklist

• Add a required authorship classification field to your PR template this week.
• Choose a study period of four to eight weeks and pull the raw change data.
• Classify changes; report the unknown rate. If above 30%, fix classification first.
• Run the self-report audit on 20 PRs per cohort. Record the disagreement rate.
• Match changes by repository, task type, scope, and time window.
• Compute throughput, review burden, and follow-up defect rate for both cohorts.
• Run within-repository comparisons for every repository with at least 10 matched pairs.
• Perform the Simpson's paradox check: do aggregate and within-repo results agree in direction?
• Fill in the cohort comparison worksheet, including the self-report audit and claim boundary sections.
• Present results to stakeholders with the claim boundary and misclassification rate visible on the same page as the metrics.
• Schedule the next quarterly comparison on the team calendar.
• If results are ambiguous and the decision is high-stakes, propose a phased rollout design for one repository.