The pilot that proved nothing
An engineering director approves a three-month AI coding assistant pilot for forty engineers. The vendor suggests measuring "productivity lift." At the end of the trial, the team reports a mix of anecdotes: some engineers love it, some ignore it, a few worry about code quality. Someone calculates that lines of code per engineer increased 18%. Leadership asks whether to expand. Nobody can answer confidently because nobody defined what decision the pilot was supposed to inform, what quality bar the generated code had to meet, or what would constitute a reason to stop early.
The pilot consumed budget, attention, and goodwill without producing a decision. The problem was not the tool. It was the experiment design.
This guide helps you structure an AI coding pilot as a bounded experiment with explicit costs, guardrails, evaluation windows, and stopping conditions -- so you end with a decision, not a debate.
Quick-start path
If you need to move fast, follow these five steps and fill in the charter worksheet below. Each step is expanded in its own section.
- Name the workflow and the decision. Pick one or two specific workflows (e.g., unit test generation for a particular service, boilerplate CRUD endpoints) and write down the decision the pilot must inform ("Should we purchase annual licenses for these teams?").
- Select participants deliberately. Choose participants to reduce self-selection and novelty bias. Document who opted in and why.
- Budget all costs. Include licenses, token usage, onboarding time, extra review time, and incident investigation overhead.
- Set guardrails before measuring speed. Define quality, security, and operational constraints that generated code must meet. Speed gains that violate a guardrail do not count.
- Write the charter with evaluation windows and stop conditions. Specify when you will look at the data, what evidence you need, and what triggers expand, change, or stop.
Specify the workflow and the decision the pilot must inform
Most failed pilots fail here. "Try AI coding tools and see what happens" is not an experiment -- it is an exploration. Explorations are fine for individuals, but they cannot justify a budget decision.
Pick a concrete workflow, not a category. "Writing code faster" is a category. "Generating initial unit tests for the payments service, where the team currently writes tests manually after implementation" is a workflow. The narrower the scope, the clearer your signal.
Why this matters: different workflows respond differently to AI assistance. Boilerplate generation, test scaffolding, documentation drafting, and complex algorithmic work each have different quality risks and different effort profiles. A pilot that mixes all of them produces an average that describes none of them.
Write the decision sentence. Before you start, complete this sentence: "At the end of this pilot, we will decide whether to ___." Examples:
- "...purchase annual licenses for the backend platform team."
- "...expand AI-assisted test generation to three more services."
- "...change our review process to include AI-generated code checks."
If you cannot fill in the blank, the pilot is not ready. You are spending money without knowing what question it answers.
Adapt for team context. A 200-person engineering org might run parallel pilots on different workflows across multiple teams. A 15-person startup might pilot on a single codebase with the whole team. The principle is the same: one pilot, one workflow, one decision.
Select participants and document bias
Self-selection bias
Engineers who volunteer for an AI coding pilot are typically more enthusiastic about AI tools, more experienced with prompt engineering, or both. Their results will not generalize to reluctant adopters. This does not mean you should force participation -- it means you should record who opted in and factor that into your conclusions.
Novelty effects
New tools produce a burst of engagement. Engineers experiment, explore edge cases, and invest extra effort in making the tool work. This effort inflates early results. Expect the first two weeks of data to be unreliable in either direction -- some engineers will over-invest, others will be frustrated by the learning curve.
Comparable tasks
If you want to compare AI-assisted work to unassisted work, you need tasks that are roughly similar in complexity. Assigning the hardest tickets to the AI group and the easiest to the control group (or vice versa) will contaminate your signal. You do not need perfect randomization, but you do need to document the assignment logic so reviewers can judge whether the comparison is fair.
Practical approach for small teams: If you have fewer than ten engineers, a controlled comparison is unlikely to produce meaningful signal. Instead, use a within-person design: each participant works with and without the tool on similar tasks across different weeks. Record both conditions per person.
Practical approach for larger teams: Select two comparable sub-teams working on similar services. One uses the tool; the other does not. Document the teams' experience levels, the services' complexity, and recent velocity patterns before the pilot starts.
What to record about participants
For each participant, document: years of experience, familiarity with AI coding tools (none / casual / regular), primary language and framework, and whether they volunteered or were assigned. This is not for statistical analysis -- it is for honest interpretation when results arrive.
Budget all costs, not just licenses
License fees are the visible cost. The invisible costs are often larger.
Cost categories
| Category | What to estimate | Why it is easy to miss |
|---|---|---|
| Licenses | Per-seat cost multiplied by participants multiplied by pilot duration | Straightforward, but watch for per-token overage charges |
| Token / API usage | Estimated monthly token spend per engineer for the chosen workflow | Varies wildly by workflow; test generation consumes more tokens than autocomplete |
| Onboarding | Hours per engineer for setup, training, prompt practice | Often untracked; engineers absorb it into their sprint time |
| Incremental review time | Extra minutes per PR to review AI-generated code | Reviewers may not flag this unless you ask; measure it explicitly |
| Incident investigation | Hours spent investigating issues caused by or related to generated code | Rare per incident, expensive per hour; budget a contingency |
| Pilot administration | Time spent collecting data, running retrospectives, writing the final report | Someone has to do this; if nobody is assigned, it will not happen |
Hypothetical budget example
Suppose (hypothetically) you are piloting AI-assisted test generation with eight engineers for six weeks:
- Licenses: 8 seats x $19/month x 1.5 months = $228
- Token overage estimate: $50/month x 1.5 months = $75
- Onboarding: 8 engineers x 3 hours x $75/hour (fully loaded) = $1,800
- Incremental review time: 2 extra hours/week x 8 engineers x 6 weeks x $75/hour = $7,200
- Incident investigation contingency: 10 hours x $75/hour = $750
- Pilot administration: 4 hours/week x 6 weeks x $85/hour = $2,040
Total hypothetical cost: approximately $12,093
In this example, the license cost ($228) is less than 2% of the total. The largest line item is incremental review time. If you budget only for licenses, you will undercount by an order of magnitude.
Teams using FeatureFactory to examine engineering delivery and code quality, including AI-assisted work, can look at review-time patterns during the pilot period. But no tool eliminates the need to budget upfront for the human time.
Define guardrails before measuring speed
A pilot that shows engineers ship 30% faster but introduces security vulnerabilities is not a success. Define your guardrails first; then measure speed within those constraints.
Quality guardrails
- Test coverage: AI-generated code must meet the same coverage requirements as human-written code. If your bar is 80% line coverage on new code, that bar does not change.
- Review approval: Every AI-generated change goes through the same review process. No fast-tracking because "the AI wrote it."
- Linting and static analysis: Generated code must pass existing linters, formatters, and static analysis rules with no new suppressions.
Security guardrails
- No secrets in prompts: Engineers must not paste API keys, customer data, or internal credentials into AI tool prompts. Document this rule explicitly; it is easy to violate accidentally.
- Dependency policy: If the AI suggests adding a new dependency, normal dependency review applies.
- Data classification: Code that handles PII, financial data, or other sensitive categories may need additional scrutiny. Decide before the pilot whether these areas are in scope.
Operational guardrails
- Incident attribution: If an incident traces back to AI-generated code during the pilot, record it. You do not need to treat it differently from a human-caused incident, but you need to track provenance.
- Rollback readiness: Ensure that any AI-generated change can be reverted with the same process as any other change.
Experience measures
Guardrails tell you whether the floor held. You also need to know what engineers actually experienced:
- Perceived helpfulness per task: After each AI-assisted task, the engineer records a simple rating (helped / neutral / hindered) and a one-sentence note.
- Time-on-task for comparable work: If using a within-person design, engineers record time for similar tasks with and without the tool. These are self-reported estimates, not precise measurements. Treat them accordingly.
- Proportion of AI-generated code kept vs. discarded: If engineers frequently discard generated output, the tool is not saving time -- it is adding a review step.
Pilot charter worksheet (copyable)
Copy this charter template and fill in each field before starting your pilot. The filled example below is entirely hypothetical.
Instructions: Fill in every field. If you cannot fill a field, the pilot is not ready to start. Review the charter with your pilot participants and your manager before day one. Revisit it at each evaluation window.
=== AI CODING PILOT CHARTER ===
Pilot name: [AI-Assisted Test Generation - Payments Service]
Sponsor: [VP Engineering, J. Morales]
Pilot lead: [Senior Engineer, R. Chen]
Start date: [2025-02-03]
End date: [2025-03-14]
--- WORKFLOW ---
Specific workflow being tested:
[Generating initial unit test files for new endpoints in the
payments service. Engineers use the AI tool to draft tests,
then review, modify, and finalize before merging.]
Decision this pilot must inform:
[Should we purchase annual licenses for the backend platform
team (12 engineers) for AI-assisted test generation?]
--- PARTICIPANTS ---
Number of participants: [8]
Selection method: [Volunteers from the payments team, plus two
engineers from the orders team for cross-team perspective]
Self-selection noted: [Yes -- all participants volunteered.
Results may not generalize to reluctant adopters.]
Control comparison: [Within-person. Each participant alternates
weeks: AI-assisted (weeks 1, 3, 5) and manual (weeks 2, 4, 6).]
--- BUDGET ---
Total estimated cost: [$12,093]
License cost: [$228]
Non-license cost: [$11,865]
Budget owner who approved: [J. Morales, approved 2025-01-20]
--- GUARDRAILS ---
Quality: [Same 80% line coverage bar. Same review process.
All existing linters and static analysis enforced.]
Security: [No secrets in prompts. No PII in prompts.
Dependency additions follow normal review.]
Operational: [Incident provenance tracked. Standard rollback.]
--- MEASURES ---
Primary: [Self-reported time-on-task for test writing, compared
across AI-assisted and manual weeks per engineer.]
Secondary: [Proportion of generated tests kept vs. rewritten.
Perceived helpfulness rating per task. Review comments
related to AI-generated code quality.]
Not measuring: [Lines of code. Commit count. "Productivity."]
--- EVALUATION WINDOWS ---
Window 1 (Week 2): [Check onboarding completion. Are all
participants using the tool? Any setup blockers?
Decision: Continue / pause to fix onboarding.]
Window 2 (Week 4): [Review time-on-task data and helpfulness
ratings. Check guardrail compliance (any violations?).
Decision: Continue / adjust workflow scope / stop early.]
Window 3 (Week 6): [Final data review. Compile report.
Decision: Expand / change scope / stop.]
--- STOPPING CONDITIONS ---
Stop immediately if:
- A security guardrail is violated (secret or PII in prompt)
- An incident is attributed to AI-generated code AND the root
cause is a pattern the tool systematically produces
- Fewer than 4 participants are still actively using the tool
(insufficient signal)
Stop at next evaluation window if:
- More than 50% of generated tests are fully rewritten before
merge (tool is adding work, not reducing it)
- Average helpfulness rating is "hindered" for two consecutive
weeks
--- EXPAND / CHANGE / STOP CRITERIA ---
Expand to annual licenses if:
- At least 6 of 8 participants report net time savings
- Guardrails held with zero security violations
- Review quality did not degrade (no increase in post-merge
defects attributed to test gaps)
Change scope if:
- Tool helps with test scaffolding but not assertion logic
(narrow the use case before expanding)
Stop if:
- Stopping conditions triggered
- Cost exceeds budget by more than 20%
- Decision-maker leaves the org (no sponsor, no pilot)
Worked hypothetical scenario: from charter to decision
Setup (hypothetical). R. Chen's team starts the pilot on February 3. Eight engineers begin alternating AI-assisted and manual weeks for test generation on the payments service.
Week 2 check-in. Seven of eight engineers completed onboarding. One engineer had a configuration issue that consumed a full day; the pilot lead resolves it. All seven report using the tool for at least three test files. No guardrail violations. Decision: continue.
Week 4 check-in. Time-on-task data shows a pattern: for straightforward CRUD endpoint tests, engineers report spending roughly 40% less time with the tool (hypothetical). For tests involving complex payment state machines, engineers report spending about the same time or slightly more, because the generated tests miss edge cases and need heavy rewriting. Helpfulness ratings: 5 engineers rate it "helped" for simple tests; 6 rate it "neutral" or "hindered" for complex tests. One security near-miss: an engineer almost pasted a test API key into a prompt but caught it before submitting. The team adds a pre-prompt checklist reminder. Decision: continue, but narrow the recommended use case to CRUD and simple validation tests.
Week 6 final review. Compiled data shows:
- Proportion of generated tests kept without major rewrite: 72% for simple tests, 31% for complex tests.
- Self-reported time savings (simple tests only): median 35% reduction (hypothetical, self-reported, small sample).
- Guardrail compliance: zero security violations after the week-3 near-miss. No incidents attributed to AI-generated code.
- Review load: reviewers report AI-generated tests are "mostly fine but occasionally miss null-input cases," requiring one or two extra comments per PR.
Decision. The team recommends purchasing annual licenses for the backend platform team, scoped to simple test generation. They do not claim a productivity percentage for leadership. Instead, they present the charter, the data, and the narrowed scope. The sponsor approves. The team plans a follow-up pilot for more complex test scenarios with tighter prompt templates.
This outcome is useful precisely because it did not promise "35% productivity gain." It said: "For this workflow, under these conditions, with these guardrails, the tool helped. Here is what it did not help with."
Comparing realistic options for pilot structure
| Option | Strength | Weakness | Best fit |
|---|---|---|---|
| Within-person alternating weeks | Controls for individual skill differences; works with small teams | Carryover effects (skills learned in AI weeks carry into manual weeks) | Teams under 15 engineers |
| Parallel teams | Cleaner comparison if teams are well-matched | Hard to match teams on skill, codebase, and task complexity; requires enough people for two groups | Organizations with multiple teams on similar services |
| Before/after on same team | Simple to run; no control group needed | Cannot distinguish tool effect from other changes (new hire, seasonal workload shift, codebase improvements) | When a controlled comparison is impractical and you acknowledge the ambiguity |
| Opt-in free-form exploration | Low overhead; fast to start | Produces anecdotes, not evidence; self-selection dominates; no stopping conditions | Individual learning, not budget decisions |
No option is perfect. Pick the one whose weaknesses you can document honestly. The charter worksheet has a field for this.
If your team uses FeatureFactory for product planning, you can track the pilot as a time-boxed initiative with explicit success criteria, keeping it visible alongside other planning work.
Failure symptoms, corrective actions, and escalation points
| Symptom | Likely cause | Corrective action | Escalate if... |
|---|---|---|---|
| Most participants stop using the tool by week 3 | Poor onboarding, tool does not fit workflow, or workflow was too broad | Interview dropouts; narrow the workflow; extend onboarding | Fewer than half of participants remain active -- consider stopping |
| Time-on-task data is missing or inconsistent | Engineers forget to log; no reminder system | Add a lightweight end-of-day prompt; simplify the logging format | Data is too sparse to support any conclusion at the week-4 window |
| Reviewers report significant quality issues in generated code | Tool output does not meet your codebase standards; prompts are under-specified | Tighten prompt templates; pair reviewers with authors to document patterns | Quality issues appear in production or a guardrail is violated |
| The sponsor asks for a single productivity number mid-pilot | Pressure to justify the spend with a headline metric | Share the charter; explain that a single number without context will mislead | Sponsor insists on a number you cannot support -- this is a stop condition for the pilot's credibility |
| License costs exceed budget due to token overages | Underestimated per-engineer usage; some engineers experimenting heavily | Set per-user token limits; review usage patterns | Overage exceeds 20% of total budget |
| Engineers paste sensitive data into prompts | Habit, convenience, unclear policy | Immediate retraining; add a pre-prompt checklist; consider tool-level data loss prevention if available | Any confirmed exposure of secrets, PII, or customer data -- escalate to security team immediately |
Explicit limitations
- Self-reported time data is unreliable. Engineers estimate poorly, especially when they know they are being observed. Use self-reports as directional signal, not precise measurement.
- Small samples do not generalize. Eight engineers on one service for six weeks cannot tell you what will happen across a 200-person org. They can tell you whether the next experiment is worth running.
- You cannot isolate the tool's effect cleanly. Engineers learn, codebases change, and priorities shift during any pilot. Your conclusion will always be "the tool probably helped with X under these conditions," not "the tool caused Y."
- AI tools change rapidly. A model update mid-pilot can alter output quality. Pin the model version if the tool allows it; otherwise, note version changes in your log.
- This guide does not evaluate specific AI coding tools. It provides experiment structure. Vendor selection, model comparison, and feature evaluation are separate decisions.
- Novelty wears off. A six-week pilot captures initial adoption. Sustained value (or sustained friction) may only emerge over months. Plan a follow-up review.
Ongoing review routine
After the pilot concludes and you have made your expand/change/stop decision, the work is not done. AI coding tools evolve, teams change, and the workflow you tested may shift.
Monthly (first quarter after expansion):
- Review guardrail compliance. Any security violations? Any incidents with AI-generated code provenance?
- Check adoption breadth. Are the engineers who received licenses actually using the tool, or did usage drop after the pilot spotlight faded?
- Sample three to five PRs containing AI-generated code. Read the review comments. Are reviewers catching patterns that the tool consistently gets wrong?
Quarterly (ongoing):
- Reassess the scoped workflow. Has the team expanded into workflows not covered by the original pilot? If so, those need their own evaluation, not an assumption that results transfer.
- Review costs against budget. Token usage patterns stabilize after the first month; adjust budget forecasts.
- Decide whether to run a follow-up pilot for a new workflow or a new team.
Annually:
- Revisit the tool choice. The market changes. A tool that was best-fit twelve months ago may not be now.
- Review whether the guardrails need updating based on incidents, near-misses, or changes in your security posture.
Examining delivery patterns over time through tools like FeatureFactory can surface trends in review cadence or code quality that correlate with AI-assisted work periods, but always interpret those patterns with the limitations above in mind.
Next-action checklist
Use this after reading the guide to move from "we should try an AI coding tool" to a structured pilot.
- Identify one specific workflow where AI assistance might help, and write it in a single sentence.
- Write the decision sentence: "At the end of this pilot, we will decide whether to ___."
- Name the pilot sponsor (budget authority) and the pilot lead (day-to-day operator).
- Select participants and document the selection method and any self-selection.
- Estimate the full cost using all six categories in the budget table.
- Get budget approval in writing from the sponsor.
- Define quality, security, and operational guardrails. Write them down; do not assume shared understanding.
- Choose your pilot structure (within-person, parallel teams, before/after) and document its known weaknesses.
- Fill in the pilot charter worksheet completely.
- Review the charter with participants and the sponsor before day one.
- Set calendar reminders for each evaluation window.
- Run the pilot. Collect data. Hold evaluation windows on schedule.
- At the final window, make the expand/change/stop decision based on the criteria you wrote before the pilot started.
- Publish the decision and the reasoning to your engineering organization, including what the pilot did not prove.
The last item matters more than it seems. Teams that see a transparent, well-structured pilot -- even one that ends in "stop" -- are more likely to trust the next experiment. Teams that see a vague pilot followed by a confident claim learn to discount both.
Measure what you ship - for real.
Put your team’s AI web-development on the outcome loop.
Get early access