Skip to main content
FeatureFactory
← Blog
metrics measurement15 min read·FeatureFactory

Build a change-failure metric when incidents and deployments do not line up

A field guide for teams whose deployment records and incident logs do not cleanly connect. Build a change-failure rate with explicit attribution tiers, confidence notes, and an instrumentation backlog -- instead of silently treating unknown links as successes.

The metric that quietly declared everything fine

A platform team ships a quarterly reliability report. Change failure rate: 4.2%. Leadership is satisfied. But when an engineer traces the number back to its source, the picture falls apart. The team counted deployments from their CI/CD pipeline and incidents from their on-call tool. To calculate the rate, someone matched incidents to deployments by timestamp proximity -- any incident within two hours of a deploy was "caused by" that deploy. Incidents that did not fall in a two-hour window were excluded. Deployments with no nearby incident were counted as successes.

The result: 23 incidents were silently dropped because they appeared more than two hours after a deploy, even though five of them were later confirmed as deployment-related. Another eight incidents involved changes from multiple deployments, so the analyst picked the closest one. And 14 deployments had no incident data at all -- not because they succeeded, but because the service they changed did not have alerting configured yet. Every gap in the data quietly became a success.

This guide is for teams in that situation. Your deployment records and your incident records do not line up neatly. You need a change-failure metric anyway, but you need it to be honest about what it knows and what it does not.

Quick-start path

If you need a working metric this week, follow these five steps. Each is expanded in the sections below.

  1. Write down your inclusion boundary. Define what counts as a deployment and what counts as a failure. Do this before you query anything.
  2. Link incidents to deployments in three tiers. Confirmed (root cause identified), plausible (timing and scope suggest a link), unknown (cannot determine).
  3. Calculate three rates, not one. A confirmed-only rate, a confirmed-plus-plausible rate, and a worst-case rate that treats unknowns as failures.
  4. Attach a confidence note. State the percentage of incidents with unknown attribution and the percentage of deployments with no incident data coverage.
  5. Open an instrumentation backlog. Each unknown attribution is a gap you can close. Track the gaps as work items.

Define deployment, failure, remediation, and the inclusion boundary before counting

Why definitions come first

Change failure rate is a ratio. The denominator is deployments. The numerator is deployments that caused failures. If you do not define both terms precisely, people will argue about the number instead of acting on it.

Deployment

A deployment is a change released to a production environment that is reachable by end users or downstream systems. Decide these boundary questions explicitly:

  • Config changes. If a feature flag flip or environment variable change can cause an outage, include it. If you exclude it, document why.
  • Rollbacks. A rollback is itself a deployment. If a rollback causes a new incident, it counts.
  • Canary or progressive rollouts. Count the moment the change first reaches production traffic, not the moment it reaches 100%.
  • Infrastructure changes. Terraform applies, database migrations, and network rule changes are deployments if they can affect production behavior.

Failure

A failure is a user-facing or system-facing degradation that triggers remediation. Define what qualifies:

  • Severity threshold. You might include only incidents at SEV-2 or above, or you might include anything that required a human response. State the threshold.
  • Partial degradation. A 200ms latency increase that violates an SLO is a failure even if no pages fired.
  • Delayed symptoms. A memory leak introduced by a deploy may not surface for 48 hours. Your attribution window must account for this.

Remediation

Remediation is the action taken to resolve the failure. Track it because it tells you whether the response was a rollback (suggesting the deploy was the cause) or a config change elsewhere (suggesting the deploy was not the cause). Remediation type is one of your strongest attribution signals.

The inclusion boundary

Write a single paragraph that states: which environments, which services, which time range, which severity levels, and which deployment types are in scope. Post it alongside every report. Here is a hypothetical example:

Inclusion boundary (hypothetical): All production deployments to the payments service and the checkout service between January 1 and March 31, including application code, database migrations, and feature flag changes. Incidents at SEV-3 and above. Staging and development deployments excluded. Scheduled maintenance windows excluded.

Why this matters across team contexts

A team with a single monolith and one deploy pipeline has a simple boundary. A platform team supporting 15 microservices may need per-service boundaries. A team using trunk-based development with continuous deployment may have hundreds of deployments per week, making the boundary question about granularity -- do you count every merge-to-main, or every pipeline run that reaches production? State your answer before querying.

Handle incidents linked to multiple deployments and deployments with delayed symptoms

The many-to-many problem

In practice, one incident can involve changes from multiple deployments, and one deployment can contribute to multiple incidents. Forcing a one-to-one link distorts the metric.

One incident, multiple deployments. A checkout failure occurs after three services were deployed within an hour. The root cause involves an API contract change in Service A and a timeout configuration in Service B. Both deployments contributed.

One deployment, multiple incidents. A single deploy introduces a bug that causes a payment processing error (SEV-2) and a reporting dashboard blank page (SEV-3). These are separate incidents with separate user impact.

Attribution tiers

Instead of forcing a binary link, record each incident-deployment pair with one of three tiers:

Tier Definition Evidence required Example
Confirmed Root-cause analysis identifies this deployment as a cause Post-incident review, rollback resolved the issue, code diff matches the failure mode Deploy #412 introduced a null-pointer exception; rollback resolved the 500 errors
Plausible Timing, scope, and circumstantial evidence suggest a link, but root cause is not proven Deploy touched the affected service within the attribution window; no contradicting evidence Deploy #415 changed the same module 90 minutes before latency increased, but no RCA completed
Unknown Cannot determine whether this deployment caused this incident No matching deploy found, or incident data is insufficient to attribute Alert fired on a service with no recent deploys logged; deploy records for that service are incomplete

The attribution window

Set an explicit window -- for example, 72 hours after a deployment. Incidents that surface within that window and affect the deployed service are candidates for plausible or confirmed attribution. Incidents outside the window require stronger evidence (such as a post-incident review) to be linked.

The right window depends on your system. A stateless API might surface failures within minutes. A batch processing system might not run the affected code path for 24 hours. A memory leak might take days.

Recording the links

For each incident, record:

  • Incident ID
  • Linked deployment ID(s), or "none found"
  • Attribution tier for each link
  • Evidence summary (one sentence)
  • Reviewer who assigned the tier

This is not a heavy process. It adds 2-3 minutes per incident during or after triage. The payoff is that your metric stops lying.

Attribution worksheet (copyable)

Use this worksheet to record attribution for each incident in your reporting period. Copy it into a spreadsheet, wiki page, or incident management tool.

Instructions: Fill one row per incident-deployment pair. If an incident has no candidate deployment, fill a single row with "None found" in the Deployment ID column and "Unknown" as the tier. After completing all rows, use the summary section to calculate rates.

=== CHANGE-FAILURE ATTRIBUTION WORKSHEET ===

Inclusion boundary:
  Environment(s): [e.g., production]
  Service(s): [e.g., payments, checkout]
  Date range: [e.g., 2025-Q1]
  Deployment types included: [e.g., app code, migrations, flag changes]
  Incident severity threshold: [e.g., SEV-3 and above]
  Attribution window: [e.g., 72 hours post-deploy]

--- INCIDENT-DEPLOYMENT PAIRS ---

Row | Incident ID | Deployment ID | Tier        | Evidence (1 sentence)              | Reviewer
----|-------------|---------------|-------------|------------------------------------|----------
 1  | INC-301     | DEP-412       | Confirmed   | Rollback of DEP-412 resolved 500s  | J. Lee
 2  | INC-302     | DEP-415       | Plausible   | Same module changed 90min prior    | M. Chen
 3  | INC-302     | DEP-416       | Plausible   | Timeout config may have interacted | M. Chen
 4  | INC-305     | None found    | Unknown     | No deploy records for alert-svc    | S. Patel
 5  | INC-308     | DEP-421       | Confirmed   | Code diff matches stack trace      | J. Lee

--- SUMMARY ---

Total deployments in scope:            [count]
Total incidents in scope:              [count]

Deployments linked at Confirmed tier:  [count]
Deployments linked at Plausible tier:  [count]  (not also Confirmed)
Incidents with Unknown attribution:    [count]
Deployments with no incident coverage: [count]  (service lacked alerting)

--- RATES ---

Confirmed-only rate:       Confirmed deployments / Total deployments
Confirmed+Plausible rate:  (Confirmed + Plausible deployments) / Total deployments
Worst-case rate:           (Confirmed + Plausible + Unknown incidents treated as
                            distinct deployment failures) / Total deployments

--- CONFIDENCE NOTE ---

[X]% of incidents have Unknown attribution.
[Y]% of deployments are in services with no alerting or incomplete incident records.
This metric is most reliable for: [list services with full coverage].

Hypothetical worked scenario: one quarter, three attribution tiers

The following scenario uses entirely hypothetical numbers to illustrate how the worksheet produces different rates depending on attribution quality.

Setup. A hypothetical team runs two services: payments and notifications. In Q1, they record 80 deployments across both services and 9 incidents at SEV-3 or above.

Linking incidents to deployments.

Incident Service Deployment(s) Tier Evidence
INC-301 payments DEP-412 Confirmed Rollback fixed it
INC-302 payments DEP-415, DEP-416 Plausible Timing match, no RCA yet
INC-303 payments DEP-420 Confirmed Stack trace matches diff
INC-304 notifications DEP-430 Plausible Deploy 40min before alert
INC-305 notifications None found Unknown No deploy records for that day
INC-306 notifications None found Unknown Alert-svc lacks deploy tracking
INC-307 payments DEP-445 Confirmed Post-incident review completed
INC-308 payments DEP-448 Plausible Same endpoint, no RCA
INC-309 notifications DEP-455 Confirmed Memory leak traced to this deploy

Counting unique deployments that caused failures.

  • Confirmed: DEP-412, DEP-420, DEP-445, DEP-455 = 4 deployments
  • Plausible (not already confirmed): DEP-415, DEP-416, DEP-430, DEP-448 = 4 deployments
  • Unknown (no deployment linked): INC-305, INC-306 = 2 incidents with no deployment candidate

Calculating three rates (hypothetical).

Rate Numerator Denominator Result
Confirmed-only 4 80 5.0%
Confirmed + Plausible 8 80 10.0%
Worst-case (unknowns as failures) 10 80 12.5%

Confidence note: 2 of 9 incidents (22%) have unknown attribution. The notifications service has incomplete deploy tracking, covering roughly 60% of its actual changes. The confirmed-only rate is a floor, not a ceiling.

Interpretation. If the team reports only the 5.0% confirmed rate, they understate the problem. If they report only the 12.5% worst-case rate, they overstate it. Reporting all three, with the confidence note, gives the reader the information they need to make their own judgment. The gap between 5.0% and 12.5% is itself a signal: it says "we have an attribution problem, not just a reliability problem."

Comparing realistic options for attribution method

Approach How it works Strength Weakness Best fit
Timestamp proximity Link any incident within N minutes of a deploy Simple, automatable High false-positive rate; misses delayed failures; treats coincidence as causation Teams just starting, as a temporary step
Three-tier manual attribution (this guide) Human reviewer assigns Confirmed, Plausible, or Unknown per pair Transparent uncertainty; catches delayed and multi-deploy causes Requires 2-3 minutes per incident; depends on reviewer judgment Teams with 5-30 incidents/quarter who want honest numbers
Deploy-tag correlation Each deploy tags affected services; incidents auto-link to recent deploys of those services More precise than timestamp alone Requires consistent tagging discipline; still needs human review for confirmation Teams with good CI/CD metadata but incomplete RCA coverage
Full post-incident attribution Every incident gets a root-cause review that identifies the causal deploy Highest accuracy Expensive; RCA backlog grows; only practical for high-severity incidents Teams that already do thorough post-incident reviews for all severities in scope

No approach eliminates uncertainty. The question is whether your metric acknowledges it or hides it. Teams that examine their engineering delivery signals alongside attribution tiers can spot patterns -- such as services where plausible links cluster -- that point to instrumentation gaps rather than reliability gaps.

Failure symptoms, corrective actions, and escalation points

Symptom Likely cause Corrective action Escalate when
Unknown tier dominates (>40% of incidents) Deploy records are incomplete for some services Add those services to the instrumentation backlog; prioritize deploy event emission Unknown tier stays above 40% for two consecutive quarters
Confirmed-only rate drops to 0% while incidents continue RCA process is not completing; incidents close without attribution Require attribution tier assignment before incident closure RCA completion rate falls below 70% for two sprints
Gap between confirmed and worst-case rates keeps widening Plausible links are never resolved; no one revisits them Schedule a monthly 30-minute attribution review to upgrade or downgrade plausible links Gap exceeds 3x the confirmed rate for two quarters
Metric is reported but never discussed Leadership does not know how to act on three rates Add a one-sentence interpretation to the report: "The range means X" Two consecutive reports with no follow-up action from any stakeholder
A team games the metric by splitting deploys Smaller deploys reduce the chance any single deploy is linked to an incident Monitor deploy size alongside failure rate; very small deploys may indicate batching avoidance Deploy count increases >50% with no change in feature throughput
Delayed failures are systematically missed Attribution window is too short for some service types Extend the window for batch or background services; document per-service windows The same service appears in unknown attribution repeatedly

Explicit limitations

  • This metric does not measure reliability. Change failure rate measures how often changes cause failures, not how reliable the system is overall. Infrastructure outages, dependency failures, and capacity issues are not captured unless they trace to a deployment.
  • Human attribution is subjective. Two reviewers may assign different tiers to the same incident-deployment pair. Calibration sessions help but do not eliminate disagreement.
  • Small sample sizes distort rates. A team with 10 deployments per quarter will see its rate swing by 10 percentage points with a single incident. Report absolute counts alongside percentages for teams below roughly 30 deployments per period.
  • The metric does not prescribe action. A 15% confirmed rate could mean "deploy smaller batches," "improve test coverage," "add canary analysis," or "fix the one flaky migration script." The rate identifies where to look, not what to do.
  • Incomplete records cannot be fully compensated for. The three-tier approach makes uncertainty visible, but it does not fill the gaps. You still need to invest in instrumentation.
  • This guide does not cover automated anomaly detection. If your system auto-detects deployment-correlated anomalies, the output feeds into the plausible tier, but the detection system itself is out of scope.

Instrumentation backlog and ongoing review routine

The instrumentation backlog

Every "unknown" attribution entry is an instrumentation debt item. Track them:

=== INSTRUMENTATION BACKLOG ===

Item | Service        | Gap                              | Priority | Owner    | Status
-----|----------------|----------------------------------|----------|----------|--------
 1   | notifications  | No deploy events emitted to log  | High     | D. Kim   | In progress
 2   | alert-svc      | Alerting covers only HTTP errors | Medium   | R. Gupta | Not started
 3   | batch-jobs     | Attribution window too short (2h) | Medium   | J. Lee   | Not started

Each item closed reduces the unknown tier in future reports. This is the mechanism by which the metric improves itself over time.

Ongoing review routine

Weekly (5 minutes during existing standup or triage): Any new incidents from the past week get a preliminary tier assignment. This prevents a backlog of unattributed incidents.

Monthly (30 minutes, dedicated): Review all plausible-tier links. Upgrade to confirmed if RCA is complete. Downgrade to unknown if evidence weakened. Close instrumentation backlog items that are done.

Quarterly (60 minutes, with stakeholders): Recalculate all three rates. Compare to prior quarter. Present the confidence note. Decide whether the inclusion boundary needs adjustment (new services, changed severity thresholds). Identify the top instrumentation gap and assign an owner.

Teams using product planning tools can integrate instrumentation backlog items into their regular prioritization so these gaps compete for attention alongside feature work rather than languishing in a separate tracker.

Next-action checklist

Use this checklist to get from "we should track change failure rate" to "we have a reportable metric with honest confidence bounds."

  • Draft the inclusion boundary. List environments, services, date range, deployment types, severity threshold, and attribution window. Get sign-off from one engineering lead and one on-call representative.
  • Audit deployment records. For each in-scope service, confirm that deploy events are logged with timestamp, service name, and change identifier. Flag services with gaps.
  • Audit incident records. For each in-scope service, confirm that incidents include severity, affected service, start time, and remediation action. Flag services with gaps.
  • Link incidents to deployments for one month. Use the attribution worksheet. Assign tiers. This is your calibration round -- expect disagreements and use them to refine your tier definitions.
  • Calculate the three rates and write the confidence note. Share internally before presenting to stakeholders.
  • Open the instrumentation backlog. Convert every unknown-tier entry into a backlog item with an owner.
  • Schedule the ongoing review routine. Add the weekly, monthly, and quarterly cadences to your team calendar.
  • Present the first report with all three rates. Explain the gap between confirmed and worst-case. Frame the gap as a signal, not a flaw.
  • After two quarters, revisit the inclusion boundary. Adjust for new services, changed thresholds, or closed instrumentation gaps. Compare the unknown-tier percentage across quarters to see if your data quality is improving.

The distinctive challenge here is not calculating a ratio -- it is refusing to let missing data silently become good news. Every gap in your records, if left unacknowledged, turns into a zero in the numerator. The three-tier approach forces those gaps into the open, where they can be measured, prioritized, and eventually closed.

Measure what you ship - for real.

Put your team’s AI web-development on the outcome loop.

Get early access

Related reading