The dashboard that celebrated doing nothing new
A backend team migrates from a monolith to four independent services. Their deployment frequency jumps from eight deploys per week to thirty-two. Leadership applauds the improvement. But when a product manager asks what shipped last week, the answer is the same as before: two small features and a bug fix. The remaining deploys were automated config propagations, a rollback that triggered a redeploy, a certificate rotation that cycled every pod, and identical artifact promotions across staging environments that the pipeline counted as production events.
The metric went up. Delivery capability did not change. The team spent the next quarter defending a number that measured pipeline activity, not the ability to get useful changes into users' hands.
This guide helps you define what counts as a deployment, exclude events that inflate the number, pair frequency with signals that expose whether those deployments matter, and maintain the definition as your architecture evolves.
Quick-start path
If you need a working definition this week, do these five things in order:
- List every production service and the pipeline that deploys it. Include services you own but rarely change.
- Write a one-sentence deployment event definition -- specifically, what artifact change in what environment counts as one deployment. Post it where the team can see it.
- Exclude rollbacks, config-only changes, and redeploys of the same artifact. Tag them separately; do not delete them from your data.
- Pair the frequency number with at least one quality signal -- change failure rate or the count of user-facing changes -- every time you report it.
- Review the definition quarterly, or whenever a new service, pipeline, or deployment pattern is introduced.
Each step has a corresponding section below with the reasoning and mechanics.
Decide which services and events are in scope
Why this matters
Deployment frequency is only comparable over time if the boundary of what you are counting stays stable. Adding a new service to the count without adjusting the baseline makes last quarter's number look worse by comparison -- even if nothing changed about your delivery speed.
How to do it
Create a service registry entry for every independently deployable unit that your team owns. For each, record:
- Service name and repository
- Deployment target (production cluster, edge function, mobile app store -- each has different semantics)
- Pipeline identifier (the CI/CD job or workflow that performs the deploy)
- Whether the service is in scope for your deployment frequency metric
A service is in scope if:
- It runs in production (not just staging or preview).
- Your team controls when it deploys (not a shared platform service deployed by another team on their schedule).
- It serves end users or other production services (not a one-off migration job).
If a service is borderline -- say, an internal admin tool that technically runs in production -- make a decision and write it down. The specific choice matters less than consistency.
Adapting to team context
- Monolith teams: You likely have one deployable unit. Your scope decision is simple, but watch for the moment a second service appears -- that is when the count can silently double without reflecting new capability.
- Microservice teams (5-15 services): List every service explicitly. Do not rely on "everything in the Kubernetes namespace" because shared infrastructure (sidecars, proxies) will inflate the count.
- Platform teams deploying shared libraries or SDKs: A library release consumed by ten services may trigger ten downstream deploys. Count the library release as one event, not ten, unless you specifically want to measure consumer adoption speed (a different question).
Distinguish deployment types and tag them
Not every production event is a meaningful deployment. The following classification keeps your count honest.
Deployment type definitions
| Type | Definition | Counts toward frequency? | Why |
|---|---|---|---|
| New-code deploy | A new build artifact containing at least one source-code change is promoted to production. | Yes | This is the event you care about -- new work reaching users. |
| Config-only change | Environment variables, feature flag values, or infrastructure parameters change; no application code changes. | No (track separately) | Config changes are valuable but measure operational agility, not delivery frequency. |
| Rollback | A previous artifact is redeployed to replace a failing release. | No (track separately) | A rollback undoes a deploy; counting it inflates frequency and rewards failure. |
| Redeploy of same artifact | The identical artifact is deployed again (pod restart, node migration, scaling event). | No (exclude entirely) | No change occurred. This is infrastructure maintenance. |
| Canary promotion | A canary release is promoted from a subset of traffic to full traffic. | No (it is the completion of the original deploy, not a new one) | Counting both canary start and promotion doubles every deploy. |
| Independent service release | A new-code deploy to a service that your team owns and independently releases. | Yes (one event per service) | Each service's release reflects a separate delivery decision. |
How to implement tagging
Most CI/CD systems let you add metadata to a deployment record. At minimum, attach:
deploy_type: one of the types aboveartifact_sha: the build artifact hashprevious_artifact_sha: the artifact it replacedhas_code_change: boolean, derived from comparing the two SHAs or from the commit list
If artifact_sha equals previous_artifact_sha, the event is a redeploy and should be excluded automatically. If the commit list between the two SHAs contains only changes to config files (defined by path pattern), tag it as config-only.
When artifact-SHA deduplication breaks down
Artifact-SHA comparison assumes that the same source code always produces the same artifact hash. This is true for reproducible builds, but many build pipelines are not fully reproducible. Timestamps embedded in binaries, non-deterministic dependency resolution order, or build-environment differences can produce a different SHA from identical source commits. When that happens, a pod restart or node migration generates a new SHA, and your pipeline counts it as a new-code deploy even though no source changed.
Two practical defenses:
- Compare source commit ranges, not just artifact SHAs. If the list of commits between the previous deploy's HEAD and the current deploy's HEAD is empty, treat the event as a redeploy regardless of artifact SHA.
- Track the source commit SHA alongside the artifact SHA. Even if your build is not reproducible at the artifact level, the Git commit is deterministic. Two deploys with the same source commit SHA are the same code, regardless of artifact differences.
If your team relies solely on artifact SHA matching, periodically spot-check by pulling the commit lists for a sample of deploys. If you find deploys tagged as new-code where the commit range is empty, your build reproducibility is insufficient for SHA-based deduplication and you should switch to source-commit comparison.
This tagging is the foundation for every downstream analysis. Without it, you are counting pipeline runs, not deliveries.
How a hypothetical service split changes the count without changing capability
Setup
Imagine a hypothetical team -- call them Team Relay -- that owns a single API service. They deploy it roughly ten times per week. Each deploy contains one or more pull requests with code changes. Their deployment frequency is ten per week.
The migration
Team Relay splits the API into three services: auth-service, billing-service, and core-api. The split takes two months. Afterward, each service has its own pipeline. In the first month post-split:
auth-servicedeploys three times per week (mostly stable, few changes).billing-servicedeploys four times per week (active development on a new feature).core-apideploys eight times per week (absorbs most of the old API traffic and gets the most patches).
Total: fifteen deploys per week. A 50% increase over the pre-split number.
What actually changed
Team Relay has the same number of engineers. They merged roughly the same number of pull requests per week (hypothetically, around twelve). The per-PR lead time did not improve. The additional five deploys come from:
- Deploying
core-apiindependently whenever a small patch lands, instead of batching it with unrelated auth changes (good -- smaller batch size). - Deploying
billing-serviceafter config-only changes that the old pipeline would have bundled with code changes (inflated -- these should be tagged config-only). - One rollback of
auth-servicethat the pipeline counted as a deploy (inflated).
After correcting the tags, the meaningful new-code deployment frequency is twelve per week -- a modest real improvement attributable to smaller batch sizes, not a 50% leap in capability.
The lesson
Whenever the number of independently deployable services changes, re-baseline. Compare frequency per service or normalize by the number of in-scope services. If you report a single aggregate number, annotate it: "15 deploys/week across 3 services (previously 10/week across 1 service)."
Pair frequency with context signals
Deployment frequency alone answers one question: "How often do we push changes to production?" It does not answer whether those changes are valuable, safe, or fast. Reporting frequency without companions rewards gaming and punishes caution.
Companion signals and what they reveal
| Signal | What it adds | How to pair it |
|---|---|---|
| Change failure rate | Whether frequent deploys are also reliable. A team deploying 20 times/week with a 25% failure rate is not outperforming a team deploying 10 times/week with a 2% failure rate. | Report side by side. If frequency rises and failure rate rises proportionally, the gain is illusory. |
| Lead time for changes | Whether the pipeline is fast or whether deploys are just batched differently. | A frequency increase with no lead time improvement suggests architectural change (more services), not process improvement. |
| Mean time to restore (MTTR) | Whether the team can recover quickly when a deploy fails. | High frequency with low MTTR suggests a team that ships confidently and fixes fast. High frequency with high MTTR suggests a team pushing broken changes. |
| User-facing change count | Whether deploys contain work users notice. | Count the number of deploys that include at least one user-visible change (new feature, UI fix, performance improvement a user would perceive). If 30% of deploys have user-facing changes, the other 70% are infrastructure or internal -- valuable, but the frequency number overstates product velocity. |
| Deploy-to-PR ratio | Whether frequency growth reflects real throughput or mechanical splitting. | Track the ratio of counted deploys to merged PRs per week. A ratio near 1.0 means each PR ships independently -- healthy in trunk-based flows. A ratio consistently above 1.5 warrants investigation: either commits are being split artificially, non-reproducible builds are generating phantom events, or pipeline events are being duplicated. Pair this with your source-commit deduplication check from the definition card; if the ratio stays elevated even after deduplication, the issue is likely commit-splitting behavior, not pipeline mechanics. |
The deploy-to-PR ratio is particularly useful as a diagnostic bridge between frequency and the other companion signals. If frequency climbs but the ratio also climbs, you know the increase is mechanical, not substantive -- before you even look at failure rate or lead time. This saves time in reviews because you can skip the "is this real?" debate and move directly to root cause.
How gaming manifests differently by workflow
The way deployment frequency gets inflated depends on the team's branching and release model. Recognizing the pattern helps you design the right countermeasure.
Trunk-based continuous deployment: The main gaming vector is splitting trivial changes into separate commits that each trigger a deploy. A developer reformats a file in one commit, updates an import in the next, and fixes a typo in a third -- three deploys, one logical change. The countermeasure is to pair frequency with merge count or PR count. If the ratio of deploys to PRs consistently exceeds 1.5, investigate whether commits are being artificially split.
Release-branch workflow with scheduled deploys: Gaming here looks like creating more release branches or shortening the release cadence on paper without changing the volume of work included. Frequency goes up, but each release is thinner. The countermeasure is to track the number of distinct PRs or commits included per release. A release containing one commit is not a meaningful increase in delivery capability.
Automated dependency update tools (Renovate, Dependabot): These tools can generate a high volume of lockfile-only PRs. Your has_code_change boolean may flag them as code changes because the lockfile is technically a source file that affects runtime behavior. Decide explicitly: do lockfile-only dependency bumps count? A reasonable default is to count them if they change a direct dependency version (since that alters the code your application executes) but exclude lockfile-only changes to transitive dependencies (since those are build-environment maintenance). Document the decision on your definition card.
When you present deployment frequency in a review, always show at least one companion signal. A standalone frequency chart invites misinterpretation.
Examining delivery signals together is more useful than tracking any single number in isolation.
Deployment definition card and duplication checks (copyable worksheet)
Use this worksheet to document your team's deployment frequency definition. Copy it into your team wiki or runbook. Fill in the bracketed fields. Review it when the instructions in the "Ongoing review routine" section apply.
DEPLOYMENT FREQUENCY DEFINITION CARD
=====================================
Team: [Team Relay -- replace with your team name]
Last reviewed: [2025-01-15 -- replace with current date]
Review trigger: quarterly, or when services/pipelines change
IN-SCOPE SERVICES
------------------
| Service | Repo | Pipeline ID | In scope? | Notes |
|-------------------|-----------------------|------------------|-----------|------------------------------|
| core-api | org/core-api | deploy-core-api | Yes | Primary user-facing service |
| billing-service | org/billing | deploy-billing | Yes | Active development |
| auth-service | org/auth | deploy-auth | Yes | Low change rate |
| admin-dashboard | org/admin-dash | deploy-admin | No | Internal tool, not in metric |
WHAT COUNTS AS ONE DEPLOYMENT
------------------------------
A deployment is counted when: a new build artifact containing at least one
source-code commit is promoted to the production environment for an in-scope
service. Each service's promotion counts as one event.
DEDUPLICATION METHOD
---------------------
Primary: source commit SHA comparison (preferred if builds are not reproducible)
Fallback: artifact SHA comparison
Decision on lockfile-only dependency bumps: [Count direct-dependency bumps,
exclude transitive-only lockfile changes -- replace with your team's decision]
EXCLUDED EVENTS (tracked separately, not in frequency count)
-------------------------------------------------------------
- Rollbacks (tag: rollback)
- Config-only changes (tag: config-only)
- Redeploys of the same artifact (auto-excluded by artifact SHA match)
- Canary promotions (the initial canary deploy is the counted event)
- Staging or preview deployments
- Transitive-only lockfile updates (tag: dep-transitive)
DUPLICATION CHECKS
-------------------
1. [ ] Confirm no service is counted in two teams' metrics.
2. [ ] Confirm canary start + canary promotion are not both counted.
3. [ ] Confirm multi-region deploys of the same artifact count as one event
(or document that you count per-region intentionally).
4. [ ] Confirm automated dependency-bump deploys are tagged; decide whether
they count (see deduplication method above).
5. [ ] Spot-check three deploys where artifact SHA differs but source commit
SHA is the same -- these indicate non-reproducible builds and should
not be counted as separate events.
MISSING-EVENT CHECKS
---------------------
1. [ ] Verify each in-scope pipeline reported at least one event in the
last 30 days. If not, investigate: is the service dormant, or is
the integration broken?
2. [ ] Compare deploy count to merge count. If merges substantially exceed
deploys, batching may be hiding true frequency. If deploys substantially
exceed merges, pipeline events may be duplicated.
3. [ ] Spot-check three recent deploys: open the artifact, confirm at least
one source-code commit is present.
COMPANION SIGNALS REPORTED ALONGSIDE FREQUENCY
------------------------------------------------
- Change failure rate (same time window)
- User-facing change count per deploy (sampled monthly)
- Deploy-to-PR ratio (weekly, flag if consistently above 1.5)
SIGN-OFF
---------
Definition agreed by: [Tech lead + EM -- replace with names]
Usage instructions
- Copy the card into your team's documentation.
- Fill in every bracketed field with your actual values. The examples above (Team Relay, core-api, etc.) are hypothetical placeholders.
- Run the duplication checks the first time you fill in the card, and again after every service addition or pipeline change.
- Run the missing-event checks monthly.
- Bring the card to your quarterly metric review (see "Ongoing review routine" below).
Comparing realistic options for counting deployments
Teams often debate what to count. Here are four common approaches with their consequences.
Option A: Count every pipeline run to production
- Advantage: Simple to implement. No tagging required.
- Disadvantage: Includes rollbacks, config changes, redeploys. The number inflates whenever infrastructure churn increases. A bad week with three rollbacks looks like a good week with three extra deploys.
- Best fit: Prototype-stage teams with one service and no rollback automation, where every pipeline run is likely a real code change.
Option B: Count every merge to main (as a proxy for deployment)
- Advantage: Decoupled from pipeline mechanics. Easy to query from version control.
- Disadvantage: Only accurate if every merge triggers a deploy and every deploy comes from a merge. Breaks down with release branches, cherry-picks, or manual deploys. Also counts merges that do not reach production (documentation, test-only changes).
- Best fit: Trunk-based teams with continuous deployment where merge equals deploy, and who exclude non-code paths.
Option C: Count artifact promotions with code-change verification (recommended)
- Advantage: Directly measures "new code reached production." Resistant to inflation from config changes, rollbacks, and redeploys. Requires tagging, but the tagging also powers failure analysis.
- Disadvantage: Requires pipeline instrumentation to compare artifact SHAs and classify commit content. More setup work. Breaks down if builds are not reproducible (see "When artifact-SHA deduplication breaks down" above -- mitigate by using source commit comparison).
- Best fit: Any team that reports deployment frequency to stakeholders or uses it for improvement decisions.
Option D: Count user-facing deployments only
- Advantage: Directly measures product delivery. Filters out internal refactors, dependency bumps, and infrastructure changes.
- Disadvantage: Requires manual tagging of "user-facing" per deploy, which is subjective and labor-intensive. Under-counts valuable reliability and infrastructure work, potentially discouraging it.
- Best fit: Product teams reporting specifically to product leadership on feature delivery cadence -- as a supplementary metric, not a replacement for overall deployment frequency.
Most teams should start with Option C and add Option D as an occasional overlay when product context is needed. Organizing product signals alongside delivery data can help connect deployment events to the work they represent.
Failure symptoms, corrective actions, and escalation points
| Symptom | Likely cause | Corrective action | Escalate when |
|---|---|---|---|
| Frequency jumps after a service split with no change in team throughput | New services added to scope without re-baselining | Re-baseline. Report per-service frequency or normalize by service count. | Leadership is making resourcing decisions based on the inflated number. |
| Frequency drops but lead time and merge rate are stable | A pipeline integration broke, or a service was removed from scope without updating the definition card | Run missing-event checks from the worksheet. Verify pipeline tokens and webhook configurations. | The drop has been reported externally before the cause is identified. |
| Frequency is high but change failure rate is also climbing | Team is deploying more often but skipping review, testing, or staged rollout | Pair frequency with failure rate in every report. Investigate whether the deploys with failures share a pattern (e.g., all from one service, all on Fridays). | Failure rate exceeds the team's stated acceptable threshold for two consecutive periods. |
| Config-only changes are counted as deploys, inflating the number | Pipeline does not distinguish config changes from code changes | Implement artifact SHA comparison and commit-path classification. Re-tag historical data if possible. | The metric is being used in performance conversations or external reports. |
| A single service dominates the count, masking low frequency in others | One high-churn service (e.g., a frontend with frequent copy changes) skews the aggregate | Report frequency per service, or at least identify and annotate the dominant contributor. | A team is claiming high frequency while most of their services rarely deploy. |
| Rollbacks are counted as deploys | Pipeline does not tag rollback events | Add rollback detection: if the deployed artifact SHA matches a previously deployed SHA that was replaced, tag as rollback. | The count is being used to demonstrate reliability improvement. |
| Deploy-to-PR ratio exceeds 1.5 consistently | Commits are being artificially split, or non-reproducible builds are creating phantom new-code events | Compare source commit SHAs across deploys. If multiple deploys share the same source commit, consolidate or switch to source-commit-based deduplication. | The ratio is being used to justify staffing or velocity claims. |
Explicit limitations
- Deployment frequency does not measure value delivered. A team can deploy thirty times a week and ship nothing users care about. Frequency measures the mechanical ability to push changes, not the quality or importance of those changes.
- Cross-team comparison requires identical definitions. If Team A counts config changes and Team B does not, their numbers are not comparable. Standardize the definition card across teams before comparing.
- The definition card captures intent, not enforcement. Unless your pipeline automatically tags and filters events, someone can inadvertently (or deliberately) inflate the count. Periodic spot-checks are necessary.
- Aggregate frequency hides distribution. A team averaging ten deploys per week might deploy nine on Monday and one on Friday. If you care about flow consistency, look at the distribution, not the total.
- This guide does not cover deployment frequency targets. There is no universally correct frequency. A team deploying twice a week with high reliability and short lead times may be healthier than a team deploying twenty times a week with frequent failures.
- Mobile and embedded deployments have different semantics. App store releases involve review queues and user-initiated updates. The definition of "reached production" is murkier. Adapt the definition card accordingly.
- Build reproducibility affects measurement accuracy. If your builds are not reproducible, artifact-SHA-based deduplication will misclassify infrastructure events as new-code deploys. The source-commit-comparison method described above mitigates this, but adds implementation complexity.
Ongoing review routine
Quarterly review (30 minutes)
- Open the definition card. Confirm the service list is current.
- Check whether any new pipeline or deployment pattern has been introduced (e.g., a new canary process, a GitOps controller).
- Run all duplication and missing-event checks.
- Compare the current quarter's frequency to the previous quarter. If the number changed by more than 25%, investigate whether the change reflects a real capability shift or a definition/scope change.
- Review the companion signals. If frequency improved but failure rate or lead time worsened, discuss what happened.
- Check the deploy-to-PR ratio. If it has drifted above 1.5, investigate the cause before presenting the frequency number.
- Update the "Last reviewed" date on the card.
Trigger-based review (when it happens)
- A service is added or removed from the team's ownership.
- The team migrates to a new CI/CD platform or changes deployment topology (e.g., adds multi-region).
- A stakeholder questions the number or compares it to another team's number.
- An automated dependency update tool (Renovate, Dependabot) is added or its configuration changes significantly.
Examining delivery and code quality signals together can surface discrepancies between frequency and the outcomes that matter.
Next-action checklist
- List every production service your team owns and the pipeline that deploys it.
- Write your deployment event definition in one sentence and post it in your team's documentation.
- Implement source-commit-SHA comparison in your pipeline to auto-exclude redeploys (preferred) or artifact-SHA comparison if builds are reproducible.
- Tag rollbacks and config-only changes as separate event types.
- Decide and document whether lockfile-only dependency bumps count, distinguishing direct from transitive dependency changes.
- Run the duplication checks from the worksheet against your last 30 days of deployment data.
- Run the missing-event checks: compare deploy count to merge count for each service.
- Add at least one companion signal (change failure rate or user-facing change count) to your next deployment frequency report.
- Schedule a quarterly review using the routine above; put it on the calendar now.
- If your team recently split or absorbed services, re-baseline your frequency number and annotate the change in your dashboard.
- Share the definition card with anyone who consumes or reports your deployment frequency number, so they understand what it includes and excludes.
Measure what you ship - for real.
Put your team’s AI web-development on the outcome loop.
Get early access