A technology company’s most valuable assets are its people and data, especially data about the organization itself. By knowing what data to track over time, engineering leaders can measure how efficiently their DevOps teams are operating and enable them to maximize their value stream to deliver the best possible product to end users.
Table of Contents
- Lead Time for Changes
- Deployment Frequency: Key to Continuous Delivery
- Mean Time to Recovery
- Change Failure Rate
- Putting it All Together With DORA Metrics
Lead Time for Changes
Lead time for changes (LTC) is the time between a commit and production. LTC indicates how agile a team is—it not only tells you how long it takes to implement changes but how responsive the team is to the ever-evolving demands and needs of users. The DORA team identified these benchmarks for performance in their Accelerate State of DevOps 2021 report:
- Elite performers: Less than one hour
- High performers: One day to one week
- Medium performers: One month to six months
- Low performers: More than six months
Understanding LTC Benchmarks
LTC can reveal symptoms of poor DevOps practices: If it takes teams weeks or months to release code into production, there are inefficiencies in their process. One can minimize their LTC through continuous integration and continuous delivery (CI/CD). Encourage testers and developers to work closely together so everyone has a comprehensive understanding of the software. Consider building automated tests to save more time and improve the CI/CD pipeline.
Tracking LTC Progress
Because there are several phases between the initiation and deployment of a change, it’s wise to define each step of the process and track how long each takes. Examine the cycle time for a thorough picture of how the team functions and further insight into where they can save time.
Balancing Speed and Quality
One should be careful not to let the quality of their software delivery suffer in a quest for faster changes. While a low LTC may indicate that a team is efficient, if they can’t support the changes they’re implementing or they’re moving at an unsustainable pace, they risk sacrificing the user experience. Rather than compare the team’s Lead Time for Changes to other teams’ or organizations’ LTC, one should evaluate this metric over time and consider it an indication of growth (or stagnancy).
Deployment Frequency: Key to Continuous Delivery
Deployment frequency (DF) reflects how frequently changes are shipped, indicating the consistency of software delivery. Understanding this metric is pivotal for assessing a team’s adherence to continuous delivery objectives. The DORA team outlines the benchmarks for Deployment Frequency:
Performance Benchmarks:
- Elite performers: Multiple times a day
- High performers: Once a week to once a month
- Medium performers: Once a month to once every six months
- Low performers: Less than once every six months
Enhancing Deployment Frequency
Boosting DF involves shipping a series of small changes, offering several advantages. High deployment frequency may uncover development process bottlenecks or project complexity. Frequent shipping enables continual service improvement, facilitating easier issue detection and resolution.
Tailoring Deployment Strategies
For large teams, shipping numerous small changes may not be viable. Alternatively, constructing release trains and adhering to regular shipping intervals can prevent overwhelming team members while maintaining a consistent deployment rhythm.
Mean Time to Recovery
Mean time to recovery (MTTR) is a critical metric that measures the average duration it takes for your team to restore service after an incident such as an outage. This metric provides insights into both the stability of your software and the agility of your team in responding to challenges.
Benchmark Benchmarks from State of DevOps Report
The State of DevOps report identifies the following benchmarks for MTTR across different performance levels:
- Elite performers: Less than one hour
- High performers: Less than one day
- Medium performers: One day to one week
- Low performers: Over six months
Minimizing Downtime Impact
To minimize the impact of degraded service on your value stream, it is crucial to reduce downtime as much as possible. If your team takes more than a day to restore services, consider implementing feature flags. These flags allow you to quickly disable a change without causing significant disruption to your system. Additionally, shipping changes in small batches can make it easier to identify and resolve problems swiftly.
Time to Detect Issues
While mean time to discover (MTTD) differs from mean time to recovery, the speed at which your team detects an issue directly influences MTTR. Rapid issue detection enables faster service restoration, highlighting the importance of proactive monitoring and efficient incident response processes.
Quality Solutions Over Quick Fixes
Avoid implementing sudden changes that compromise the quality of your solution. Instead, focus on deploying durable and comprehensive changes that address underlying issues. Tracking MTTR over time provides valuable insights into your team’s improvement trajectory and enables you to strive for consistent and stable growth.
Change Failure Rate
Understanding Change Failure Rate
Change failure rate (CFR) is the percentage of releases that result in downtime, degraded service, or rollbacks, which can provide insights into a team’s effectiveness in implementing changes. As you can see, there is not much distinction between performance benchmarks for CFR:
- Elite performers: 0-15%
- High, medium, and low Performers: 16-30%
Importance of Change Failure Rate
Change Failure Rate is a particularly valuable metric because it can prevent a team from being misled by the total number of failures they encounter. Teams that aren’t implementing many changes will see fewer failures, but that doesn’t necessarily mean they’re more successful with the changes they do deploy. Those following CI/CD practices may see a higher number of failures, but if CFR is low, these teams will have an edge because of the speed of their deployments and their overall success rate.
Implications for the Value Stream
This rate can also have significant implications for the value stream: It can indicate how much time is spent remedying problems instead of developing new projects. Because high, medium, and low performers all fall within the same range, it’s best to set goals based on the team and the particular business rather than compare to other organizations.
Putting it All Together With DORA Metrics
As with any data, DORA metrics need context, and one should consider the story that all four of these metrics tell together. Lead time for changes and deployment frequency provide insight into the velocity of a team and how quickly they respond to the ever-changing needs of users. On the other hand, mean time to recovery and change failure rate indicate the stability of a service and how responsive the team is to service outages or failures.
By comparing all four key metrics, one can evaluate how well their organization balances speed and stability. If the LTC is within a week with weekly deployments but the change failure rate is high, then teams may be rushing out changes before they’re ready, or they may not be able to support the changes they’re deploying. If they are deploying once a month, on the other hand, and their MTTR and CFR are high, then the team may be spending more time correcting code than improving the product.
Because DORA metrics provide a high-level view of a team’s performance, they can be beneficial for organizations trying to modernize—DORA metrics can help identify exactly where and how to improve. Over time, teams can measure where they have grown and which areas have stagnated.
Those who fall into the elite categories can leverage DORA metrics to continue improving services and gain an edge over competitors. As the State of DevOps report reveals, the group of elite performers is rapidly growing (from 7% in 2018 to 26% in 2021), so DORA metrics can provide valuable insights for this group.
But remember that data will only get you so far. To get the most out of DORA metrics, engineering leads must know their organization and teams and harness that knowledge to guide their goals and determine how to effectively invest their resources.