How to create a DORA metrics dashboard

Tim Wheeler

Director of Engineering Excellence, SquaredUp

A DevOps DORA metrics dashboard is essential in DevOps if you want to understand how effectively your teams are developing, delivering, and maintaining software. Here’s how to build a powerful DORA metrics dashboard in SquaredUp using data from Azure DevOps.

In case you’re curious what a DORA metrics dashboard looks like, here’s the final dashboard I built using SquaredUp’s Azure DevOps data.

Create your free dashboard

Read on to see how I calculated and visualized the four DORA metrics in SquaredUp. Feel free to skip ahead by clicking on the links below.

Deployment Frequency
Lead Time for Changes
Change Failure Rate
MTTR

We have also have out-of-box Azure DevOps dashboards available if you'd like to check them out in our Dashboard Gallery.

What are DORA metrics and why do I need a dashboard for them?

As a brief overview, here are the 4 original DORA metrics and why you want to track them:

Deployment frequency: see how frequently your organization successfully deploys code to production or releases software to check you're meeting your goals.
Lead time for changes (LTC): get an insight into how long it is between a commit and production – this indicates how agile the team is.
Mean time to recovery (MTTR): when there’s an outage or service disruption, you need to know how agile your team is in responding.
Change failure rate (CFR): knowing the percentage of releases that result in issues that impact users can reveal how effective your team is at implementing changes.

As with any set of metrics, you need to take DORA metrics in context together. That’s why building a DORA metrics dashboard will optimize your insights into your DevOps teams. There are many data points you could use to define DORA metrics. For this blog, I’ve chosen ones that work well with Azure DevOps. SquaredUp can generate DORA metrics in many ways and with many different tools, so use these examples as a starting point in building your own DORA metrics.

If you’re new to SquaredUp, you will need to create an account and add the Azure DevOps data source.

How to create the Deployment Frequency visualization

The first visualization on the DORA metrics dashboard that I built is the Deployment Frequency visualization to see if we’re hitting our goals for releases.

One way to calculate deployment frequency is to count the number of successful pipeline runs in Azure DevOps that build and release code into Production. Our production pipeline runs might look like this in Azure DevOps.

From this list of pipelines, we are only interested in successful ones and those for ‘master’ branch, since we know those are the ones deploying to Production.

In SquaredUp, we can display deployment data from Azure DevOps by creating a new data tile and following these steps:

Create a new scope, selecting the ‘Azure DevOps’ data source
Select ‘Build Pipelines’ and select or search for your chosen pipeline
Select the ‘Build Runs with Stages’ data stream
Set the timeframe to 7 days, for example.
Add a filter for the Source Branch so we only return master branch runs
Add a filter for ‘State’ so that we only return successful runs

This is a simple example, but with the same tile you can return data for multiple pipelines across multiple projects and even pick specific stages in the pipeline. It all depends on what ‘Deploying to Production’ represents within your business.

At this point you could pick a visualization, save the tile, and it will show deployments carried out that week. However, for Deployment Frequency, we want the average daily value. To do this we click on the ‘Switch to Analytics’ button in the top right. SquaredUp's SQL Analytics provides a layer of abstraction on top of your data sources that lets you filter and run calculations over it. It provides a universal query language regardless of where your data is coming from in the back end.

The Analytics screen will display ‘table1’ by default. It will have brought across the scope and filters we configured on the previous screen. At this point, you can add more tables if you want to merge in data from other tools. For example, you may also want to see deployments from CircleCI.

Next, we click on ‘SQL Query’ from the left navigation bar. In the SQL query view, enter the following:

SELECT count(*)/7 as Average_Daily_Builds FROM table1

This will count the number of rows of data and divide by 7.

If we had more tables of data, we could join them together before doing the same count. Click ‘Run Query’ in the top right and our result is returned. Finally, we will select our visualization.

Select ‘Visualization’ on the left-hand side. You can now pick a visualization from the drop down. Only visualizations that the current data maps to will be available.

In the example below, I have chosen a Gauge visualization. This can be helpful to show a target. For instance, an organization could be targeting deployments of at least once per day. The Monitoring tab in the top right allows you to set thresholds to indicate the health of the tile (red/amber/green) based on the value shown on the gauge.

Build it in SquaredUp

How to build the Lead time for changes visualization

Next I created the LTC visualization for the DORA metrics dashboard. I wanted to get oversight of how agile our team is in getting updates to production.

To calculate our Lead Time for Change, we are going to use the difference between our pull request’s creation date and when it merges to the master branch. In Azure DevOps, the data could look something like this:

In SquaredUp, we can bring back this data by creating a new data tile and following these steps:

Create a new scope for ‘Azure DevOps’, selecting ‘Repos’ and then picking your chosen repo
Select ‘Completed Pull Requests’ as a data stream
Set the timeframe to 30 days

Now, click the ‘Switch to Analytics’ button so we can use SQL to calculate our metric. Once on the Analytics screen click ‘SQL Query’ and enter the following SQL:

SELECT avg(DATEDIFF(day, creationDate, GETDATE())) as LeadTime 

FROM table1

This query will calculate the number of days each Pull Request (PR) has been open for and then work out the average for all the PRs in scope. Click ‘Run Query’ to see the result.

Finally, we can select a visualization to display the value. In the example below, I am using a Scalar with a label of ‘Days’. Click ‘Save’ to return to the dashboard.

Build it in SquaredUp

How to create the change failure rate visualization

Next on the DORA metrics dashboard is the change failure rate visualization to keep a check on the quality of our implementations.

This metric can be one of the more subjective to define since you need to decide what defines a Change and what you are deeming a Failure.

To create this metric from Azure DevOps data, we will calculate this based on the number of pipelines runs that have failed to deploy to Production. The data we need is in the Pipelines section of Azure DevOps, like our Deployment Frequency example.

In SquaredUp, we can bring up this data by creating a new data tile and following these steps:

Create a new scope, selecting the ‘Azure DevOps’ data source
Select ‘Build Pipelines’ and select or search for your chosen pipeline
Select the ‘Build Runs with Stages’ data stream
Set the timeframe to 7 days, for example
Add a filter for the Source Branch so we only return master branch runs

Now click the ‘Switch to Analytics’ button so we can use SQL to calculate our metric. Once on the Analytics screen click ‘SQL Query’ and enter the following SQL:

Select ( 

    (select count(*) from table1 where result = 'error') / (select count(*) from table1) * 100 

) as Failure_Rate_pct

The query counts the number of pipelines resulting in an error and divides it by the total number of pipeline runs. We multiply by 100 to convert it to a %.

Finish by picking a visualization type. A gauge visualization is always a good choice for a percentage value.

Build it in SquaredUp

How to build the MTTR visualization

Finally, we’ll add the mean time to recovery visualization in the DORA metrics dashboard. This lets me keep a close eye on how fast the team is to identify and resolve an issue.

For this metric, also called Mean Time to Recovery, we will assume that outages are being logged as Work Items in Azure DevOps. These could be stored as bugs with a Priority of ‘1’. To calculate the metric, we will take the average time that a bug is closed as fixed.

To access this data in SquaredUp we’ll use the Azure DevOps native query language WIQL (Work Item Query Language). This query language allows you to search Work Items in the Boards area of Azure DevOps, including Bugs.

In SquaredUp:

Create a new data tile
Create a new scope, selecting the ‘Azure DevOps’ data source
Select ‘Projects’ and select the configurable data stream ‘+ WIQL’

When prompted enter a name for your data stream and enter the following query, replacing ‘My Project’ with your target project.

SELECT system.Title, system.State, Priority, System.CreatedDate, Microsoft.VSTS.Common.ClosedDate
FROM workitems
WHERE system.WorkItemType = 'Bug' and [System.TeamProject] = ‘My Project’

This will return all bugs from within this project. When you click ‘Save’ you have created your own custom data stream.

To finish configuring the tile:

Make sure the data stream you have created is selected
Add a filter for ‘Priority’ ‘Equals’ ‘1’

Now click the ‘Switch to Analytics’ button so we can use SQL to calculate our metric. Once on the Analytics screen click ‘SQL Query’ and enter the following SQL:

SELECT avg(datediff(day, [Created Date],[Closed Date])) as Days 

FROM table1 

where [Closed Date] < getdate()

This will calculate the average time it takes for these priority 1 bugs to be closed. Once you have run the query and a value is returned you can choose a visualization and save the tile.

For a lot of organizations, the data will come from a monitoring tool. The same pattern could be used to calculate the metric with data from tools such as PagerDuty or StatusPage.

Build it in SquaredUp

Creating DORA metrics from other sources

For the purposes of this blog, we have only used simple data from the Azure DevOps plugin. However, with the tile Analytics screen it is possible to combine multiple tables of data sourced from several different tools.

For example, we could have a software development process that uses Jira for task management, GitHub for source control, Azure DevOps for CI/CD, and PagerDuty for service outages. We would create the tiles in the same way using different scopes and data streams. The SQL would have to be updated with the correct field names for the data, but everything else would work the same.

4 metrics on one dashboard, regardless of where your data lives.