Dashboard Stories

How to create a DORA metrics dashboard

Get full oversight of your DORA metrics with this step-by-step guide on how to build a DORA metrics dashboard in SquaredUp.

Tim Wheeler

Director of Engineering Excellence, SquaredUp

A DevOps DORA metrics dashboard is crucial for understanding the effectiveness of your teams in developing, delivering, and maintaining software. SquaredUp offers a sample dashboard for Azure DevOps that can be installed when setting up your data source. This guide will walk you through the steps to customize the dashboard tiles to reflect the metrics most relevant to your needs.

Create your free dashboard

Read on to see how to take the sample Azure DevOps DORA dashboard and tailor it to your needs. Feel free to skip ahead by clicking on the links below.

Deployment Frequency
Lead Time for Changes
Change Failure Rate
MTTR

We have also have other out-of-box Azure DevOps dashboards available if you'd like to check them out in our Dashboard Gallery.

What are DORA metrics and why do I need a dashboard for them?

As a brief overview, here are the 4 original DORA metrics and why you want to track them:

Deployment frequency: see how frequently your organization successfully deploys code to production or releases software to check you're meeting your goals.
Lead time for changes (LTC): get an insight into how long it is between a commit and production – this indicates how agile the team is.
Mean time to recovery (MTTR): when there’s an outage or service disruption, you need to know how agile your team is in responding.
Change failure rate (CFR): knowing the percentage of releases that result in issues that impact users can reveal how effective your team is at implementing changes.

As with any set of metrics, you need to take DORA metrics in context together. That’s why building a DORA metrics dashboard will optimize your insights into your DevOps teams.

Getting started

If you’re new to SquaredUp, you will need to create an account.

To get set up:

Connect your Azure DevOps instance to SquaredUp
Set up the Azure DevOps data source
Install the DORA metrics dashboard

Below is an image of a connected Azure DevOps data source. In the top right there is an option to install sample dashboards. Use this if you didn’t add them when you first setup the data source.

Customizing your dashboard

The first change you can make is to use the dashboard variable at the top of the dashboard to select the pipelines that you want to include. This defaults to all, but can be one or several specific pipelines.

Deployment Frequency tiles

The Deployment Frequency metric displays the number of successful pipelines deployed to production over a specified time period. The second tile shows a line graph of daily deployments over a week.

The following steps detail options to customize this tile. These steps can be used in several of the tiles below:

Edit the tile and navigate to the tab ‘dataset1’
Select the ‘Parameters’ tab as below

There are several parameters on this tab you may want to edit to customise the tile to your needs:

Top - by default limited to 20 results per pipeline. You may want to remove this value or set it to a higher number.
Trigger - The method that the pipeline was started. For instance you may only want to see those triggered by a PR.
Result - Set to Succeeded to filter out failed builds
Branch - Defaulted to main. If blank all pipelines are returned. Typically you want to target the name of your trunk branch.

On the timeframe tab you can select a different data range such as ‘30 days’. Note that when you do this, you will need to update the SQL tab to match. The example below is updated for 30 days.

-- total of all successful pipelines / no. of days 
-- using 5 for working days
select 
    count (*)/30
from dataset1

This metric can also be created for release pipelines. To do this:

Select the ‘Data Stream’ tab
Clear filter, so you can see all data streams.
Select or type in ‘Release Runs’ and click ‘Next’
On the Objects tab select or search for the release pipelines you want to include
Select ‘7 days’ for the timeframe

The rest of the settings will be remembered so you can now save the tile.

You can apply all of the above to the second tile as it uses the same data stream. Should you need to switch to the ‘Release Runs’ data stream then you will need to also edit the ‘Shaping’ tab with the following settings:

Group by: Created On
Bucket by: Day
Aggregation type: Count

This will count the number of rows of data and divide by 7.

If we had more tables of data, we could join them together before doing the same count. Click ‘Run Query’ in the top right and our result is returned. Finally, we will select our visualization.

Build it in SquaredUp

Lead time for change tiles

These tiles show the total time for a PR to be created and then merged. In future articles we will look at how to track a code commit through to the build pipeline.

The following steps can be applied to both the Scalar and Line graph tiles. To customize:

Edit the tile and navigate to the tab ‘dataset1’
Select the Objects tab. This data stream is pulling back data from each Repo in your Azure DevOps organisation, in this tab you can pick a specific repos or use a search term with Dynamic selection to bring in multiple repos.
Select the Parameters tab, it lets you configure the following:
- Timeframe basis - Whether the timeframe winder, 7 days for instance, is based on the Open or Closed data of the PR.
- Top - Defaults to 20 records per repo, but can be removed to bring back all PRs for every repo in the timeframe
- Creator - Limit to PRs created by a particular user
- Reviewer - Limit to PRs reviewed from a particular user

On the timeframe tab you can select a different data range such as ‘30 days’
Switch to the SQL tab in the top left to see the SQL calculation

SELECT 
avg(DATEDIFF(hour, creationDate, closedDate)) as LeadTime
FROM dataset1

You can replace ‘hour’ with other date values such as ‘day’ if you want to see the metric at a different scale. Note, update the Label text on the right hand side to match the new date value.

What if you use a different tool for your pull requests? Not a problem, you can easily edit this tile to use a data stream from another data source. Below are the steps for GitHub as an example:

Select the ‘Data Stream’ tab
Select or type in ‘Pull Requests / Closed’ and click ‘Next’
On the Objects tab select or search for the repos you want to include
Select a timeframe
Switch to the SQL tab and update the SQL to use the column names from GitHub. In this case you need to replace ‘creationDate’ and ‘closedDate’ with ‘created_at’ and ‘closed_at’.

Build it in SquaredUp

Change failure rate tiles

Next on the DORA metrics dashboard is change failure rate, this metric measures implementation quality by dividing the number of successful deployments by the number of incidents. To calculate this metric you first have to defines what an incident is, typically this is an outage or service degradation. In this example tile we are defining an incident as a priority 1 and severity 1 bug logged as an Azure DevOps work item.

This tile combines two different sets of data:

Successful build runs, as used in the Deployment tiles
Work Items with a priority 1 and severity 1

To customize:

Edit the tile and select the ‘Pipelines’ tab, this tile uses the Build Runs data stream as shown in the deployment tile section
For the WorkItems tab:
- Select the projects we will query to look for work items. The Dynamic toggle in the top right searches for all projects if the search field is blank or searches for projects matching the search term if one is entered. When the dynamic toggle is off, you can select one or multiple projects by checking the boxes.
- Select the Objects tab, here you can select the projects to query for work items. You can pick a specific project or use a search term with Dynamic selection to bring in multiple projects.

The parameters tab contains the work item query language (WIQL):

SELECT
        [System.Id],
        [System.WorkItemType],
        [Title],
        [Created Date],
        [Closed Date],
        [System.TeamProject]
FROM workitems
WHERE [System.WorkItemType] = 'Bug'
AND [Priority] = '1'
AND [Severity] = '1 - Critical'
AND [Created Date] > @Today - 7
AND [System.TeamProject] = @project

The query can be adjusted to look for different types, priority, severity or dates. Note that when the query has a ‘Created Date’, then this will override the tile’s timeframe.

For more details on using WIQL:

https://learn.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops

If you use a different tool for tracking incidents, you can easily edit this tile to use a data stream from another data source. Below are the steps for switching the source of incidents to Jira, this assumes you have a saved Jira filter for your incidents:

Edit the tile
Select the ‘WorkItems’ dataset tab
Select the ‘Data Stream’ tab
Select or type in ‘JQL query’, then click ‘Next’
- NB: Returning the results from an existing Jira filter is supported by the ‘Issues’ data stream
Select the correct Jira instance from Objects, then click 'Next'
In the parameters tab enter the query from Jira eg

TYPE = 'Bug' and Priority = 'Incident'

Check the viewing pane is now returning the correct data and save the tile

The second tile in this row is for deployment failures. This is to support any investigation into pipelines that are failing to complete successfully. It uses the pipelines selected in the variable at the top of the dashboard to return data.

The Deployment Failures tile returns pipelines builds that have failed to complete successfully. This is configured on the Parameters tab.

The Shaping tab is also grouped on the finish time of each pipeline, so the data can be show as a bar chart.

To customize this tile we can make the same changes to the Build Runs data stream as shown in the deployment tile section.

Build it in SquaredUp

Mean Time to Recover tiles

The last of the 4 core DORA metrics is mean time to recovery (MTTR).

In this example tile we assume that outages are being logged as Work Items in Azure DevOps stored as bugs with a Priority of ‘1’ and a Severity of ‘1 - Critical’. To calculate the metric, we will take the average time that a bug is moved to Done. This done using the SQL tile.

To customize:

Edit the tile and select the ‘dataset1’ tab
Select the objects tab to pick the projects which the WIQL query will run against

Select the parameters tab to edit the WIQL
- The query can be adjusted to look for different types, priority, severity or dates. Note that our example query has a ‘Created Date’ and this will override the timeframe.
- NB: If you already have a query saved in Azure DevOps then you can use the ‘Query work items’ data stream and select the DevOps query directly from SquaredUp

SELECT
        [System.Id],
        [System.WorkItemType],
        [Title],
        [Created Date],
        [Closed Date],
        [System.TeamProject]
FROM workitems
WHERE [System.WorkItemType] = 'Bug'
AND [Priority] = '1'
AND [Severity] = '1 - Critical'
AND [Created Date] > @Today - 7
AND [System.TeamProject] = @project

Select Shaping to filter out specific results based on any of the columns available
The SQL tab contains a query to calculate the average time between the created and closed dates for the work item. This query can be edited to use different date values from Azure DevOps, you can also replace ‘day’ for ‘hour’ if you want a more granular metric.

SELECT avg(datediff(day, [Created Date],[Closed Date])) as Days 
FROM dataset1 
where [Closed Date] < getdate()

The Daily Incidents tile uses the same outage data as above and groups it by day to return a bar chart. You can edit the Objects and Parameters tab in exactly the same way as the MTTR tile. The key difference is that this tile uses grouping on the Shaping tab and doesn’t use SQL for any calculations. To customize:

Edit the tile and select the Shaping tab
- To use a different date field select it from the drop-down ‘Group-by’
- To change the granularity of the grouping select an option from ‘Bucket by ‘+ WIQL’

Build it in SquaredUp

The final dashboard

Your bespoke DORA metrics dashboard should provide an overview of throughput with the deployment frequency and lead time metrics. Displaying these over time is a powerful way to see the impact caused by process change or personnel shifts.

In the bottom two rows, we focus on stability by tracking the change failure rate and MTTR. These crucial metrics helps us understand the amount of deployments that fail to enter production and the time spent resolving incidents.