End-to-end CI/CD pipeline visibility for Azure DevOps

Dashboard Preview

Challenge 

Azure DevOps (ADO) is a feature rich and powerful tool for CI/CD, testing and work management. However, the native Azure DevOps dashboards are very limited and limit the scope of what you can visualize and how you can display it. For example, there is no way to bring together metrics across projects on one page. You also do not have the ability to aggregate metrics and monitor metrics so you can be proactive in managing DevOps in your organization.  

As a specific example, when you can’t understand the current status of your pipelines, you're not going to know if you are blocked when trying to deploy to production. 

Solution 

You can build powerful Azure DevOps dashboards in SquaredUp using the Azure DevOps plugin to combine data across your whole organization and visualize these in a single tile or dashboard. You can bring in data from your pipelines, releases, work items, test cases, artifacts, agents, deployment pools, resources, and repos. SquaredUp also has out-of-the-box data streams that will let you focus in on pipeline failures and agent usage in a couple of clicks. 

By combining data from across several pipelines and projects, you get the complete picture of the state of your CI/CD processes at a glance. You can quickly highlight failures or slow pipelines that are impacting your business. You can also monitor these tiles and get notified when they break the thresholds that you set. 

SquaredUp Azure DevOps dashboards allow you to split the data across projects, products, or even down to microservices. You will quickly be able to create several dashboards tailored to the needs of different teams. For instance, if you have several microservices for a single product that deploy regularly to production, you can see the state of those pipelines in one tile. You can easily and quickly investigate how efficient you are being with your resources by displaying agent usage and job queues. 

Dashboard walk-through 

To set up an Azure DevOps dashboard in SquaredUp, all you need is the ADO plugin and your organization name and a personal access token from Azure DevOps. Then, you can start building your dashboard. In fact, as soon as you've connected to the plugin, the five out-of-the-box ADO dashboards in SquaredUp will be automatically populated. Plus, you can build custom dashboards to suit your teams.

Here's a quick overview of all the tiles on dashboards used in SquaredUp for ADO. (And a quick overview of the out-of-box dashboards at the end.)

SquaredUp Platform dashboard for DevOps

This is the Azure DevOps dashboard we use internally for a full overview of ADO that lets us react to immediate issues and track our efficiency long-term. The most important metric, in the top left, is the scalar metric showing ‘Build Failures in Master’. Directly beneath, you can see the ID of the pipeline that has failed, in red.

We also get visibility over anything that is stuck in a long build duration and have built an alert for that. Plus, we can see the most common build stage failures to keep track of where we need to improve.

In the second row of visualizations we can also see job queues and agent usage to track efficiency.

Scrolling down the dashboard, we have build durations over the past 7 days and the build run details.

Even further down the dashboard is a table that shows the build runs. The red dot immediately identifies the failed build and even which stage has failed . Then use the hyperlink to drill directly into Azure DevOps platform to see the error in detail.

This table uses a data stream to return all the stages within a pipeline.

If you use work items, you can use the work item query language to write a data stream that will pull back any work item that you use, whether for a use case, test case, or theme. There are plenty of options open to you.

Builds Focused dashboard

There’s also a Builds Focused Azure DevOps dashboard that we use to track all the most important metrics directly related to builds.

The ‘Main branch Build Failures’ tile is a scalar that counts the number of build pipelines that have failed in the last 24 hours. We are also monitoring this tile so that if this number is above ‘0’ then the tile and dashboard will go red. This means you should scope this tile to your most critical pipeline. 

The ‘Build Runs’ tile is a health blocks visualization where each tile is showing the status of a pipeline result. 

‘Build Duration for Main Branch’ is displaying a line graph of the duration of a specific pipeline. 

'Task Failures’ is a Donut visualisation that lets you easily see which tasks are the most common cause of a failure in your pipelines. 

These two tiles allow you to look at the performance of your pipelines by looking at the size of job queues and how many agents are in use at a specific time. 

DevOps overview dashboard

Our DevOps straightforward overview dashboard shows which dashboard we need to worry about and helps us keep track of our overall Azure costs.

Dashboard Server product overview

Finally, our Azure DevOps dashboard looks very different for our on-premises product because failures aren’t a reflection on what we’re putting into production. So, we care more about job queues and build durations because they’re impacting our engineers. You have the flexibility to tailor your dashboards to show what’s most important to your team.

Out-of-the-box dashboards

That’s a run-through of the dashboards that we use in production. But we also have some out-of-the-box Azure DevOps dashboards that can be used as the basis for creating your own tailored versions. These will automatically pull in your live data once you're connected to the plugin.

1. Overview dashboard

This overview dashboard surfaces nearly every object type that you can get in Azure DevOps.

2. My DevOps Organization

This is a high-level view of your DevOps organization with oversight of projects, pull requests, artifact packages, task and deployment groups, and more.

3. Pipeline: simple overview

If you rely on ADO for running pipelines and want to see their aggregated metrics, this simple overview dashboard shows top-line build, deployment and release metrics.

4. Pipeline overview

In addition to build metrics such as failures, runs and health, the Pipeline Overview dashboard surfaces more granular pipeline data including job queues and agent usage to improve efficiency with right-sizing.

5. Single build pipeline

This dashboard is much like the pipeline overview dashboard, but is intended to shine a light on a single, specific build pipeline of your choice.

It will initially use data from the first pipeline it can find from your environment, but this can be easily customized by editing the tiles and selecting a specific pipeline from the Objects section.

Get the dashboards

All of these Azure DevOps dashboards have been built using our out of the box dashboards, so you can expect to be up and running in minutes.

1. Create a free SquaredUp account
2. Add the Azure DevOps data source
3. Just hit the toggle to automatically add out of the box dashboards to your workspace, and instantly view your live data

To see what other dashboards you can create, check out our Dashboard Gallery.