10 metrics every Azure DevOps engineer should monitor

As a DevOps engineer maintaining deployment pipelines in Azure DevOps, it's crucial to monitor several aspects to ensure the pipelines are efficient, reliable, and secure. Here are 10 key metrics you should focus on:

1. Pipeline Execution and Success Rates

• Build Success/Failure Rates: Track how often builds succeed or fail.
• Deployment Success/Failure Rates: Monitor the success rates of deployments across different environments (e.g., dev, staging, production).
• Deployment Frequency: Track how often deployments happen in different environments to assess deployment cadence.

2. Resource Utilization

• Agent Utilization: Monitor the usage of build agents, ensuring they are efficiently allocated.
• Pipeline Queue Times: Keep an eye on how long jobs wait in the queue before starting, as long queue times can indicate resource constraints.

3. Code Quality Metrics

• Test Coverage: Ensure that unit tests cover a significant portion of your codebase.
• Code Analysis Results: Use tools like SonarQube or Azure DevOps’s built-in tools to monitor code quality, looking for issues like technical debt, code smells, and security vulnerabilities.
• Test Failure Rates: Track the frequency of test failures to catch recurring issues.

4. Security and Compliance

• Dependency Scanning: Monitor for vulnerabilities in third-party libraries using tools like WhiteSource Bolt or GitHub Dependabot.
• Pipeline Security: Ensure that sensitive data (e.g., secrets, credentials) is securely managed within the pipeline.

5. Performance and Availability

• Average Build/Deployment Time: Measure how long builds and deployments take to identify bottlenecks.
• Response Times: Measure the time it takes for the pipeline to respond to triggers, which can impact overall deployment speed.
• Pipeline Downtime: Track any periods where the pipeline or parts of the Azure DevOps service are unavailable.

6. Artifact Management

• Artifact Storage Usage: Monitor the storage usage for build artifacts to ensure you aren’t running out of space.
• Versioning and Retention: Keep an eye on how many versions of artifacts are stored and manage retention policies.

7. Environmental Metrics

• Environment Availability: Ensure that the environments where code is deployed are available and ready for deployments.
• Deployment Success: Check the last deployment to that environment was successful.

8. User Activity and Permissions

• Audit Logs: Regularly review audit logs for any unauthorized access or changes to the pipeline configuration.
• Permission Changes: Monitor who has access to what in your Azure DevOps project to prevent unauthorized changes.

9. Feedback Loops

• User Feedback: Implement mechanisms to gather feedback from developers and stakeholders on the pipeline’s performance and usability.
• Automated Feedback: Use automated tools to report on post-deployment issues like performance degradation or increased error rates in the target application.

10. Cost Management

• Resource Costs: Monitor the costs associated with running pipelines, including build agents and artifact storage.

Tools for Monitoring

These monitoring practices will help ensure that your pipelines are running smoothly, with minimal disruptions and maximum efficiency. But how and where do you monitor these?

The built-in options for monitoring Azure DevOps are fairly limited. If you're familiar with Azure Monitor then particular Application Insights is a good option to go deep on troubleshooting – here's a blog that covers this topic in detail.

If you're looking for a simpler option, SquaredUp is the easiest way to get up and running with dashboards and monitoring for Azure DevOps. It comes with a selection of dashboards out of the box, and is flexible enough to easily roll your own metrics. You can sign-up for a free account to get started.