How KPIs lose their meaning and what to do about it

Once you've published more than a handful of KPIs, you eventually need a way to summarize them. A total cost. An overall health status. An organization-wide SLA. Something that lets you answer the big questions without opening ten different dashboards.

Summarizing those into a handful of KPIs usually feels straightforward. You add things together, average them, or collapse several statuses into one. The dashboard becomes easier to read, and nothing looks obviously wrong.

So why do these KPIs end up misleading you? Let's look at a few examples.

Why a "healthy" status can hide a real outage

Let's start with the simplest example: reducing application health to a single KPI.

At first glance, this seems completely reasonable. An executive dashboard doesn't need to show the health of every individual service. It only needs to answer one question: is the application healthy?

Imagine an application with four services. Three are healthy, while one is failing. If the dashboard reports the application as healthy, that might seem reasonable. After all, most of the application is working.

But what if that one failing service handles payments?

Now the same 3-out-of-4 healthy result tells a very different story.

You could decide that any critical service failure should make the whole application unhealthy. But then what happens when a non-critical reporting service fails? Should that have the same effect?

Why averaging uptime and SLA numbers is misleading

Percentages feel more objective than health because they're already numbers. In reality, they can hide just as much.

Take uptime. A service that's completely unavailable for one full day and then runs perfectly for the other 364 can still report 99.73% uptime for the year. The calculation is correct, but it tells you very little about what customers actually experienced during that outage.

Things become more complex once dependencies are involved.

Imagine your application depends on a database, a CDN and an authentication service. If any one of those services fails, your application fails too. In that situation, your overall availability isn't the average of the three individual SLAs.

Assuming those services fail independently, the combined availability is the product of their probabilities. Two services with a 99.9% SLA don't combine to 99.9%. They combine to roughly 99.8%. Add another dependency and the gap widens further.

Again, the maths isn't wrong. So why does the number still feel off?

Why comparing teams can be just as misleading

The same problem appears when KPIs move beyond systems and start comparing teams.

I ran into exactly this while building monthly KPI reports for senior management. Every infrastructure team had a simple KPI beside its name — Windows, VMware, SCCM and PeopleSoft — that showed whether it was meeting its operational targets. It gave leadership a quick way to see where attention was needed.

Looking at those reports month after month, the Windows team almost always appeared to be one of the strongest performers.

It wasn't obvious why until we looked more closely at how that KPI was being calculated.

A significant number of Windows incidents originated from noisy alerts that should never have counted as incidents in the first place. The KPI had no way of telling the difference. It simply reflected what the monitoring system recorded.

The PeopleSoft team had a very different reality. Each incident often took hours of investigation before it could be resolved, so a single incident represented considerably more operational effort than the numbers suggested.

On paper, both teams were being measured using the same KPI. So how could the same number mean two different things?

What’s common across these examples?

At first glance, these examples don't seem related. One is about application health. Another is about availability. The third is about comparing teams. But look closer at what actually went wrong in each one.

In each case, we're taking something complex and reducing it to a single KPI. That reduction requires decisions about what matters, how different signals should be weighted, and what the final number or status should mean.

With application health, “Healthy” isn't a fixed measurement. It depends on the rules used to decide which component states matter and how they should roll up into an overall status.

The availability KPI broke down because the three services weren't independent. Averaging their SLAs treated them as if they had nothing to do with each other, when in reality one failure could take the whole system down.

The team comparison broke down because the same KPI was quietly measuring two different things. For Windows, it was catching alert noise. For PeopleSoft, it was reflecting hours of real investigation.

Three different KPIs, three different failure modes, yet none of them involve a calculation error. The health KPI followed the rule it had been given, the availability calculation was mathematically correct, and the team comparison faithfully reflected the data it had been been given.

Before you decide how a KPI should be calculated, you first need to understand what kind of information it's summarizing. Different kinds of KPIs behave differently, and that's where many dashboards quietly lose meaning.

Quantities are built from metrics such as total spend, tickets closed or requests served. Each value contributes independently to the total, so adding them together preserves what the KPI is trying to measure.

Relationships are different. Availability is a good example. The result depends on how components interact with one another, so the aggregation needs to reflect those dependencies rather than simply averaging the numbers.

States describe conditions rather than quantities. Health, risk and status don't become more meaningful because they're averaged. They need explicit business rules that determine how individual states become one overall KPI.

Comparisons only work when the underlying measurements mean the same thing everywhere they're collected. A KPI comparing teams only makes sense if every team is being measured using the same definitions.

Treating all of these as though they can be combined in exactly the same way produces dashboards that look perfectly reasonable while gradually drifting away from what the KPI was originally meant to represent.

Before you aggregate a KPI, ask these questions

The examples above cover some common patterns, but they won't cover every KPI you'll come across. These four questions can help you decide how to summarize the next one.

Does this KPI represent an amount?

If every value contributes independently to the total, adding them together usually preserves the meaning. Total spend, requests served and tickets closed often fall into this category.

Does the relationship between the parts matter?

Availability falls into this category because the result depends on how the underlying components interact, not simply on the values themselves.

Is this KPI describing a state rather than measuring a quantity?

Health, risk and status need explicit business rules. There isn't a meaningful average between healthy and unhealthy.

Does this KPI mean the same thing everywhere it's measured?

Before comparing teams, systems or business units, make sure they're measuring the same thing. Otherwise you're comparing differences in definitions rather than differences in performance.

These questions won't tell you exactly how a KPI should be combined. They do make it much harder to choose an aggregation rule that changes what the KPI is supposed to represent.

Closing thoughts

As dashboards grow, they naturally move towards summary KPIs. That's exactly what they're supposed to do. They simplify large amounts of information into something people can understand quickly.

The goal is making sure the summary still represents what you're trying to measure. Once you understand what a KPI is really representing, choosing how to summarize it becomes much easier.