Why engineering needs (a new kind of) BI

As engineers, we are swimming in data. Drowning even. So why do we find it hard to answer even simple questions like – “how are we doing?”. We believe it’s time for a new way for engineering teams to interact with their data.

Data, everywhere.

The digital systems we build, the tools we use to build them, and the systems we use to operate them all overflow with data. From modern observability stacks to DevOps automation. The potential to use this data to develop more efficiently, operate more reliably and deliver better user experiences is immense.

We have all this data, yet we find the simplest of questions ­­– "how are we doing?" – difficult to answer. Harder still is answering questions like "what is going wrong?", "where can we optimize?" and "how can we improve?".

The reason we find it difficult is, of course, the familiar problem of “missing the forest for the trees”. Day-to-day, we are immersed in the data we generate, but data in each tool is locked away behind an API or query interface we don’t have time to learn. It is siloed from other data and difficult to share.

The problem of visibility gets worse with scale. Engineering teams race ahead to develop new features, start new projects and adopt new technologies. As our systems get larger, more complex, and more dynamic, we increasingly lose control – of costs, security, reliability, performance, quality, and productivity. As the stakes get higher, the insights get harder.

Everyone loves a dashboard?

Dashboards have become a bit of a guilty pleasure. We know we should be doing something better, but can’t get enough of them. Are dashboards enough? Can we do something better?

When someone asks ‘I want to see a dashboard’, what they usually mean is ‘I want to feel informed and in control’.

Like this:

What they ask for... An SLO dashboard

What they want... Quality of services to be measured, incidents to be resolved quickly, reliability issues identified and improved.

Or this:

What they ask for... A cost dashboard

What they want... Budgets not to be exceeded, overspend explained, costs attributed to owner, efficiencies identified, future spending predicted.

The desire for a ‘dashboard’ flows down to every level of the engineering organization. Support and incident teams need to know where to look when things go wrong. Infrastructure teams need to spot trouble before it impacts the business. And security teams need to know, well, everything.

So when it comes to data, while we like to talk about dashboards, we need more than just dashboards. We need a way to plug into our data, visualize it, perform an analysis, share the insights. And we need to do this continuously, iteratively and collaboratively, ultimately so that we understand what is happening and make better decisions.

Outside of engineering, these capabilities are called business intelligence (BI). Business teams have established tools like PowerBI and Tableau for this need.

BI tools have three important characteristics:

• independent of the where the data is generated, collected or stored
• designed specifically for analytics and sharing answers
• flexible enough to be tailored to a business’ specific context

While engineering teams have been overflowing with data, we haven’t had the analytics tools to match. We think it’s time engineering teams have their own BI – one that’s built for their data, to model their complex systems, and for the real-time analysis engineering demands.

That’s why we built SquaredUp.

A new kind of BI, designed for engineering teams.

The challenges that modern product, engineering and IT teams face are similar to those of using business data, but distinctly different.

A business intelligence platform for engineering needs to:

• Plug into anything – Engineering teams need the freedom to adopt new tools and technologies. Data will always be distributed.
• Correlate data across tools – Collecting data is a challenge of the past, connecting data is the challenge of the future.
• Provide answers in real-time – Reporting may be monthly, but engineering decision-making happens in real-time.
• Enable collaboration across teams – Engineering teams are full of nice people but opinions differ, knowledge is scattered, loyalties run deep and walls get built. Teams need to be able to work independently, but play nicely.
• Ultimately, simplify – Engineering systems are complex. To be successful, we must simplify.

This is our mission

We believe engineering teams can interact with data in a new way. To provide visibility, create shared understanding, and to enable better decisions. We want to empower engineering teams to move fast, stay in control and build better technology.

We’re on a mission to empower engineering teams with data.