Most teams assume logs and metrics are what an AI agent needs during an incident. When Kyle Forster measured it, they accounted for about 30% of the data.
Kyle Forster has spent his career on systems that aren't allowed to fail. He learned reliability engineering at Cisco, founded the software-defined networking company Big Switch Networks — acquired by Arista — and then joined Google's Kubernetes team, where the SRE culture reminded him of patterns he'd seen before. RunWhen, his current company, builds agents that run and write automation against production systems.
That last point is the hard part. Plenty of tools will summarise an alert. Very few will touch a Fortune 100 production system and make a change. Getting there forced RunWhen to measure what its agents were actually consuming — and the answer reframes what observability data is for.
Key takeaways
- Logs and metrics accounted for roughly 30% of the data reaching LLMs in RunWhen's platform. The other 70% was CLI calls, API calls and SQL queries.
- Most of that 70% is historic — gathered by agents exploring the system between incidents, not during them.
- Repeated names across environments are the single biggest accuracy killer, and metrics labelling is where it does the most damage.
- Agents that explore and fail produce higher final accuracy than agents that get things right first time and never test the edges.
- Diversity of engineering background in the automation library was one of the strongest predictors of accuracy — stronger than any single team's expertise.
- 85% accuracy on alert triage isn't a milestone on the way to something useful. Forster's view is that it has negative value.
- L1 observability work is going to tokens. L3 judgment work isn't, and Forster expects it to be paid considerably more.
Logs and metrics are only 30% of what the model sees
In a blog post last December, Forster published an analysis of his own platform's LLM calls, asking whether humans still need to read logs. The finding that caught attention was the split: logs and metrics made up around 30% of the data going to the models. Honeycomb's Charity Majors asked the obvious follow-up — what's the other 70%?
It turned out to be CLI tool calls, API calls and SQL queries. And the timing mattered more than the composition. Roughly 70% of the data reaching an LLM during an incident was historic, collected while agents explored the system between incidents.
Forster credits Majors with the reframe that made it click: that behaviour looks like a senior SRE poking around a large system when nothing is on fire. They live on the CLI, they make API calls, they explore well beyond the dashboard — and that accumulated familiarity is what makes them effective when something breaks. RunWhen's data showed agents converging on the same pattern without being designed to.
The conclusion he draws is blunt: "you can't build AI on top of metrics and logs alone. It's the wrong data set."
If you're chasing very high accuracy, metrics and logs alone are the wrong data set.
Why that 70% deserves a pillar of its own
Forster is careful to position himself as a visitor to observability rather than a native — he was a networking engineer, and he sat out the original arguments about whether there were two pillars or four. But his view as an outsider is that a new one is forming.
What he describes is a semi-structured, tagged record of what agents and humans found while crawling a system: long traces of what was checked, what was ruled out, what turned out to be irrelevant. Not telemetry about the system's state, but a record of the investigation into it.
Its value is corrective. Most estates don't have clean metric ontologies, and most have observability gaps — some known, many not. A trace of what an agent actually found while exploring is what lets you interpret imperfect telemetry, because it supplies the context the telemetry is missing. In Forster's framing, it's the data set you mine when you feed any of the existing three pillars into a model.
The naming problem that breaks humans and models alike
RunWhen identified roughly thirty recurring problem areas in logs and metrics. The one Forster keeps returning to is names — the same word meaning different things, and different words meaning the same thing. Is "store" the same as "location"? Is a "collection" a vector database or a Java error?
His example is a microservice that exists in both staging and test under an identical name. A readout of what happened to it over the last two hours is useless without knowing which environment it came from. Humans manage this by carrying an unwritten sense of where they are — and get it wrong regularly, as anyone who has run a kubectl command against the wrong context knows. Forster's point is that this isn't primarily a code problem. It's an interface problem, and it degrades both kinds of reader: "it crushes LLM algorithms and it crushes humans typing in."
Metrics labelling is where it gets worst, because there's little surrounding text to disambiguate from. A model can generate a paragraph-long metrics query that silently sums staging, test and production together, and verifying whether it carried the right context through is slow, manual work. Forster thinks this is much of why early metrics-based AI SRE tools performed so poorly on estates without near-perfect labelling.
Exploration and failure are what make agents accurate
There's an ongoing argument in coding agents about whether accuracy comes from a larger model getting things right first time, or from a smaller one allowed to run, make mistakes and catch them in verification loops. Forster's position — and he acknowledges it's hard to benchmark — is that the second approach wins, and that the trace of the failures is why.
Exploring the edges and finding that things don't work generates information that getting it right immediately never produces. He calls the accumulated result peripheral intelligence, and treats it as the reason RunWhen reaches high-90s accuracy rather than a proprietary technique. He also sees the same pattern in the more advanced competitors: they've clearly built this kind of memory in, and they're using data well beyond metrics and logs.
The same logic showed up in something RunWhen didn't set out to test. Early on, the team split over what the automation library should cover — the SRE-leaning group wanted kubectl and cloud CLI commands, the developers wanted deep log analysis. The argument got personal enough that building both was simply easier than resolving it. Single-track versions of the library performed reasonably. Merged versions performed markedly better. Adding a data specialist's perspective produced another jump. Adding real user monitoring-style testing produced another.
Forster's conclusion is that "diversity of engineering backgrounds, all focused on troubleshooting the same problem, is one of the greatest determinants" of accuracy — which makes the question of who contributes to an agent's operational knowledge a design decision rather than a staffing one.
Why 85% accuracy is worse than nothing
Forster draws a sharp line between two problems that look adjacent and aren't: alert triage, where a system reaches 80-85%, and open-ended investigation, where the bar is 95% and up.
He describes a technical lead who had run a competitor's tool and was shutting the project down. Their summary of what it achieved is the sharpest line in the episode: twenty incoming alerts had been successfully converted into forty root cause hypotheses.
Twenty incoming alerts became forty root cause hypotheses. That added no value at all.
A system that offers three hypotheses and promises one is correct hasn't reduced anyone's work. Forster's assessment is that it consumes tokens, human time and investor capital while producing nothing — and that DIY versions typically look impressive on a laptop and fall apart once they meet a real system.
The threshold matters because the valuable capability sits on the other side of it. Not triage, but deep research: asking an open question and getting a thoroughly investigated answer. That problem is orders of magnitude harder, and it's the one that lets people operate well above their current level.
Which SRE roles go to tokens, and which get a pay rise
Forster's answer here has changed over the past year, and he offers it as career advice he acknowledges is unpopular.
L1 operational support — read the observability data, reach a conclusion, escalate — he expects to disappear into tokens. Every employer he speaks to is trying to move that work across as fast as possible. Anyone whose ambition stops at that role should be worried.
L2 has longer tenure, because reasoning over tribal knowledge, documents and CLI exploration is a much messier problem than reading a dashboard back. But employers want as much of it tokenised as possible too.
L3 is different in kind. This is where production operations become SRE: deciding whether to make an emergency infrastructure change, or spend the time getting context to the development team for a patch, or do the first to buy time for the second. Forster doesn't expect that judgment to be automated at all.
If you're using AI to read your dashboards, that job is going away. If you're using AI to write code that automates work, that job is about to pay considerably more.
The people he expects to gain are those in senior roles who learn to build automation — the subject of a post he wrote earlier this year arguing SREs would see something like a 30% pay rise. His example is a senior SRE who hadn't written code professionally in twenty years, who used RunWhen to automate a set of recurring questions in about 45 minutes. She supports 300 developers; the questions she automated were ones she fielded daily.
It reshapes on-call, too. At RunWhen the goal of a shift isn't to resolve incidents — it's to leave the system more automated than you found it, with more automated diagnostics able to confirm or rule out causes without a person. Forster compares the emerging role to flying a fighter jet: an intense five or six hours that produces what used to take months, followed by a real need to recover.
Chapters
- What RunWhen does, and what "safe for production" means — 4:09
- When a coding agent escalated its own privileges — 5:35
- The 30/70 split in LLM data — 7:38
- Why repeated names break models and humans — 10:05
- Is exhaust a new pillar of observability? — 15:52
- Why failure traces improve accuracy — 18:39
- Which SRE roles survive, and which don't — 21:04
- Diversity of engineering background and accuracy — 29:34
- Why 85% accuracy has negative value — 31:31
- The SRE role in five years — 36:35
About the guest
Kyle Forster is the founder of RunWhen, which builds agents that run and write automation safely against production systems. He previously founded Big Switch Networks, an early software-defined networking company acquired by Arista, and worked in product management on Google's Kubernetes team. RunWhen's registry of agent skills templates is the clearest illustration of what the platform does.
Where to listen
Apple Podcasts · Spotify · YouTube
Related reading
[Placeholder — needs three to five internal links with descriptive anchor text. Candidates depend on what's live: existing Operationally Intelligent episodes, and anything on observability data quality, Kubernetes monitoring, or AI in IT operations.]
