Most teams assume logs and metrics are what an AI agent needs during an incident. When Kyle Forster measured it, they accounted for about 30% of the data. Forster founded RunWhen, which builds agents that run and write automation against production systems — after founding Big Switch Networks (acquired by Arista) and working on Google's Kubernetes team.
He explains why the other 70% — CLI calls, API queries and dead ends agents accumulate exploring a system between incidents — behaves like the knowledge a senior SRE builds over time, why repeated names across environments are the biggest accuracy killer, and why 85% accuracy on alert triage has negative value rather than being a milestone.
Chapters
- 4:09 — What RunWhen does, and what "safe for production" means
- 5:35 — When a coding agent escalated its own privileges
- 7:38 — The 30/70 split in LLM data
If you're chasing very high accuracy, metrics and logs alone are the wrong data set.
- 10:05 — Why repeated names break models and humans
- 15:52 — Is exhaust a new pillar of observability?
- 18:39 — Why failure traces improve accuracy
- 21:04 — Which SRE roles survive, and which don't
- 29:34 — Diversity of engineering background and accuracy
- 31:31 — Why 85% accuracy has negative value
Twenty incoming alerts became forty root cause hypotheses. That added no value at all.
- 36:35 — The SRE role in five years
If you're using AI to read your dashboards, that job is going away. If you're using AI to write code that automates work, that job is about to pay considerably more.
Key takeaways
- Metrics and logs only account for about 30% of the data an AI agent draws on — the rest is the CLI calls, API queries and dead ends it accumulates exploring a system over time.
- Repeated names across environments are one of the biggest accuracy killers, for models and humans alike.
- Failure traces, not just successful ones, measurably improve an agent's accuracy.
- Diversity of engineering background feeds into how accurately an agent performs.
- 85% accuracy on alert triage isn't a milestone — it has negative value, turning twenty incoming alerts into forty competing root-cause hypotheses.
- The SRE roles built around reading dashboards are at risk; the ones built around writing automation that does the work are becoming more valuable, not less.
