LLM observability has to start before you ship

Most teams treat observability as something you bolt on after a product ships. Karthik Kalyanaraman, co-founder and CTO of Langtrace, says that's already too late for AI.

He explains why LLM observability has to start during development rather than after launch, what to trace first (raw model calls, retrieval, and the framework layer that can silently rewrite your prompts), and why prompt injection currently has no fix — only visibility.

Chapters

• 0:02 — Karthik's path: blockchain nodes to LLM observability
• 2:53 — Building Langtrace out of a 50/50 user reaction

It's a really hard problem to solve, but the least you can do is have visibility into it.

• 7:22 — Agentic AI, tools, and what MCP actually is
• 10:22 — Two real risks: invisible value and prompt injection
• 13:02 — What to trace first: calls, retrieval, frameworks

The unfortunate answer is that it doesn't exist. We are all trying to build that product, but it doesn't exist today.

• 16:20 — How deep to trace multi-step agent trajectories
• 21:43 — Why OpenTelemetry keeps observability vendor-neutral
• 26:52 — Where AI breaks traditional observability assumptions
• 30:18 — Observability during development, not just production

Right now with AI, observability plays an important role during the development cycle itself.

• 33:17 — From tracing tokens to tracing business value

Key takeaways

• Traditional observability mostly matters after a product ships; LLM observability matters just as much before launch, because AI outputs aren't predictable from day one.
• When Langtrace's own team shipped an AI log-summarisation feature, close to half of users called it delightful and the rest said it didn't work — the gap wasn't the feature, it was the team's blindness to why the model behaved inconsistently.
• The single highest-leverage thing to trace first isn't the fanciest part of the stack — it's the raw LLM calls: inputs, outputs, token counts and cost, which alone can catch one user burning through an entire project's budget.
• Frameworks like LangChain or Vercel's AI SDK can quietly rewrite a carefully engineered prompt before it reaches the model — without tracing at that layer, teams can spend hours debugging the wrong thing.
• Once an AI agent has tool access, a single malicious email or document can attempt to redirect its behaviour — prompt injection doesn't yet have a reliable fix, only visibility.
• There's no universal definition of a "good" agent trajectory: 1,000 iterations over a week and five iterations in a day can both represent a successful outcome, so teams have to define what good means for their own product before they can trace toward it.