Most teams treat observability as something you bolt on after a product ships. Karthik Kalyanaraman, co-founder and CTO of Langtrace, says that's already too late for AI.
Karthik Kalyanaraman is co-founder and CTO of Langtrace, an open-source observability platform for AI agents built on OpenTelemetry. Before that, he spent years on observability at Coinbase, tracing blockchain nodes that had none of the standard protocols traditional monitoring tools expect.
Every team that ships an LLM-powered feature eventually hits the same wall: the same feature that delights half its users frustrates the rest, and nobody can say why. Kalyanaraman built Langtrace after running into exactly that problem himself — and argues that observability for AI has to start much earlier in a product's life than most teams are used to.
Key takeaways
- Traditional observability mostly matters after a product ships; LLM observability matters just as much before launch, because AI outputs aren't predictable from day one.
- When Langtrace's own team shipped an AI log-summarisation feature, close to half of users called it delightful and the rest said it didn't work — the gap wasn't the feature, it was the team's blindness to why the model behaved inconsistently.
- The single highest-leverage thing to trace first isn't the fanciest part of the stack — it's the raw LLM calls: inputs, outputs, token counts and cost, which alone can catch one user burning through an entire project's budget.
- Frameworks like LangChain or Vercel's AI SDK can quietly rewrite a carefully engineered prompt before it reaches the model — without tracing at that layer, teams can spend hours debugging the wrong thing.
- Once an AI agent has tool access, a single malicious email or document can attempt to redirect its behaviour — prompt injection doesn't yet have a reliable fix, only visibility.
- There's no universal definition of a "good" agent trajectory: 1,000 iterations over a week and five iterations in a day can both represent a successful outcome, so teams have to define what good means for their own product before they can trace toward it.
Why LLM observability starts earlier than traditional monitoring
In traditional software, observability mostly earns its keep after launch. A team ships a product, sets up alerts, and finds out something's wrong when PagerDuty goes off. Kalyanaraman argues that pattern doesn't hold for AI: observability plays an important role during the development cycle itself, because getting predictable outcomes out of a model doesn't happen on the first try, or the tenth.
That changes what observability is for. Instead of a safety net for production incidents, it becomes the mechanism a team uses to find out whether their prompt, their retrieval pipeline, or their overall setup is actually working before anyone outside the team sees it. Once the product does ship, the same loop keeps running: teams watch for edge cases, adjust, and watch again. Kalyanaraman compares it to a fast-moving game of whack-a-mole — the fixes have to be quick, because a slow correction cycle means users churn before the team even notices what went wrong.
The loop doesn't stop at "it works." Trace data becomes raw material for the next iteration: some teams label good and bad traces from production, pull the good ones into fine-tuning datasets, and retrain until accuracy climbs toward 95 or 96%. Others skip fine-tuning and run evaluations against the same labelled data instead, treating quality as something to keep measuring rather than a box to check once. Traditional monitoring rarely asks a team to keep re-training the thing it's watching; LLM observability does, and that's the difference Kalyanaraman thinks most teams still haven't priced in.
Right now with AI, observability plays an important role during the development cycle itself.
The blind spot that led Langtrace to build an observability product
Before Langtrace existed, Kalyanaraman's team ran into the exact problem it now sells a fix for. While still working on blockchain-node observability at Coinbase — later spun out as a company then called Scale3 Labs — the team experimented with using GPT-3.5 to summarise the logs on a client's observability dashboard, letting a user hover over a confusing log line and get an AI-generated explanation of where it came from. It shipped fast, and the reaction split almost exactly down the middle: nearly 40 or 50% of users called it delightful, and the rest said the feature simply didn't do what it was supposed to.
The gap wasn't a product problem so much as a visibility problem. The team hadn't built any way to see why the model behaved well for some users and badly for others, because — as Kalyanaraman puts it — they'd been flying blind around a system that's inherently non-deterministic and opaque. What started as an internal OpenTelemetry-based library — tracking every model call and every trip to the vector store — let them watch prompts and retrieval pipelines in production and tune both. Accuracy moved from roughly 60% to over 90%.
Telling that story to other founders already building with AI turned into the actual pitch: everyone building AI features was running into the same blind spot. Within a few months the team had built OpenTelemetry-based SDKs for the major model providers, vector databases and frameworks, and launched Langtrace as a standalone product in February 2024.
It's a really hard problem to solve, but the least you can do is have visibility into it.
What to trace first: calls, retrieval, and the framework layer
For teams starting from zero, Kalyanaraman's advice is specific rather than aspirational. Start with the raw LLM calls themselves — every input going into the model and every output coming back. Alongside that, trace token counts and cost per call. It sounds mundane, but it catches real problems: a single misbehaving user or a bug in a loop can quietly burn through an entire project's token budget before anyone notices the bill.
The second layer is retrieval. Any pipeline that pulls context into the model — a vector database lookup, a SQL query — needs its own visibility. The question isn't just whether the pipeline runs; it's whether what it retrieves is actually the material the team intended to hand the model.
The third, and the one Kalyanaraman thinks teams underrate, is the framework layer itself. Tools like LangChain, LlamaIndex or Vercel's AI SDK transform prompts and requests before anything reaches the model's API. Skip tracing at that layer, and debugging becomes guesswork: the team assumes the prompt they wrote is the prompt the model saw, which frequently isn't true. Together, these three layers — model calls, retrieval, and framework behaviour — are what Kalyanaraman calls building a comprehensive visibility into what's actually going on in an AI pipeline, rather than a single dashboard that only shows the parts that are easy to observe.
The unfortunate answer is that it doesn't exist. We are all trying to build that product, but it doesn't exist today.
The new risk profile: agent trajectories and prompt injection
Once a system moves from single model calls to multi-step agents, the tracing problem changes shape. A coding agent on a platform like Lovable or Replit might run for a hundred or two hundred steps to finish a single task, calling tools thousands of times along the way. Kalyanaraman's answer is to stop trying to observe everything uniformly: teams start vibes-based, watching customer feedback to find where things actually break, then instrument that region deliberately, optimising for signal over noise rather than total coverage.
The other risk is closer to a security problem than a monitoring one. Once an agent has access to external tools, it's also exposed to prompt injection: content inside those tools that's written to redirect the model's behaviour. Kalyanaraman is candid that this is a really hard problem to solve — the realistic goal for now is visibility into when it's happening, not prevention of it happening at all.
Why open standards matter more as AI and analytics converge
Langtrace is built entirely on OpenTelemetry, the open standard that already let observability avoid the worst of vendor lock-in for traditional software. Kalyanaraman built Langtrace on the same standard deliberately, so a team unhappy with Langtrace's own dashboards could switch providers without re-instrumenting their entire codebase. He sits on OpenTelemetry's GenAI semantic conventions working group alongside engineers from AWS and Google.
That matters more for AI for two reasons. First, multimodal inputs like images and video don't fit assumptions built for text-based traces. Second, the line between product analytics and observability is blurring, since both are now looking at the same underlying data — raising an obvious question about why the same conversation data, including whatever personal information it contains, ends up duplicated across separate systems.
From tracing tokens to tracing value
The furthest-out question Kalyanaraman was asked is also the one this podcast is built around: can observability move from tracking token usage to tracking whether an AI feature is actually delivering business value? He thinks the industry will get there, but the obstacle is practical — cutting down how much manual labour it takes to judge a trace as good or bad. Langtrace is experimenting internally with running batch inference over stored trace data to flag good and bad traces against criteria a team defines themselves.
His closing advice undercuts any temptation to treat this as something that automates itself away just yet: there's still no substitute for someone looking at the data on a regular basis, deciding for themselves what "working" looks like for their own product.
Chapters
- 0:02 — Karthik's path: blockchain nodes to LLM observability
- 2:53 — Building Langtrace out of a 50/50 user reaction
- 7:22 — Agentic AI, tools, and what MCP actually is
- 10:22 — Two real risks: invisible value and prompt injection
- 13:02 — What to trace first: calls, retrieval, frameworks
- 16:20 — How deep to trace multi-step agent trajectories
- 21:43 — Why OpenTelemetry keeps observability vendor-neutral
- 26:52 — Where AI breaks traditional observability assumptions
- 30:18 — Observability during development, not just production
- 33:17 — From tracing tokens to tracing business value
About the guest
Karthik Kalyanaraman is co-founder and CTO of Langtrace, an open-source observability and evaluation platform for AI agents built on OpenTelemetry. Before starting the company, he worked on blockchain-node observability at Coinbase, and earlier built low-level infrastructure at HP (via Teradici's remote-desktop streaming technology) and VMware's data-centre storage layer. He also sits on OpenTelemetry's GenAI semantic conventions working group.
Where to listen
Related content
- How to avoid blowing the budget on Azure AI
- Building a unified enterprise observability strategy
- The Single Pane of Glass in Modern Observability
- Why dashboards still matter in the age of AI
Most of what Kalyanaraman describes — tracing calls, watching retrieval, catching drift before it reaches users — depends on being able to see that data in one place to begin with. If that's the gap for your team, you can start using SquaredUp for free.
