Case Study: Agent Observability with LangFuse Rollout 2026

Agent observability done badly is invisible until the second outage. This case study walks the ninety-day rollout that took a multi-product SaaS from blind production agents and weekly cost surprises to a stack where on-call engineers walk real incidents from page to root cause inside an afternoon — built on LangFuse with OpenTelemetry semantic conventions, per-tenant cost attribution, drift detection, and deterministic trace replay.

The customer is a mid-sized SaaS with three production agent surfaces: a customer-facing support agent embedded in their product, an internal research agent used by their go-to-market team, and an experimental code-generation agent shipped behind a feature flag to a subset of paying tenants. None of them had observability worth the name. Trace propagation broke at the first tool boundary. Cost showed up monthly at the invoice. Drift was detected by customers, not engineers.

What follows is the rollout exactly as it ran — vendor selection, OpenTelemetry adoption, cost attribution, drift detection, replay infrastructure, and the outcomes measured at day 90. It is meant to be replicated. The phased plan is documented separately in our 30/60/90-day rollout playbook; this post is what happened when a real team ran that plan.

The customer came to the engagement with the usual symptoms of late-stage observability debt. Three agent surfaces had shipped over eighteen months — each instrumented to a different standard, none of them adequate. The support agent had request IDs that propagated through the first model call and disappeared at the first tool boundary. The research agent had spans, but they lived in a homegrown wrapper that no one had touched in six months. The code-generation agent had nothing — print statements to stdout and a Slack alert when error rate crossed five percent.

The triggering event was a Tuesday-morning Slack message from the head of finance: agent inference cost on the prior month's AWS bill had come in twenty-three percent over forecast, with no obvious cause, no per-product breakdown, and no way to tell whether the overrun was a runaway tenant, a model regression, or a price change that nobody had noticed. The CTO asked the senior platform engineer for a root cause. The platform engineer estimated four weeks of investigation, conditional on whether log retention had captured enough request bodies. The CTO called Digital Applied that afternoon.

The brief was simple: get visibility, get cost predictability, get incident response that didn't depend on guessing. The constraint was the budget cycle — ninety days, one quarter, no extension. The deliverable at day 90 was a live walkthrough in front of the executive team showing that the on-call engineer could trace a real production incident from alert to root cause without leaving the observability surface.

The starting metrics were the kind that look fine on a quarterly dashboard and reveal themselves the moment anyone asks a question. Mean time to resolution on the prior two agent-related incidents had been thirty-one hours and fifty-two hours — both ultimately resolved by an engineer scrolling stdout logs and grepping for clues. Cost variance from forecast to actual was running between fifteen and twenty-five percent month over month. Drift was a word the team used in retrospectives, never in dashboards. The brief was not subtle.

The vendor decision was the first hard call of the engagement. The customer had been talking about vendor selection for six months without a decision; everyone had an opinion, no one had a scorecard, and the conversation had become its own form of paralysis. The first week of phase one was time-boxed to five business days for a structured bake-off — four vendors, six audit axes, one decision by Friday.

Each vendor got a half-day of hands-on instrumentation against the same target workload (the support agent), scored against trace coverage, span depth, eval signals, drift detection, cost tracking, and incident response. The scorecard was shared with the customer's platform lead, head of security, and head of finance — each contributing the constraints from their own seat. Security needed self-host or sovereign cloud, finance needed predictable cost, engineering needed portability.

The decisive criterion in the end was not feature parity — all four had passable agent-observability stories — but commercial fit plus the self-host option. LangFuse's self-host shipped through security review in two days, because the data residency story collapsed an entire sub-process the customer had been planning to run with the other vendors. The decision was made on Friday afternoon of week one, the contract signed the following Tuesday, and week two opened with the OpenTelemetry convention spec already in draft.

The most consequential architectural decision of the rollout was made before a single trace shipped: every span would emit in OpenTelemetry semantic-convention shape. That decision cost roughly two extra days in week two — codifying a small internal spec, agreeing attribute names across the three feature teams, and writing a thin internal helper that wrapped LangFuse's SDK to emit OTel-shaped attributes by default. It is the single decision the engagement would most want to make again.

The reasoning is portability. If LangFuse stops fitting in year two — pricing change, capability gap, acquisition risk — the customer can migrate the backend with under ten percent of the instrumentation rewritten. Vendor-specific spans would have made the same migration a quarter of engineering work. The two days spent in week two pre-pays a quarter of work that may never be needed and is cheap if it isn't.

Two implementation details from this phase are worth capturing. First, identity propagation. User-ID and tenant-ID were added to the trace context (OTel baggage) from day one — every span automatically inherited them through the trace context regardless of where in the call tree it lived. The teams that try to propagate identity through ad-hoc argument passing always lose it at the first tool boundary; baggage was the fix. Second, body storage with PII redaction. The redaction pipeline ran inline on the span exporter — no raw PII ever reached LangFuse — and stored prompt and response bodies on 100% of traces with seven-day retention. That decision became load-bearing in weeks 9-10 when replay was built; without verbatim bodies, replay would have been impossible.

The cost-attribution stream was the highest-leverage piece of phase two for this customer specifically — the original triggering event had been a cost surprise on the AWS bill, and the finance partner was watching closely. The implementation was straightforward because identity propagation had already been built in phase one. Token counts on leaf model spans rolled up to the root span; user-ID and tenant-ID tags propagated through trace context; a versioned unit-price table in code converted tokens to cost. Within a week of the surface going live, two outlier tenants had been identified and routed to commercial conversations.

Two implementation details. First, the unit-price table is versioned in code rather than fetched from a vendor API. Provider prices change quarterly; an in-code table dated per update lets the finance team audit cost calculations historically — a prior incident's cost attribution can be recomputed at the prices that applied that day, not today's prices. Second, eval cost is tracked as a separate line item from production cost. LLM-judge calls consume tokens; budgeting them with the production agent obscures both. Separated, both are interpretable.

The final two capabilities of the rollout — drift detection in phase two, deterministic replay in phase three — are the ones that turned observability from a forensic tool into an operating one. Drift detection earned the team the right to alert: instead of static thresholds that fired on noise, hourly rolling-window cron computed rate-of-change on the signals that mattered and routed step-changes to severity- graded alerts with trace URLs attached. Replay closed the loop on the other end: every alert now ended with a walkthrough rather than a guess.

Three drift incidents fired during the engagement, each instructive. The first, in week seven, was a prompt-template regression that dropped golden-dataset eval score by twelve percent on the support agent — caught by drift in under two hours, root cause identified in fifteen minutes (the prompt template change had been deployed without test coverage), fix shipped before the next business day. The second, in week ten, was a tool-schema breakage on the code-gen agent — an upstream API changed a field type, tool-call rejection rate climbed past ten percent on the canary tenants within an hour, the on-call engineer routed the trace through replay and validated the candidate fix against twenty failed traces before deploy. The third, in week eleven, was a runaway tenant on the research agent — per-user cost spiked 50x median, drift fired, the commercial conversation was scheduled the same afternoon.

The single most important implementation detail of phase three was that replay only worked because prompt and response bodies had been stored verbatim from week two. Teams that try to save storage cost in phase one by hashing prompts always discover the trade-off when they reach replay and find it impossible to reconstruct historical inputs. Storage cost here was approximately $40 a month for seven-day retention on full bodies across all three agents — cheap insurance for the replay capability that anchored the day-90 readout.

The day-90 readout was scheduled three weeks in advance and run in front of the executive team — CTO, head of finance, head of customer success, two product leads. The format was the same one the platform lead had used internally for the gate reviews: a real production trace from the prior forty-eight hours, walked live, with replay invoked midway to demonstrate root-cause validation against historical failures. Every number below is from the customer's own measurement, taken at the gate.

The narrative outcomes mattered as much as the numerical ones. The on-call engineer who had previously dreaded agent-related pages came out of the day-90 readout asking for the same observability stack to be extended to the two agents that had been left out of scope. The finance team asked for the per-tenant cost surface to be wired into their forecasting model directly. The CTO scheduled the second-quarter program — coverage breadth across the remaining agents, plus a second tier of eval automation — on the basis of the day-90 walkthrough. The replay demonstration in particular was what made the case; the audit document had been compelling on paper, the live walkthrough was undeniable.

What the customer chose not to do is worth naming. They did not extend to coverage breadth during the ninety days — the two operational agents that had been out of scope stayed uninstrumented. They did not build a custom dashboard layer on top of LangFuse — LangFuse's own surfaces were sufficient for the on-call cadence. They did not migrate the existing logging pipeline; trace IDs cross-referenced into product logs were enough integration for the ninety days. The discipline of depth-on-three-agents rather than breadth-on-five was the program's defining choice.

Five lessons travel cleanly from this engagement to any multi-product SaaS running agents in production. Three traps are worth naming explicitly, because they are the ones the program lead would warn other teams about most firmly. The patterns are general; the implementation details vary with vendor and stack, but the shape of the rollout is replicable across most engineering organisations of similar scale.

For organisations starting from scratch, the replication path is straightforward. Read the 30/60/90-day rollout playbook for the phased plan. Use the vendor TCO calculator to anchor the bake-off scorecard against your own cost profile. The replication risk is rarely technical — the patterns above are well-understood and the vendor surfaces are mature. The replication risk is governance: a program without phase gates and named stream owners is the program that reaches day sixty with patchy coverage and a vendor invoice but no operating leverage.

For teams who want outside help running the program, our AI transformation engagements ship exactly this phased rollout — vendor selection, OpenTelemetry instrumentation, cost attribution, drift detection, replay infrastructure — against whichever vendor fits your stack and sovereignty constraints best. The ninety-day shape is the same; the implementation details vary; the day-90 walkthrough is the bar we hold ourselves to in every engagement.