Case Study: Agentic Customer Support at Fintech, 50% Deflection

Agentic customer support at a mid-market fintech reached 51% deflection at month five without CSAT damage. The function was carrying roughly ten thousand inbound tickets a month, sixty percent of them concentrated in a handful of repeating archetypes, and the team had a year-end mandate to reduce cost-per-ticket without conceding the customer experience that had defined the brand. This piece is the case study underneath that outcome — the program that produced it, the gates that held, and the lessons that replicate.

The headline number is not the interesting part. Fifty-percent deflection has been achievable for several years with sufficient tolerance for CSAT damage; the harder problem is fifty-percent deflection while CSAT holds. That problem is mostly about program design — knowledge audit, RAG build sequencing, pilot governance, escalation rubric — and very little about model selection. The model is the easy part of an agentic support program; the program around it is the expensive part to get right.

What follows is the engagement as it ran. The situation as we found it, the four-stage approach (knowledge audit, RAG build, CSAT-controlled pilot, escalation rubric), the outcomes against the original targets, and the lessons that any other support function can apply. The fintech vertical produced specific constraints — regulated communication, hallucinated-policy sensitivity, audit-trail expectations — but the underlying playbook generalises broadly beyond that vertical.

The client was a mid-market fintech, two years past Series B, with a customer base that had grown faster than the support organisation underneath it. Inbound volume was running at roughly ten thousand tickets per month across email and in-product chat. The team had grown to twenty-two agents and was about to make the call to either keep hiring or invest in automation; the math on the hiring path no longer fit the unit-economics targets for the year, so the decision came down to whether automation could safely take a meaningful slice of volume.

The ticket mix was concentrated. A rough sixty percent of volume sat inside the top twenty archetypes — account-status questions, payment confirmations, settlement-timing inquiries, password and access flows, document re-issuance, basic on-boarding questions. The remaining forty percent was long-tail, including the categories where automation was already disallowed by policy: dispute initiation, fraud reports, churn-saves, and anything regulated. The shape of the volume distribution was the engagement's opportunity — a deflection ceiling that was likely real, but only if the program could avoid contaminating the long-tail with confident-but-wrong responses.

The original brief was "reach 60% deflection" — a number the client had pulled from a vendor pitch deck. The engagement reframed the target inside the first two weeks. The right target was the highest deflection number the CSAT band would allow, instrumented and gated, with hallucinated-policy rate as a hard ceiling. That reframing is the most consequential decision we made in the entire engagement.

Two further situational notes mattered for the design of the program. First, the client had a strong existing knowledge base that had not been audited in roughly eighteen months — every article was reachable from the help centre, but product behaviour had moved underneath several dozen of them. Second, the regulatory context was sensitive enough that any policy statement produced by the AI had to be defensibly grounded in current documentation, with an audit trail back to the source. Both points shaped the knowledge audit and the RAG build that followed.

The first three weeks of the engagement were spent entirely upstream of the model. No customer traffic touched any AI in this window; the work was an inventory of every help-centre article, internal SOP, and macro template, mapped against the top archetypes pulled from the previous ninety days of ticket data. The output was a four-bucket intent catalog and a flag list — every document that was stale, missing, contradictory, or unowned. That flag list became the work backlog for the remainder of the audit window.

The decision to organise the audit into four intent categories rather than a flat archetype list was load-bearing for the rest of the program. Each category carried a different deflection ceiling, a different confidence floor, and a different escalation pattern. Bundling them into a single "top archetypes" list would have produced an undifferentiated model with under-tuned confidence thresholds — the most common failure mode we see in support-AI deployments.

One operational detail is worth surfacing from the audit work. The five-column row shape (doc ID, title, owner, last-updated date, product-match status, mapped archetype) was deliberately narrow — the goal was a spreadsheet the support team would actually maintain after the engagement closed, not an elaborate tracker that would atrophy. The audit found a hundred and forty-three articles needing some level of update; roughly forty were updated before the pilot opened, the rest were triaged into "update by day 60" or "deprecate" buckets. The deflection ceiling at month five would have been lower by an estimated ten to fifteen percentage points without this work.

The retrieval layer was built against the top-200 ticket archetypes identified during the audit. Two-hundred rather than one-hundred because the category split needed enough breadth inside each bucket to avoid retrieval gaps — the long tail inside a category produces the awkward edge-case answers that erode CSAT faster than the head archetypes produce deflection wins.

Chunking strategy and re-ranker selection were category-driven rather than uniform. Account-and-access flows tolerated coarser chunking because the resolution paths were short. Policy-and-documentation flows required fine-grained chunking with policy-version metadata attached to every chunk, so the model could refuse to answer when the most recent policy version was not in the retrieval set. Payment-status queries pulled from structured API data, not embedded text, and were treated as a tool-call path rather than a retrieval path. This category-aware design was the most consequential engineering decision in the build.

Retrieval quality was validated against a held-out set of three hundred and fifty real customer queries — fifty per category, plus an additional long-tail batch — before any generation step ran through it. The held-out evaluation produced a per-category retrieval-quality score that drove chunking adjustments for the categories that landed below target. The discipline of validating retrieval before generation is one of the cheapest improvements to a RAG program; debugging a hallucination caused by a retrieval gap is meaningfully harder than debugging one caused by the model itself, and the held-out evaluation surfaces the former before it ever reaches a customer.

One nuance worth noting on tool integration. The payment-status category was the place where the engagement spent the most engineering time relative to language-model time. Settlement-timing answers in particular required not just the current state of the payment but the predicted settlement window from the payment processor — a derived value, not a direct API field. Getting this right meant the agent could answer the question definitively ("your transfer is scheduled to settle Tuesday morning") rather than vaguely ("transfers typically take 1-3 business days"), which had a measurable effect on resolution CSAT inside this category.

The pilot opened in week four at 1% of inbound volume. Routing was a simple session-hash rule — one customer in a hundred, deterministic, repeatable. The other ninety-nine percent of volume stayed on the human-only path for the duration of the pilot window, which ran four weeks.

Three measurement layers were live before the pilot opened. Resolution CSAT measured immediately after each conversation closed; delayed CSAT measured at the seventy-two-hour mark to catch the cases where the conversation closed cleanly but the underlying issue resurfaced; and a model-scored conversation CSAT — the model rating its own confidence and resolution quality on every turn — was surfaced to a daily QA queue. Baseline readings on all three tiers were taken from human-only traffic in the fortnight before the pilot opened, so every ramp gate had a defensible comparison point.

The pilot produced two early findings that would have been invisible at faster ramp speeds. First, the model-scored conversation CSAT diverged from resolution CSAT inside the policy category — the model was rating its own confidence as high while customers were rating the resolution as middling. The signal pointed to a confidence-calibration issue rather than a quality issue, and surfaced two weeks earlier than either survey-based layer would have caught it. That kind of leading signal is the entire reason model-scored CSAT exists inside the instrumentation stack.

Second, the false-positive escalation rate in the account-and-access category landed above expectations — the model was handing off cleanly resolvable password-reset conversations because the confidence floor for the category had been set conservatively at launch. The floor was lowered on a per-archetype basis at the month-one review and the category's deflection rate climbed substantially in month two without CSAT damage. Both adjustments were the gates doing their job — surfacing tuning decisions inside a window small enough to make them safely.

The escalation rubric was the document with the highest operational consequence in the engagement. For each of the top-200 archetypes, the rubric defined a confidence floor (below which the AI handed off automatically), a never-deflect flag (always escalates, regardless of confidence), a context payload (transcript, intent classification, confidence score, relevant customer state, retrieved policy chunks), and a routing queue. Sign-off came from the support lead, the legal lead for the regulated categories, and the product owner before the pilot opened.

The single most consequential design decision inside the rubric was the breadth of the never-deflect list. We pushed for a generous launch posture — dispute initiation, fraud reports, churn-saves, account-compromise reports, anything regulated, and any conversation where the customer used a specific set of trigger phrases ("legal", "regulator", "cancel", "close my account"). The cost of an AI deflection on the wrong archetype was much higher than the value of dozens of correctly-deflected tier-one tickets, and the never-deflect list was the mechanism that priced that asymmetry into the program.

The handoff context payload was the second load-bearing piece of the rubric. When an escalation fired, the receiving agent received the full conversation transcript, the model's stated intent, the confidence score, the customer's relevant account state, and any policy chunks the model had retrieved during the conversation. Agents were trained on a thirty-second confirmation pattern — the agent confirms the AI summary inside the first thirty seconds of the live conversation, so the customer experiences continuity rather than a context restart.

That thirty-second confirmation was, in our analysis of the month-three data, the single largest driver of escalation CSAT. Conversations where the agent confirmed the AI summary inside the window scored substantially higher on resolution CSAT than conversations where the agent opened with "can you tell me what this is about". The number is not surprising in retrospect — customers experience the handoff as a continuity test, not as a feature — but the discipline of measuring it, training to it, and QAing it explicitly is what made the number show up in the program rather than as an after-the-fact regret.

A post-handoff QA loop sampled escalations weekly, flagging any case where the customer had to repeat themselves. Flags drove both rubric refinement and individual agent re-training. The pattern that emerged across the engagement is that context loss is, by a wide margin, the single largest source of CSAT damage in support-AI deployments — worse than the bot being wrong, because the customer experiences wrong-and-frustrating rather than wrong-and-escalated.

The headline outcomes ran against a five-month timeline: month one, audit and build, no customer-facing traffic; month two, 8% deflection with the pilot opened to 1% then ramped to 8% mid-month; month three, 22% deflection with the ramp gate opened to 25%; month four, 38% deflection at the same gate with the model and the rubric tuning settling; month five, 51% steady state. CSAT held inside a ±1.5-point band against baseline across all three measurement tiers throughout the engagement. Hallucinated-policy rate landed under 0.4% by month three and held there.

The ramp shape is the part of the chart worth reading twice. Deflection at month two was 8%, not 25% or the originally-promised 40% — the gates were holding the program at the level the archetype mix and the CSAT trend could actually support, rather than the level the ramp plan promised on slide eleven of the kick-off deck. Month three climbed substantially because the audit work paid off, the confidence thresholds had been tuned against real traffic, and the escalation rubric had been refined twice. Month four and five were the program reaching its CSAT-protected ceiling and settling.

The CSAT story is the more important half of the outcome. Across the engagement, resolution CSAT moved by 1.2 points (positive) against baseline, delayed CSAT moved by 0.8 points (positive), and model-scored conversation CSAT held essentially flat. None of the three drifted outside the ±2-point gating band at any point during the ramp, which meant no rollback event fired. Hallucinated-policy rate spiked briefly to 0.7% in week six of the pilot — a retrieval issue caused by a policy update that had not propagated to the index — and was resolved inside the same day; the rate sat under 0.4% from month three onward.

Unit-economics outcomes were consistent with the deflection ramp. The cost-per-resolved-ticket curve fell substantially once month-five steady-state held, while the function avoided the hiring path that had triggered the engagement in the first place. The agent team did not shrink — it shifted upmarket, with the highest-complexity disputes, fraud reports, and churn-saves now consuming a larger share of human attention. That is the right destination for the human side of the function.

Most of the engagement is replicable. The knowledge audit shape, the four-category split, the category-aware RAG build, the CSAT-controlled pilot structure, the escalation rubric template, and the gating discipline are vertical-independent patterns. We have shipped variants of all of these inside ecommerce, SaaS, and B2B service contexts. The fintech-specific adaptations — hallucinated-policy rate as a first-class compliance metric, policy-version metadata on retrieval chunks, generous never-deflect lists for regulated trigger phrases — are the parts that need to be re-thought for a different vertical, but the rest of the program transfers.

What did not replicate from the original brief is the sixty-percent deflection target. The lower number we delivered is not a shortfall — it is the gates working. Any team committing to a deflection ceiling before measuring the archetype mix and the CSAT response of their own customer base is committing to a number that was pulled from somebody else's slide deck. The right target is the highest deflection the CSAT band will hold, instrumented and gated, on this customer base, for this archetype mix.

For teams designing a comparable program, the sequencing matters as much as the components. Knowledge audit before anything else, RAG build against the audited corpus rather than the raw documentation set, CSAT instrumentation wired before the pilot opens, escalation rubric signed off by support and legal before any traffic touches the AI, and gates that fire on measurement rather than calendar. Skipping any of these is the most common way successful month-five programs become disappointing month-twelve programs.

The companion piece on the 90-day launch plan walks through the operational cadence in more detail, including the templates we ship with engagements. The companion piece on the support ROI calculator and deflection formula covers the unit-economics math underneath a program like this one — break-even thresholds, cost-per-ticket ladders, and the vendor comparison work that informs build-versus-buy.

For teams that want the program delivered as a managed engagement rather than run internally, our AI transformation engagements ship the knowledge audit, the RAG build, the CSAT instrumentation, the pilot, and the agent handoff training as a phased program — calibrated to the archetype mix of the specific support function and with measurable CSAT-controlled outcomes throughout.