RAG for Business: AI That Knows Your Company Data

Why RAG Is the Enterprise AI Killer App

The enterprise AI market has been searching for its core use case since ChatGPT launched in late 2022. After two years of experimentation with chatbots, copilots, and automated content generation, one pattern has emerged as the most consistently successful: connecting LLMs to proprietary company data through retrieval-augmented generation. A 2026 Gartner survey found that 67% of Fortune 500 companies have either deployed or are actively building RAG systems, making it the most widely adopted enterprise AI architecture.

The business case for RAG is straightforward. Knowledge workers spend an average of 9.3 hours per week searching for information across internal systems, according to McKinsey research. A well-implemented RAG system reduces this to seconds by providing a single interface that searches across all document repositories, databases, and knowledge bases simultaneously. For a 500-person company, this translates to roughly 240,000 hours per year recovered from information searching, at an average fully loaded cost of $75 per hour, that represents $18 million in annual productivity gains.

Beyond productivity, RAG addresses the institutional knowledge problem that plagues every organization. When senior employees leave, their knowledge leaves with them. RAG systems capture this knowledge in a queryable format that makes every employee as informed as the most experienced person on the team. This is particularly valuable in industries with high turnover, complex regulatory requirements, or rapid product evolution where keeping everyone current is a constant challenge. Building these systems is a core part of enterprise AI transformation strategies.

RAG Architecture: How It Works

A RAG system consists of three distinct pipelines that work together: the ingestion pipeline that processes and stores your documents, the retrieval pipeline that finds relevant documents for each query, and the generation pipeline that produces answers using retrieved context. Understanding each pipeline is essential for building a system that performs well in production.

The quality of a RAG system is determined primarily by retrieval quality, not generation quality. If the retrieval pipeline returns irrelevant or incomplete context, even the most capable LLM will produce poor answers. Conversely, if retrieval returns the right documents, even a moderately capable model will generate accurate responses. This is why the majority of engineering effort in RAG systems should focus on chunking strategy, embedding model selection, and retrieval optimization rather than prompt engineering for the generation step.

// 1. Ingestion (run once per document)
async function ingestDocument(doc) {
  const text = await parseDocument(doc.path)
  const chunks = semanticChunk(text, {
    maxTokens: 512,
    overlap: 50,
    splitOn: ["heading", "paragraph"]
  })

  for (const chunk of chunks) {
    const embedding = await embed(chunk.text)
    await vectorDB.upsert({
      id: chunk.id,
      vector: embedding,
      metadata: {
        source: doc.path,
        title: doc.title,
        section: chunk.heading,
        updatedAt: doc.modifiedDate
      },
      text: chunk.text
    })
  }
}

// 2. Query (run on every user question)
async function queryRAG(userQuestion) {
  const queryEmbedding = await embed(userQuestion)

  const results = await vectorDB.query({
    vector: queryEmbedding,
    topK: 5,
    filter: { access: currentUser.role }
  })

  const context = results
    .map(r => r.text)
    .join("\n\n---\n\n")

  const answer = await llm.generate({
    system: "Answer based only on the provided context. Cite sources.",
    messages: [
      { role: "user", content: `Context:\n${context}\n\nQuestion: ${userQuestion}` }
    ]
  })

  return { answer, sources: results.map(r => r.metadata) }
}

This basic pipeline covers the core flow, but production systems add several additional components: query rewriting to improve retrieval for ambiguous questions, hybrid search combining vector similarity with keyword matching, reranking to improve precision of retrieved results, guardrails to prevent prompt injection, and caching to reduce latency and costs for repeated queries. Each of these components is covered in the sections that follow.

Vector Database Selection

The vector database is where your document embeddings live and where similarity search happens at query time. Choosing the right one depends on your scale requirements, latency targets, existing infrastructure, and team expertise. The market has matured significantly since early 2024, and the leading options each occupy a distinct niche.

One commonly overlooked factor is hybrid search capability. Pure vector search works well for semantic queries ("how do I handle customer complaints?") but struggles with exact-match queries ("what is policy #42-B?") or queries containing specific product names, SKUs, or technical identifiers. Hybrid search combines vector similarity with traditional keyword matching (BM25) to handle both query types. Weaviate includes this natively. Pinecone added sparse-dense hybrid search in 2025. For pgvector, you can combine vector search with PostgreSQL's full-text search using a reciprocal rank fusion (RRF) approach.

The scaling characteristics also differ significantly. pgvector handles up to approximately 5 million vectors well on a single node, but performance degrades beyond that without partitioning. Pinecone and Qdrant handle billions of vectors through automatic sharding. Weaviate scales horizontally but requires more configuration. For most business applications starting with fewer than 1 million documents, any of these options will work. Choose based on your existing infrastructure and team skills rather than hypothetical future scale.

Document Chunking Strategies That Actually Work

Chunking is the process of splitting documents into smaller segments for embedding and retrieval. It is the single most impactful engineering decision in a RAG system, yet it receives the least attention in most tutorials. Poor chunking produces chunks that are either too small (losing context) or too large (diluting relevance), directly degrading retrieval quality regardless of how good your embedding model or vector database is.

// Recursive text splitter with heading-aware boundaries
const splitter = new RecursiveTextSplitter({
  // Split hierarchy: headings > paragraphs > sentences > words
  separators: [
    "\n## ",     // H2 headings (primary split)
    "\n### ",    // H3 headings
    "\n\n",     // Paragraph breaks
    "\n",        // Line breaks
    ". ",         // Sentence boundaries (last resort)
  ],
  chunkSize: 512,       // Target tokens per chunk
  chunkOverlap: 50,     // ~10% overlap for context continuity
  lengthFunction: countTokens,
})

// Add parent document context to each chunk
const chunks = splitter.splitDocuments(documents)
for (const chunk of chunks) {
  // Prepend section hierarchy for retrieval context
  chunk.metadata.contextPrefix =
    `Document: ${chunk.metadata.title} > Section: ${chunk.metadata.heading}`
}

The optimal chunk size depends on your content type and query patterns. For technical documentation with specific, factual queries, smaller chunks (256-384 tokens) work best because they contain focused information that matches precise questions. For strategic documents where users ask broad questions requiring synthesized context, larger chunks (512-768 tokens) perform better because they preserve more surrounding context. Run retrieval evaluations at multiple chunk sizes with your actual queries to find the optimal setting for your use case.

A technique that significantly improves retrieval quality is contextual chunking: prepending each chunk with its parent document title and section heading. When a chunk contains "The deadline is 30 days", that is meaningless without knowing it comes from "Employee Handbook > Leave Policy > Vacation Requests." Adding this hierarchy as a metadata prefix helps the embedding model understand the chunk's context and improves both retrieval accuracy and the LLM's ability to generate contextually appropriate answers.

Embedding Model Comparison and Selection

The embedding model converts text into high-dimensional vectors that capture semantic meaning. Two pieces of text with similar meaning produce vectors that are close together in vector space, enabling similarity search. The choice of embedding model affects retrieval quality, latency, cost, and vector storage requirements.

Dimensionality directly affects storage costs and query speed. Higher-dimensional embeddings capture more nuance but require more storage and slower search. OpenAI's text-embedding-3 models support Matryoshka dimensionality reduction, allowing you to truncate 3,072-dimensional vectors to 256 or 512 dimensions with minimal quality loss. This is particularly useful when you need to balance quality against storage costs for large document collections.

One critical constraint: you must use the same embedding model for ingestion and querying. Vectors from different models are not comparable because they occupy different vector spaces. If you switch embedding models, you must re-embed your entire document collection. This makes the initial model choice important, but do not let it paralyze you. The difference between the top models is typically 2-5% on retrieval benchmarks, which matters less than getting your chunking strategy and retrieval pipeline right.

import { openai } from "@ai-sdk/openai"

// Full dimensions (3,072) — maximum quality
const fullEmbedding = await openai.embedding(
  "text-embedding-3-large"
).doEmbed({ values: [text] })

// Reduced dimensions (512) — 80% quality, 83% less storage
const reducedEmbedding = await openai.embedding(
  "text-embedding-3-large",
  { dimensions: 512 }
).doEmbed({ values: [text] })

// Batch embedding for efficiency
const batchEmbeddings = await openai.embedding(
  "text-embedding-3-small"
).doEmbed({
  values: chunks.map(c => c.text)
})
// Each embedding is 1,536 dimensions

Evaluation Frameworks: Measuring RAG Quality

The most dangerous RAG system is one that looks like it works. Without systematic evaluation, you cannot distinguish between a system that produces accurate answers 95% of the time and one that produces plausible-sounding but wrong answers 30% of the time. Both feel the same during demo day. The difference becomes apparent when users start making decisions based on the answers.

Building an evaluation dataset is the critical first step. Create a set of 50-100 question-answer pairs that cover your most important use cases. For each question, identify the specific documents that contain the answer (ground truth). These golden examples become your regression test suite — every time you change chunking parameters, switch embedding models, or modify retrieval logic, run the evaluation suite to verify that quality improved or at least did not degrade.

// Evaluation dataset structure
const evalDataset = [
  {
    question: "What is our refund policy for enterprise clients?",
    groundTruth: "Enterprise clients receive full refunds within 90 days...",
    relevantDocIds: ["policy-doc-42", "enterprise-terms-v3"],
  },
  // ... 50-100 more examples
]

// Run evaluation
async function evaluateRAG(dataset) {
  const results = []

  for (const example of dataset) {
    const { answer, sources } = await queryRAG(example.question)

    results.push({
      question: example.question,
      faithfulness: await scoreFaithfulness(answer, sources),
      relevancy: await scoreRelevancy(answer, example.question),
      contextPrecision: scoreContextPrecision(
        sources.map(s => s.id),
        example.relevantDocIds
      ),
      contextRecall: scoreContextRecall(
        sources.map(s => s.id),
        example.relevantDocIds
      ),
    })
  }

  return {
    avgFaithfulness: avg(results.map(r => r.faithfulness)),
    avgRelevancy: avg(results.map(r => r.relevancy)),
    avgPrecision: avg(results.map(r => r.contextPrecision)),
    avgRecall: avg(results.map(r => r.contextRecall)),
  }
}

Beyond automated metrics, implement a feedback loop from users. Add thumbs-up/thumbs-down buttons on every RAG response. Track which queries produce negative feedback and use those as additional evaluation examples. Over time, your evaluation dataset grows to cover edge cases and failure modes that you would not have anticipated during initial development. This continuous improvement cycle is what separates production-grade RAG systems from demos that break under real-world usage.

Production Deployment Patterns and Scaling

Moving from a RAG prototype to a production system requires addressing several concerns that do not exist in development: latency optimization, cost management, reliability, observability, and security. The patterns below represent current best practices from teams running RAG systems serving thousands of queries per day.

async function productionRAGQuery(query, user) {
  // 1. Input validation and sanitization
  const sanitized = sanitizeInput(query)
  if (detectPromptInjection(sanitized)) {
    return { error: "Query rejected by safety filter" }
  }

  // 2. Check semantic cache
  const cached = await semanticCache.get(sanitized, {
    similarityThreshold: 0.95,
    maxAge: "1h"
  })
  if (cached) return cached

  // 3. Retrieve with access control
  const results = await vectorDB.query({
    vector: await embed(sanitized),
    topK: 10,
    filter: { accessLevel: { $in: user.roles } }
  })

  // 4. Rerank for precision
  const reranked = await reranker.rank(sanitized, results, { topK: 5 })

  // 5. Route to appropriate model based on complexity
  const model = classifyComplexity(sanitized) === "simple"
    ? "claude-haiku-4-5-20251001"
    : "claude-sonnet-4-6"

  // 6. Generate with citation tracking
  const response = await generateWithCitations({
    model,
    context: reranked,
    query: sanitized,
    systemPrompt: RAG_SYSTEM_PROMPT
  })

  // 7. Cache and log
  await semanticCache.set(sanitized, response)
  await logQuery({ query, user: user.id, sources: reranked, response })

  return response
}

Observability is non-negotiable for production RAG. Log every query, every set of retrieved documents, and every generated response. Track retrieval latency, LLM latency, and end-to-end latency separately so you can identify bottlenecks. Monitor cache hit rates, feedback scores, and document freshness. Set alerts for anomalies: if retrieval latency spikes, if feedback scores drop, or if the same query repeatedly receives negative feedback. These signals let you catch and fix quality degradation before it affects user trust.

Security requires attention at every layer. Validate and sanitize all user inputs to prevent prompt injection attacks that could cause the LLM to ignore its instructions and reveal sensitive context. Implement document-level access controls in retrieval to prevent information disclosure across user roles. Rate limit queries per user to prevent abuse. Audit log all queries for compliance purposes. For regulated industries, consider running inference in your own cloud VPC rather than using third-party APIs. Working with experienced development teams who understand both AI systems and security best practices is essential for production RAG deployments.

RAG vs Fine-Tuning: When to Use Each

The RAG vs fine-tuning debate is one of the most common questions in enterprise AI. The answer is not either-or — they solve different problems and are often complementary. Understanding when each approach is appropriate prevents wasted effort on the wrong solution.

The most effective approach for many organizations is RAG with a fine-tuned generation model. Fine-tune a smaller model to match your desired output style and domain terminology, then use RAG to provide it with current, factual context. This gives you the style consistency of fine-tuning with the factual grounding and auditability of RAG. The fine-tuned model costs less per query than a frontier model, and the RAG layer ensures it generates accurate, sourced answers rather than hallucinating.

For most businesses starting their AI journey, RAG is the right first step. It requires no ML expertise, works with any LLM provider, handles changing data naturally, and provides the source citations that compliance and legal teams require. Build RAG first, validate the use case, and consider fine-tuning only if evaluation reveals that the generation model's style or reasoning quality is the bottleneck — not the retrieval quality. This is the approach we recommend in our AI transformation engagements.

Frequently Asked Questions