Chunking strategies, hybrid search architectures, and evaluation frameworks β the three levers that separate production-grade RAG from demo-ware.
Core Concept
RAG: The Memory Prosthetic for LLMs
Imagine hiring the world's smartest analyst β photographic memory, brilliant reasoning β but their knowledge stopped updating two years ago. That's an LLM in isolation. RAG (Retrieval-Augmented Generation) is the system that lets this analyst pull fresh documents from a filing cabinet before answering. The filing cabinet is a vector database. The act of pulling is retrieval. The analyst synthesizing the answer is generation.
But here's what most introductions skip: the quality of what goes into the filing cabinet β how documents are cut up, indexed, and ranked β determines 80% of the final answer quality. The LLM can only be as good as what you hand it.
π
The RAG Pipeline β All Five Stages
Every RAG system has the same fundamental anatomy. Where teams differ is in how they implement each stage.
βοΈ
Under the Hood: How Retrieval Actually Works
When a user asks a question, the system embeds the query into the same vector space as your document chunks. It then runs Approximate Nearest Neighbor (ANN) search β algorithms like HNSW (Hierarchical Navigable Small World graphs) that find the closest vectors in milliseconds across millions of chunks.
The math: given query embedding q and chunk embeddings {cβ, cβ, β¦, cβ}, retrieve the top-k chunks maximizing cos_sim(q, cα΅’) = (q Β· cα΅’) / (|q| Γ |cα΅’|). These chunks are inserted into the LLM's prompt as context. The LLM generates an answer grounded in that context.
The embedding model is the hidden variable most teams underestimate. A mismatch between how your query space and document space are embedded β even with a great retrieval algorithm β tanks recall. Domain-specific fine-tuned embedders (e.g., BGE, E5-mistral) consistently outperform generic OpenAI embeddings on specialized corpora.
Chunking Strategies
How You Slice the Document Changes Everything
Chunking is the unglamorous step that engineers spend 40% of their RAG debugging time on. A chunk that's too large buries the relevant sentence in noise. A chunk too small loses the context needed to interpret it. The right strategy depends on your document type and query pattern.
Fixed-Size
512 tokens, 10% overlap
Semantic
split on topic shifts
Hierarchical
parent + child chunks
Late Chunking
embed-then-split
Split every N tokens with overlap. Simple, fast, predictable. The token window overlap (e.g., 50β100 tokens) prevents sentences from being cut mid-thought.
Best for: Homogeneous documents (contracts, forms). Fast to build.
β οΈ Fails when semantic units span chunk boundaries β retrieves half an argument.
Detect topic boundaries using embedding cosine distance between consecutive sentences. Split when distance exceeds a threshold. Chunks reflect logical units, not token counts.
Best for: Long-form articles, research papers, mixed-topic docs.
β οΈ Slower at index time. Chunk sizes are variable, harder to predict context usage.
Index small chunks (for precise retrieval), but retrieve the parent chunk (for full context). The small chunk finds the needle; the parent chunk gives the haystack back to the LLM.
Best for: Dense technical documentation where precision matters but context is needed for generation.
β οΈ Doubles storage. Requires parent-child linkage in your vector store.
Embed the entire document first (preserving full cross-sentence attention), then pool token embeddings into chunk-level vectors. Introduced by Jina AI in 2024. Chunks carry document-level context in their vectors.
Best for: Documents with heavy cross-reference (legal, medical, long-context knowledge bases).
β οΈ Requires long-context embedding model. Not yet universally supported.
π³
RAPTOR: The Tree-Based Alternative to Flat Chunking
RAPTOR (see Papers section) solves a fundamental flaw in flat chunking: no chunk contains cross-document synthesis. A query like "What are the three main themes across all Q4 earnings calls?" will never be answered well by retrieving individual chunks β no single chunk contains that synthesis.
RAPTOR builds a multi-level tree of summaries. Leaf nodes = original chunks. Each level up is a cluster-summary of the level below. At query time, retrieval can happen at any level β fetch a high-level summary for broad questions, drill into leaves for specific facts.
The clustering algorithm used in RAPTOR is Gaussian Mixture Models (GMM) applied to UMAP-reduced embeddings. The insight is that documents that cluster together in embedding space share latent themes β so summarizing each cluster gives you an emergent topic model for free.
Hybrid Search
Semantic + Keyword: Better Together
There are two fundamentally different ways to find a relevant chunk. Dense (semantic) retrieval embeds both query and chunk into vector space β it understands "cardiac arrest" and "heart attack" are the same thing. Sparse (keyword) retrieval uses BM25 β it's extremely precise when you search for a product code like SKU-X7J-2291 that embeddings would never cluster correctly.
Reciprocal Rank Fusion (RRF) β The Standard Hybrid Formula
RRF merges ranked lists from dense and sparse retrievers without needing to normalize scores (which is hard since BM25 scores and cosine similarities live on different scales):
RRF_score(d) = Ξ£ 1 / (k + rank_i(d)) where k=60 is a constant
A document that ranks 3rd in dense and 5th in sparse gets score 1/(60+3) + 1/(60+5) = 0.0301. Documents that appear in both lists jump dramatically. k=60 was empirically found to reduce sensitivity to outlier rankings.
π‘
Sparse vs. Dense β When Each Wins
Dense wins when the query uses different vocabulary than the document ("How do I reset my password?" matching "Account recovery procedure"). This is the classic vocabulary mismatch problem that keyword search fails on.
Sparse (BM25) wins when the query contains unique identifiers, proper nouns, product codes, or technical terms where exact match matters ("GPT-4o API latency SLA" β no synonym expansion wanted).
Production rule: implement hybrid search by default. The cost of running BM25 alongside vector search is ~5β10ms latency overhead with near-zero infrastructure cost. The recall gains on diverse query distributions are consistently 5β15 percentage points above dense-only. Skip hybrid only if your entire corpus is homogeneous and your queries never contain proper nouns.
HyDE: Flip the Query Problem
Standard retrieval embeds the query and looks for similar chunks. HyDE (Hypothetical Document Embeddings) asks: what if instead of embedding the question, we generate a hypothetical ideal answer and embed that?
The logic: a hypothetical answer lives in the same embedding neighborhood as real answers in your corpus. Queries (short, interrogative) and answers (long, declarative) have very different embedding geometries. HyDE bridges this gap β particularly powerful for zero-shot retrieval where there's no training data to fine-tune the retriever.
π
HyDE in Practice
Step 1: User sends query: "What are the side effects of metformin?"
Step 2: LLM generates a hypothetical answer (may be partially hallucinated, that's OK): "Metformin commonly causes gastrointestinal side effects including nausea, diarrhea, and abdominal discomfort. Rare but serious side effects include lactic acidosisβ¦"
Step 3: Embed the hypothetical answer (not the original query). Retrieve chunks similar to that embedding.
Step 4: Grounded generation uses retrieved real chunks β the hallucination in Step 2 never reaches the final answer.
HyDE adds one LLM call per query (cost ~$0.001 for small models). The retrieval quality improvement is usually 8β20% on BEIR benchmark, but gains are uneven β large on abstract/scientific queries, small on factual lookups.
Papers to Know
Three Papers That Defined the Field
ICLR 2024
arXiv 2401.18059
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning Β· Stanford NLP
Standard RAG retrieves only short, flat chunks β it cannot answer questions that require synthesizing information across an entire document or corpus. RAPTOR builds a hierarchical tree of summaries by recursively clustering chunks (via GMM on UMAP embeddings) and summarizing each cluster. At query time, the system retrieves from any level of the tree, enabling multi-granularity synthesis. On the QuALITY benchmark, RAPTOR + GPT-4 achieved a 20-point improvement over prior state-of-the-art.
Why it matters: RAPTOR shifted thinking from "how do I split documents?" to "how do I build an index that understands document structure at multiple levels of abstraction?" Every modern RAG framework (LangChain, LlamaIndex) now has a RAPTOR-inspired hierarchical indexing option.
arxiv.org/abs/2401.18059
ACL 2023
arXiv 2212.10496
Precise Zero-Shot Dense Retrieval without Relevance Labels
Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan Β· CMU, University of Waterloo
Dense retrievers typically require expensive labeled training data (query, relevant doc pairs) for each domain. HyDE eliminates this by using an instruction-following LLM to generate a hypothetical answer to the query, then embedding that answer instead of the query itself. The hypothetical answer β even if factually imperfect β lives in the same embedding neighborhood as real relevant documents. HyDE achieved strong performance across web search, QA, and fact-verification tasks, and generalizes across 11 languages without any language-specific training.
Why it matters: HyDE made zero-shot dense retrieval practical. Before HyDE, building a new domain-specific RAG system required weeks of annotation. HyDE removed that bottleneck, enabling rapid RAG deployment in new verticals.
arxiv.org/abs/2212.10496
EACL 2024 Demo
arXiv 2309.15217
Ragas: Automated Evaluation of Retrieval Augmented Generation
Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert Β· Cardiff University, Amazon
Evaluating RAG systems requires checking three things independently: did we retrieve the right context? Is the generated answer faithful to that context? Does the answer actually address the question? RAGAS operationalizes these as three automated LLM-as-judge metrics β Context Relevance, Faithfulness, and Answer Relevance β requiring no ground-truth human annotations. The framework chains LLM calls to decompose answers into claims, verify each claim against context, and score the pipeline holistically.
Why it matters: Before RAGAS, teams evaluated RAG by manually inspecting outputs or using blunt end-to-end metrics like ROUGE (which measure surface overlap, not faithfulness). RAGAS gave teams a systematic, automated way to find which stage of their pipeline was failing β retrieval, or generation.
arxiv.org/abs/2309.15217
Community Pulse
What the Field Is Saying Right Now
Liu has argued publicly that naive RAG is "a hack" β a stopgap before models and retrieval systems co-evolve into something more principled. His view is that the field is moving toward "agentic retrieval" where the retrieval strategy itself is planned by an LLM, not hardcoded by an engineer.
Source context: Latent Space podcast ("RAG Is A Hack") + LlamaIndex blog post "RAG Is Dead, Long Live Agentic Retrieval" β latent.space/p/llamaindex and llamaindex.ai/blog/rag-is-dead-long-live-agentic-retrieval
Karpathy has described LLMs as a new kind of operating system where the context window is RAM β and "context engineering" as the key skill: the deliberate, careful craft of deciding what information to fill the context window with at each step. This framing reframes RAG not as "retrieval + generation" but as a discipline of context management.
Paraphrased position from his public writing and talks, circa 2025. No single source URL for this specific formulation.
Willison has written extensively about RAG on his blog, focusing on practical deployment β particularly around security risks (prompt injection via retrieved content) and the challenge of knowing when not to retrieve. He's argued that the retrieval step introduces an attack surface that most RAG implementations ignore entirely.
Paraphrased from simonwillison.net/tags/rag/ β no single recent post, but a consistent theme across his 2024β2025 writing.
Platform Deep-Dive
How Each Major Platform Implements RAG
| Platform |
Chunking Default |
Retrieval Method |
Eval Built-in |
Notable RAG Feature |
| Claude (Anthropic) |
200K token context window β often bypasses chunking entirely for smaller corpora |
Native files API + vector stores (beta) |
No native RAG eval; pairs with RAGAS |
Citation-grounded generation: can return source references with each claim |
| OpenClaw |
SKILL.md-based context injection; skills define their own retrieval logic |
Hub-and-spoke: central agent calls retrieval sub-skill |
None built-in |
Plugin ecosystem allows swapping retrieval backends (Pinecone, Weaviate, etc.) via MCP skills |
| GPT / OpenAI |
Assistants API auto-chunks at 800 tokens, 400 overlap |
Hybrid (semantic + keyword) in File Search tool |
Evals API (Mar 2025) supports RAG-specific metrics |
File Search returns chunk-level citations with relevance scores; can be inspected by developers |
| Gemini (Google) |
2M token context; Vertex AI Agent Builder handles chunking for RAG Search |
Google-native Enterprise Search + vector via Matching Engine |
Vertex AI RAG Engine includes built-in grounding metrics |
Grounding with Google Search: real-time retrieval from live web, not just static corpus |
π
The Long-Context vs. RAG Debate
As context windows reach 1M+ tokens, a reasonable question emerges: why retrieve at all? Just stuff everything in the context. The answer is multi-dimensional:
Cost: A 1M-token context costs ~$15 per call with Claude 3.5 Sonnet. A well-tuned RAG pipeline retrieving 5K tokens of relevant context costs ~$0.02. For high-frequency queries, RAG wins by 3 orders of magnitude.
Needle-in-haystack degradation: LLM attention degrades on very long contexts β the "lost-in-the-middle" effect means facts buried in the middle of a 1M-token context are recalled less reliably than facts at the beginning or end.
Dynamic corpora: If your knowledge base updates daily, you can't pre-stuff it into a prompt. RAG handles living data; long-context doesn't.
Emerging hybrid: "RAG to reduce, then long-context to reason." Retrieve the relevant 20K tokens from a 10M-token corpus, then let the LLM reason carefully over that retrieved set. Best of both worlds.
Evaluation Metrics
How to Know If Your RAG Actually Works
Most teams evaluate RAG by asking "does it give good answers?" β a subjective, non-scalable approach. Production RAG requires decomposed, automated metrics that pinpoint which stage is failing.
Context Relevance
CR
Are the retrieved chunks actually relevant to the query? Low CR = retrieval problem.
Faithfulness
F
Is the answer grounded in retrieved context? Low F = LLM hallucinating beyond context.
Answer Relevance
AR
Does the answer address what the user asked? Low AR = complete but off-target answers.
Recall@K
R@K
Of all relevant chunks in the corpus, what fraction appear in top-K results? Requires labeled ground truth.
π¬
RAGAS vs. ARES β The Two Automated Eval Frameworks
RAGAS (EACL 2024): Uses LLM-as-judge calls to score CR, Faithfulness, and AR without ground truth annotations. Widely adopted; integrates with LangSmith, LlamaIndex, and most observability platforms. Limitation: metric quality depends heavily on the judge LLM's capabilities.
ARES (arXiv 2311.09476): Trains task-specific LLM judges using a small labeled dataset (as few as 150 examples), then uses prediction-powered inference (PPI) to produce statistically valid confidence intervals on scores. Outperforms RAGAS by 59.3pp on context relevance and 14.4pp on answer relevance accuracy in head-to-head evaluation.
The practical tradeoff: RAGAS is zero-shot and immediately deployable. ARES requires annotation investment but gives you calibrated confidence intervals β essential when you're making go/no-go product decisions based on eval numbers.
Expert Questions
Questions That Signal Deep Understanding
Q1
Your RAGAS faithfulness score is 0.95 but users keep complaining that answers are wrong. What's the most likely explanation, and what metric would you add to your eval suite to diagnose it?
Q2
You're building RAG over a corpus of 100,000 technical support tickets. A new engineer suggests using fixed 512-token chunks. What specific failure modes would you predict, and what chunking strategy would you recommend instead, and why?
Q3
Walk me through exactly why HyDE improves retrieval quality specifically on scientific literature queries, and under what conditions it would make retrieval worse.
Q4
Your hybrid search system uses RRF with k=60 to merge dense and BM25 rankings. A PM asks you to "weight semantic search more than keyword search." What's the technically correct way to implement this, and what's the risk of doing it naively?
Q5
As context windows reach 2M tokens, your CTO argues you should drop RAG and just stuff the entire knowledge base into context. What is the strongest business case to keep RAG, and under what specific scenario would your CTO actually be right?