Retrieval-Augmented Generation (RAG) Architecture: Chunking, Embeddings, Hybrid Search & Reranking
A comprehensive engineering guide to building production-grade RAG pipelines: document chunking strategies, vector embeddings with pgvector, hybrid BM25 search, and cross-encoder reranking.
Why Naive RAG Fails in Production#
Retrieval-Augmented Generation (RAG) is the foundational architecture for connecting Large Language Models (LLMs) to private, dynamic, or enterprise knowledge bases. While creating a prototype RAG script in Python takes less than twenty lines using standard tutorials, naive vector search frequently degrades in production.
Common failure modes of naive RAG include:
1. Context Fragmentation: Fixed-size chunking splits critical sentences in half, causing vector similarity to miss the semantic context.
2. Keyword Blindness: Dense embeddings struggle with exact identifier lookups (e.g., error codes like ERR_CONN_TIMEOUT_409, product SKUs, or function names).
3. Retrieval Noise: The top 5 cosine similarity chunks contain irrelevant text that clutters the model's prompt context, leading to hallucinations.
This guide details the end-to-end architecture of an enterprise-grade RAG pipeline designed to solve these failure points.
1. Advanced Document Chunking Strategies#
The chunking stage determines the semantic boundaries of your retrieved vectors. Rather than arbitrary character-based splits, production systems use structured chunking strategies:
A. Recursive Character Chunking with Overlap Splits on semantic separators in priority order (`\n\n`, `\n`, `. `, ` `):
interface ChunkConfig {
chunkSize: number; // e.g., 600 tokens
chunkOverlap: number; // e.g., 100 tokens
}
export function recursiveSplit(text: string, maxTokens: number = 500, overlap: number = 80): string[] {
const paragraphs = text.split(/\n\n+/);
const chunks: string[] = [];
let currentChunk = "";
for (const para of paragraphs) {
if ((currentChunk + para).length > maxTokens && currentChunk.length > 0) {
chunks.push(currentChunk.trim());
// Retain sliding window overlap for context continuity
currentChunk = currentChunk.slice(-overlap) + "\n\n" + para;
} else {
currentChunk += (currentChunk ? "\n\n" : "") + para;
}
}
if (currentChunk.trim()) {
chunks.push(currentChunk.trim());
}
return chunks;
}B. Parent-Document / Hierarchical Chunking Store small chunks (e.g., 150 tokens) for accurate vector index matching, but retrieve the entire parent section (e.g., 800 tokens) to provide the LLM with complete contextual understanding.
2. Generating High-Dimensional Vector Embeddings#
Convert processed text chunks into dense floating-point vector representations using specialized embedding models (such as Google text-embedding-004 or OpenAI text-embedding-3-small):
import { GoogleGenAI } from "@google/genai";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
export async function generateEmbedding(text: string): Promise<number[]> {
const response = await ai.models.embedContent({
model: "text-embedding-004",
contents: text,
});
if (!response.embedding?.values) {
throw new Error("Failed to generate vector embedding");
}
return response.embedding.values; // Returns 768-dimensional float vector
}3. Hybrid Search: Combining Dense Vectors with Sparse Keyword Matching (BM25)#
Dense vectors excel at semantic similarity ("find documents about authentication timeouts"), but perform poorly on exact keyword lookups ("find error code 0x80040154").
Hybrid search merges dense vector cosine similarity with sparse full-text search (BM25 or PostgreSQL tsvector) using Reciprocal Rank Fusion (RRF):
-- PostgreSQL with pgvector: Hybrid Search Query
WITH semantic_search AS (
SELECT id, content,
ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) as rank
FROM document_chunks
ORDER BY embedding <=> $1::vector
LIMIT 20
),
keyword_search AS (
SELECT id, content,
ROW_NUMBER() OVER (ORDER BY ts_rank_cd(search_vector, plainto_tsquery('english', $2)) DESC) as rank
FROM document_chunks
WHERE search_vector @@ plainto_tsquery('english', $2)
LIMIT 20
)
SELECT
COALESCE(s.id, k.id) as id,
COALESCE(s.content, k.content) as content,
(COALESCE(1.0 / (60 + s.rank), 0.0) + COALESCE(1.0 / (60 + k.rank), 0.0)) as rrf_score
FROM semantic_search s
FULL OUTER JOIN keyword_search k ON s.id = k.id
ORDER BY rrf_score DESC
LIMIT 10;4. Cross-Encoder Reranking for Maximum Precision#
Vector distance is a bi-encoder approximation computed independently. A Cross-Encoder Reranker passes both the user query and retrieved candidate chunks simultaneously through attention layers to score relevance with extreme precision:
interface ScoredChunk {
content: string;
relevanceScore: number;
}
export async function rerankCandidates(
query: string,
candidates: string[],
topN: number = 3
): Promise<string[]> {
const payload = {
query,
documents: candidates,
top_n: topN,
return_documents: true
};
const response = await fetch("https://api.cohere.ai/v1/rerank", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.COHERE_API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify(payload)
});
const data = await response.json();
return data.results.map((r: { document: { text: string } }) => r.document.text);
}5. RAG Pipeline Architectural Comparison#
| Pipeline Component | Naive RAG (Prototype) | Production RAG (Enterprise) |
|---|---|---|
| Chunking | Fixed 500 characters | Recursive semantic or hierarchical parent-child |
| Indexing | Dense vector index only | Hybrid (Dense Vector + BM25 Full-Text) |
| Retrieval Score | Cosine similarity top-K | Reciprocal Rank Fusion (RRF) |
| Post-Processing | None (direct pass to LLM) | Cross-Encoder Reranker (Top 3 re-scored) |
| Citation Attribution | Vague or missing | Strict markdown footnote grounding |
| Hallucination Risk | Moderate to High | Minimal (Near Zero with grounding constraints) |
Published by
Vyuhantrix Team
AI & Systems Engineering · Vyuhantrix
Vyuhantrix is an open technology learning platform based in Ahmedabad, India, publishing step-by-step programming tutorials, system design breakdowns, and free developer tools.
Keep Learning
Recommended Guides
How Vector Databases Power Modern AI Applications: A Practical Guide
A beginner-to-intermediate guide to high-dimensional embeddings, vector similarity search, Pinecone, pgvector, and Retrieval-Augmented Generation (RAG).
Google Gemini 1.5 Pro & Flash API: Multimodal Prompting, Function Calling & Structured JSON with Node.js
Master Google Gemini 1.5 Pro and Flash in Node.js. Learn multimodal image & video inputs, native structured JSON outputs, system instructions, function calling, and token budgeting.
Google AI Studio vs. Vertex AI: Choosing the Right Google AI Platform for Developers
Compare Google AI Studio and Google Cloud Vertex AI. Learn pricing models, IAM security boundaries, private VPC deployment, enterprise SLA guarantees, and model tuning.