How RAG Pipelines Chunk Documents Into Vectors

Your RAG chunks are destroying 40% retrieval accuracy — and most teams never realize it. The way you split a document into pieces before embedding it is the single most consequential decision in any Retrieval-Augmented Generation pipeline, yet most developers reach for the default settings in LangChain or LlamaIndex and move on. The result? Queries that return irrelevant chunks, answers that miss critical context, and a system that feels broken despite every component working as designed.

Chunking isn’t a preprocessing step you can afford to gloss over. It’s the foundation that determines whether your vector store returns the right evidence or noise. In this article, we’ll walk through the exact 8-step pipeline that transforms a raw document into retrievable vector embeddings — and show you where the defaults betray you.

Step 1: Document Loading — Preserving Structure Before You Split

Every RAG pipeline starts with a document loader that reads the source file and extracts raw text while preserving structural metadata. This metadata — headers, page numbers, section markers, table boundaries — becomes critical later when you need to trace a retrieved chunk back to its source context.

The loader builds a metadata dictionary alongside the raw text. For a PDF, this means parsing the internal structure to capture heading hierarchy (H1, H2, H3), page boundaries, and even column layouts. For markdown files, the heading syntax itself becomes the primary structural signal. For HTML, the DOM tree provides natural section boundaries.

The mistake most teams make here is stripping everything down to plain text immediately. When you lose the heading structure of a 50-page legal contract, your chunking algorithm has no natural break points to work with — and you’re forced into blind character-level splitting that slices right through the middle of a clause.

Step 2: Text Normalization — Cleaning Without Destroying

Before any splitting happens, a text normalizer removes excessive whitespace, standardizes line breaks to single newlines, and converts special Unicode characters to ASCII equivalents without altering semantic meaning. This step exists because real-world documents are messy: PDFs export with random line breaks, web scrapes inject non-breaking spaces, and legal documents use em-dashes where hyphens should be.

The critical rule: normalization must preserve meaning. Converting u2019 (right single quotation mark) to ' is safe. Stripping all punctuation is not — it destroys the sentence boundaries that semantic chunkers rely on. A common pitfall is over-aggressive regex cleaning that collapses “Dr. Smith” into “Dr Smith” and breaks downstream sentence tokenization.

Step 3: The Chunking Algorithm — Where Defaults Kill Accuracy

This is the step that makes or breaks your RAG system. The chunking algorithm splits text at specified character or token boundaries, starting from position 0 and advancing by (chunk_size - overlap_size) for each subsequent chunk.

“LangChain’s RecursiveCharacterTextSplitter defaults to 4000 character chunks with 200 character overlap, but OpenAI’s text-embedding-ada-002 performs optimally at 512 tokens or roughly 384 words”

That gap between default and optimal is not trivial. At 4000 characters, you’re stuffing roughly 800 tokens into each chunk — nearly 60% more than what the embedding model handles best. The result is dilution: the vector representation has to compress too much semantic information into a single 1536-dimensional point, and the distinctiveness of each chunk suffers.

There are three major chunking strategies in production use today:

Fixed-Size Chunking

The simplest approach: split every N characters or tokens. Fast, predictable, but blind to meaning. A 4000-character chunk might contain the end of one section and the beginning of another, creating a vector that represents neither topic well. Pinecone’s benchmark tests revealed that overlapping chunks by 10-15% of chunk size creates 18% more vectors but prevents boundary information loss that causes 23% of failed retrievals.

Semantic Chunking

Instead of counting characters, semantic chunkers split at natural meaning boundaries — typically sentence or paragraph breaks. Anthropic’s 2023 research showed semantic chunking with BERT sentence embeddings reduces context loss by 34% compared to fixed-size splitting in legal documents over 50 pages. The trade-off is computational cost: you need an additional model pass to identify sentence boundaries, and chunk sizes become variable (which complicates downstream indexing).

Attention-Aware Chunking

The most advanced method, from Stanford’s 2024 paper, uses GPT-4’s attention weights to identify natural break points, reducing average chunk count by 31% while maintaining 96% semantic coherence. This approach treats the document the way the model “reads” it — splitting where attention naturally drops between topics. It’s expensive (requires a full model forward pass), but for high-value document collections, the retrieval quality gain is substantial.

Step 4: Overlap Handling — Bridging the Gaps

When a chunk boundary falls in the middle of a thought — a sentence, a list item, a contract clause — both adjacent chunks lose critical context. The overlap handler solves this by copying the last N characters from chunk[i] to become the first N characters of chunk[i+1], where N equals the overlap parameter typically set between 10-20% of chunk size.

Here’s the math: with a chunk size of 512 tokens and 15% overlap, each chunk starts 435 tokens after the previous one (512 – 77). The 77-token overlap means that any sentence split by a boundary appears in full in at least one chunk. This redundancy is intentional — it’s the price of preventing information loss at boundaries.

But overlap has a compounding cost. A 10,000-token document with 512-token chunks and 15% overlap produces ~23 chunks instead of ~20. Each chunk requires an embedding API call and vector storage. At scale — millions of documents — that 15% overlap can mean thousands of dollars in additional embedding costs and significantly larger vector indexes.

Step 5: Metadata Enrichment — Making Chunks Traceable

After chunking, a metadata enricher attaches source document ID, chunk index number, character start/end positions, and parent section headers to each chunk object. This traceability is what lets you cite sources in your final output — “according to page 14, section 3.2 of the contract” — rather than returning a chunk of text with no provenance.

LlamaIndex’s SentenceWindowNodeParser takes this further: it maintains 3 sentences before and 3 sentences after each chunk as metadata, increasing storage requirements by 2.8x but improving answer accuracy by 27%. When a retrieved chunk needs context that wasn’t captured in the chunk itself, the surrounding sentences are immediately available without a second retrieval pass.

The metadata schema you choose here has long-term consequences. If you ever need to re-chunk (and you will — as embedding models improve, re-indexing is standard practice), having precise character positions and source references makes it possible to rebuild your index without re-processing the original documents.

Step 6: Embedding — Text to Vectors

The embedding model converts each text chunk into a dense vector representation — typically 1536 dimensions for OpenAI’s text-embedding-3-large or 768 dimensions for open-source models like BGE-M3. This happens through a transformer neural network forward pass: the chunk’s tokens are processed through self-attention layers, and the final hidden state is pooled into a fixed-size vector.

Chunk size directly impacts embedding quality. Smaller chunks (256-512 tokens) produce vectors that represent a single, focused concept. Larger chunks (1000+ tokens) produce vectors that average across multiple topics — and “average” in vector space means “not particularly close to anything specific.” This is why the default 4000-character chunks in LangChain are problematic: they force the embedding model to compress too much into a single point.

Batching matters for cost. OpenAI’s embedding API charges per token, and each chunk is a separate request. With 512-token chunks and 15% overlap, a 100-page document (~50,000 tokens) produces roughly 115 chunks. At $0.13 per million tokens for text-embedding-3-small, that’s under a cent per document — but at millions of documents, optimizing chunk size and overlap becomes a real budget decision.

Step 7: Vector Storage — Building the Index

The vector database stores embeddings with their metadata in an index structure like HNSW (Hierarchical Navigable Small World) that enables approximate nearest neighbor search in logarithmic time. HNSW constructs a multi-layer graph where each layer contains a subset of vectors, and search traverses from the top (coarsest) layer down to the bottom (finest) layer — similar to skipping through levels of a map to find a location.

The choice of index parameters matters for retrieval quality. The ef_construction parameter (default: typically 128-200) controls how many neighbors are evaluated during index building — higher values produce a better graph but slower builds. The ef_search parameter controls how many neighbors are evaluated during queries — higher values return more accurate results but with higher latency. In production, teams often set ef_construction high (since indexing is a one-time cost) and tune ef_search to meet their latency SLA.

Overlap chunks create duplicate content in the vector store, which means duplicate or near-duplicate vectors. This can skew retrieval: if two overlapping chunks both match a query, they consume 2 of your top-k slots with essentially the same information. Deduplication at retrieval time (by source section or chunk proximity) is a common post-processing step.

Step 8: Retrieval — The Final Test

The retrieval system computes cosine similarity between the query embedding and all chunk embeddings, returning top-k chunks (typically k=3 to 5) that exceed a similarity threshold of 0.7 or higher. This is where chunking quality becomes visible: if your chunks are well-sized and properly overlapped, the top-k results will contain exactly the evidence needed to answer the query. If they’re not, you’ll see one of three failure modes.

Failure mode 1: Boundary splits. The answer exists but was split across two chunks, and neither chunk alone contains enough context to be useful. This is the most common failure — and the one overlap is designed to prevent.

Failure mode 2: Chunk dilution. The chunk contains the answer buried in 800 tokens of irrelevant context. The embedding averages across all the content, and the vector drifts away from the specific concept being queried. This is the 4000-character default problem.

Failure mode 3: Missing chunks. The answer spans a concept that doesn’t align with any single chunk boundary. Semantic chunking and attention-aware chunking specifically address this by splitting at meaning boundaries instead of character positions.

The Chunking Decision Framework

Here’s how to choose the right chunking strategy for your use case:

• Short, factual documents (FAQs, product specs): Fixed-size chunking at 256-512 tokens with 10% overlap. These documents have uniform density, so simple splitting works well.
• Long, structured documents (legal contracts, research papers): Semantic chunking with section-aware splitting. Use the heading hierarchy from Step 1 as primary boundaries.
• Mixed-content collections (company wikis, documentation sites): Recursive character splitting at 512 tokens with 15% overlap, enriched with SentenceWindow metadata for context recovery.
• High-stakes retrieval (medical, legal, financial): Attention-aware chunking where the cost of missed retrieval exceeds the cost of the additional model pass.

Whatever strategy you choose, measure. Track retrieval precision (are the top-k chunks relevant?) and answer accuracy (does the LLM produce correct answers from retrieved context?). Chunking parameters are hyperparameters — tune them with data, not intuition.

But what happens when your chunks split a critical sentence in half? The next article reveals the semantic boundary detection algorithm that prevents this — and it’s simpler than you’d think.