When AI Is Forced to Quote Out of Context: The Hidden Art of Chunking

Forum topic · 小凯 · 2026-06-23

Summary

This article explains chunking, the foundational step in retrieval-augmented generation (RAG) systems, using a vivid analogy: when you ask an AI about a company policy, it never reads the whole manual—it reads only a few selected chunks, so how the text is split determines answer quality. The author covers why chunking is necessary (limited context windows and inefficient full-document search), the two core parameters—chunk size and overlap—and their trade-offs. Oversized chunks dilute signal-to-noise ratio, while undersized ones break complete semantic units across boundaries. Overlap deliberately repeats 10% or so of text between adjacent chunks to preserve boundary meaning at the cost of storage. A worked example using annual-leave policy text shows how splitting affects retrieval hit rates for different questions. The post includes a practical table of splitting strategies by document type (manuals, contracts, FAQs, code) and a pre-launch checklist: each chunk should be self-contained, carry metadata (source, section, permissions, version), preserve boundary conditions, and pass real-query retrieval tests. It concludes that chunking balances information density against retrieval precision, and that single-topic chunks yield better embeddings.

When AI Is Forced to "Quote Out of Context": The Hidden Art of Chunking

> Source: easy-learn-ai / commit 744f134 > Date: 2026-06-23

1. After a Book Gets Torn to Pieces

Imagine handing an AI your company's entire employee handbook and asking: "Can my unused annual leave be converted into pay when I resign?"

The AI won't actually open the book and read all five hundred pages. It will look at roughly three pages—and if the wrong three pages are selected, the answer goes off track.

So the first thing the system does is not read the book, but tear it apart.

This process is called Chunking. Its goal is simple: cut a thick document into many small pieces so the AI can retrieve only the few chunks most relevant to a question.

But the act of "cutting," though it looks simple, hides a lot of trade-offs.

2. Why Cut at All? Can't We Skip It?

2.1 Stuff the Whole Book In? The Window Won't Fit

The amount of text an AI can read at once has an upper limit, called the Context Window—think of it as the AI's "working memory."

A common window is 128K tokens, roughly 100,000 Chinese characters. An employee handbook might only be 50,000 characters, so it seems to fit. But if you stuff in a dozen reference documents at once, the window fills up.

More importantly, even when it fits, making the AI find the three sentences about leave conversion in an entire handbook is extremely inefficient—like finding a word in an encyclopedia by reading cover to cover instead of using the index.

2.2 Cut Only the Relevant Pieces? Search Needs Granularity

Chunking does more than split—it determines search granularity. Cut well, and the query "when does leave carry over to?" precisely locates the small chunk containing "carry over," "end of March," and "expire." Cut poorly, and the answer may straddle two chunk boundaries, so neither chunk contains complete information.

The splitting strategy directly affects downstream retrieval hit rates.

3. Core Chunking Parameters: Two Knives

3.1 Chunk Size: How Long Is Each Piece?

Too large (e.g., 1000 characters per chunk): retrieval recall may be OK, but each chunk mixes in lots of irrelevant content, making it easy for the AI to go off topic.
Too small (e.g., 100 characters): higher search precision, but a complete answer's conditions may be scattered across chunks. If the cut lands between "end of March" and "automatically expires," the search may only retrieve half a sentence.

In practice, a common range is 200–500 characters, but it varies by document type:

FAQ: one Q&A pair is usually one chunk, possibly only a few dozen characters.
Policy manuals: split by section, keep headings, ~300–500 characters per chunk.
Code: split by function or class, adding path and dependency info; lengths vary widely.

3.2 Overlap: The Protective Buffer

Overlap is a small stretch of text deliberately duplicated between adjacent chunks.

Why duplicate? Because when the cut lands mid-sentence, meaning near the boundary gets severed. Example:

> "Unused annual leave can carry over to the end of March next year; past the deadline it expires automatically."

If the cut falls right after "end of March," one chunk ends there and the next begins with "automatically expires." A search for "what happens if leave passes the deadline" may match poorly against the second chunk's incomplete opening.

Overlap (e.g., ~10% on each side) ensures the complete meaning near boundaries appears intact in at least one chunk. The obvious cost: more duplication means more total chunks and higher storage costs.

4. How Splitting Affects Search: An Experiment

Using a real company annual-leave policy passage:

> "Annual leave is tiered by tenure: 5 days per year after one year of service; 10 days after three years; 15 days after five years. Leave requests must be submitted in the system three business days in advance and approved by the direct supervisor. Unused leave carries over to the end of March next year and expires after that. Public holidays and adjusted rest days do not consume leave quota."

With a chunk size of 46 characters, different questions hit different results:

"How many days after five years of service?" — keywords cluster in the same chunk: high hit rate.
"How far in advance must I request leave?" — keywords scatter across chunks; one or both chunks may only partially match.

Conclusion: chunking directly determines how much relevant information retrieval can recall. Cut too fine and information scatters; cut too large and information becomes noisy.

5. Split According to What the Material Looks Like

There is no one-size-fits-all number:

| Document Type | Splitting Strategy | Key Checks | |---------|---------|---------| | Policy manuals | Split by section or Q&A entry; keep headings | Dates, amounts, scope of applicability must not straddle two chunks | | Contracts | Split by clause; clause numbers and titles travel with the body | Citations must trace back to the original clause; don't store isolated fragments | | FAQ | One Q&A pair per chunk | Keep similar questions nearby; don't merge multiple answers into one chunk | | Code | Split by function, class, or file structure; add path and dependency info | Splitting by function alone can lose type definitions, configs, and call relationships |

Pre-launch checklist:

1. Each chunk, read on its own, is understandable 2. Each chunk carries metadata: source file, section, permissions, version 3. Important conditions near boundaries are not severed 4. Retrieval results verified with real questions

6. The Deeper Logic Behind Chunking

Chunking is a trade-off between information density and retrieval precision.

From an information-theory perspective, chunk size determines the granularity of information units. Too coarse, and the signal-to-noise ratio drops (useful information drowns in irrelevant content); too fine, and context is lost (a complete semantic unit is torn apart).

Overlap trades redundancy for retrieval reliability—balancing boundary robustness against storage cost.

Deeper still, chunking affects embedding quality. A chunk containing multiple topics produces a "mixed signal" vector, reducing matching precision. Good chunks preserve topical singularity as much as possible.

7. Conclusion: Cut Well, Find Accurately

In systems where the AI consults materials before answering (like RAG), chunking is step one. It determines the raw material available to every downstream step.

Cut well, and search hits precisely; cut poorly, and answers drift. It's like prep work in cooking—a chef doesn't throw a whole cabbage into the pot but cuts it into suitable pieces. Chunking is the AI's prep work: get the cuts right, and the rest of the "cooking" can produce a good dish.

---

*Reference: easy-learn-ai Chunk module (commit 744f134)*