How AI Gets Forced to Quote Out of Context: The Hidden Art of Chunking

> Source: easy-learn-ai / commit 744f134, 2026-06-23

1. After a Book Gets Shredded

Imagine handing an AI your entire company employee handbook and asking: "Can my unused annual leave be converted into pay when I resign?"

The AI doesn't actually read all five hundred pages. It looks at roughly three pages of them—and if it picks the wrong three, the answer goes off track.

So the system's first job isn't reading the book. It's tearing the book into pieces.

This process is called chunking. The goal is simple: cut a large document into many small pieces so the AI can grab only the few most relevant ones to answer a question.

But this seemingly simple act of "cutting" hides a lot of trade-offs.

2. Why Chunk at All?

2.1 Stuffing the Whole Book In? The Window Won't Fit

The amount of text an AI can read at once is capped by the context window—think of it as the model's working memory.

A common window is 128K tokens, roughly 100,000 Chinese characters. A single handbook might fit, but add a dozen reference documents and the window is full.

Even if it fits, making the AI find the three sentences relevant to "leave conversion" inside an entire handbook is extremely inefficient—like scanning an entire encyclopedia instead of using the index.

2.2 Retrieval Needs Granularity

Chunking also determines the granularity of search. With good chunks, a question like "when does my leave expire" precisely retrieves the piece containing "carry over", "end of March", and "expire". With bad chunks, an answer can be split across a chunk boundary, and neither side retrieves the complete information.

Chunking therefore directly affects retrieval hit rates.

3. The Two Core Knobs

3.1 Chunk Size

• Too large (e.g., ~1000 characters): hit rates may be okay, but each chunk mixes in lots of irrelevant content, making it easy for the AI to drift off topic.
• Too small (e.g., ~100 characters): precision may improve, but a complete answer can scatter across chunks. If the cut falls between "end of March" and "it expires", the search may only retrieve half a sentence.

In practice, 200–500 characters is a common range, but it varies by document type:

• FAQs: one Q&A pair is usually one chunk (a few dozen characters).
• Policy handbooks: cut by subsection, keep headings, ~300–500 characters per chunk.
• Code: cut by function or class, add file path and dependency info; lengths vary widely.

3.2 Overlap

Overlap is a deliberately repeated segment between adjacent chunks. Why repeat on purpose? Because a cut in the middle can sever meaning near the boundary.

Example: "Unused annual leave carries over to the end of March of the next year, after which it automatically expires." If the cut lands after "end of March", a search for "what happens when leave expires" may not match the second chunk well.

Overlap (e.g., ~10% on each side) ensures the complete meaning appears intact in at least one chunk. The cost: more repetition means more total chunks and higher storage costs.

4. A Concrete Experiment

Take a real leave policy: leave accrues by tenure (5 days after 1 year, 10 after 3, 15 after 5); requests need 3 business days' notice and supervisor approval; unused leave carries over to the end of March, then expires; public holidays don't consume leave quota.

With a chunk size of ~46 characters, the text splits into several pieces:

• Query "how many days after five years?" — keywords cluster in one chunk, high hit rate.
• Query "how far in advance to request?" — keywords spread across chunks; if the cut separates them, you may get only a partial hit.

Conclusion: chunking directly determines how much relevant information retrieval can recall. Too fine and information scatters; too coarse and it drowns in noise.

5. Chunk According to Document Type

| Document type | Strategy | Watch out for | |---|---|---| | Policy handbooks | Cut by subsection/Q&A entry, keep headings | Dates, amounts, scope split across two chunks | | Contracts | Cut by clause, keep clause numbers/titles with the text | Preserve traceability to the original clause | | FAQs | One Q&A per chunk | Don't merge multiple answers; keep similar questions adjacent | | Code | Cut by function/class/file, add paths and dependencies | Cutting a function may lose type definitions, configs, call relations |

Pre-launch checklist

1. Each chunk should be understandable on its own. 2. Record metadata: source file, section, access permissions, version/date. 3. No important conditions broken across boundaries. 4. Test retrieval with real questions.

6. The Deeper Logic

Chunking is a trade-off between information density and retrieval precision. From an information-theory view, chunk size sets the granularity of information units: too large, and the signal-to-noise ratio drops; too small, and context is lost. Overlap trades redundancy for boundary robustness and retrieval reliability.

Deeper still, chunking affects embedding quality. A chunk spanning multiple topics produces a "mixed signal" vector, degrading matching accuracy. A good chunk should stay topically coherent.

7. Conclusion

In RAG-style systems, chunking is the first step—it determines the raw material for everything downstream. It's like mise en place in cooking: a chef doesn't toss a whole cabbage into the pot, but cuts it into proper pieces. Chunk well, and the rest of the "cooking" can turn out great.