Loading…
Glossary term
Splitting documents into smaller pieces so they can be indexed and retrieved.
Long documents are split into smaller pieces, chunks, before being embedded and indexed. A chunk should be small enough to be specific but large enough to make sense on its own, often a few hundred words, sometimes with overlap.
Splitting along natural boundaries like headings keeps ideas together.
Split text into overlapping chunks
def chunk(text, size=500, overlap=50):
chunks, start = [], 0
while start < len(text):
chunks.append(text[start:start + size])
start += size - overlap
return chunksPoor chunking is a common reason RAG systems retrieve irrelevant or incomplete passages.