Chunk documents for meaning before tuning an index

Chunking decides how much meaning a retrievable unit retains. A vector index decides how to find candidate units efficiently. These are related decisions, but they solve different problems.

A faster index cannot repair a chunk that separated a condition from its conclusion. Inspect document structure before tuning retrieval parameters.

Start with document structure

Headings, paragraphs, lists, tables, and code blocks often provide semantic boundaries. Avoid separating table headers from their values, prerequisites from procedures, or code from its explanation without a way to reconnect them.

Retain a document identifier, version, heading path, and original position for each chunk. Citations and neighboring-context expansion depend on this metadata.

Repeated processing of one version should produce stable or mappable chunk identifiers. Remove stale material during updates instead of only appending new chunks and leaving old conclusions searchable.

Trade off size and overlap

Small chunks may omit necessary context. Large chunks may include too much unrelated text. Overlap reduces some boundary problems but increases index size and duplicate evidence.

There is no universal chunk size. Technical instructions, tables, and narrative documents need separate inspection. Evaluate representative queries rather than judging only whether the chunks look tidy.

Inspect boundaries with a window

This example splits characters to demonstrate positions and overlap. It is not a production semantic chunker or a replacement for model-specific token counting.

def windows(text, size, overlap):
    if size <= 0 or not 0 <= overlap < size:
        raise ValueError("invalid window settings")
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append({"start": start, "end": end, "text": text[start:end]})
        if end == len(text):
            break
        start = end - overlap
    return chunks

text = "Keep the condition next to its conclusion."
chunks = windows(text, size=18, overlap=5)
assert chunks[1]["start"] == chunks[0]["end"] - 5
assert chunks[-1]["end"] == len(text)
assert all(c["text"] == text[c["start"]:c["end"]] for c in chunks)
for chunk in chunks:
    print(chunk)

Replace the sentence with a public document containing headings and tables. Inspect whether each window retains useful meaning. This often reveals problems before a vector database is involved.

Separate embeddings from indexes

An embedding maps input into a vector space. Distance supplies a retrieval signal; it does not establish the truth of an answer. Query and document encoding must follow the chosen model's requirements.

Approximate indexes such as HNSW use a graph structure to accelerate neighbor search. Their settings involve recall, latency, and memory tradeoffs. No index parameter repairs broken parsing, outdated sources, or missing authorization.

Lexical retrieval remains useful for exact terms and identifiers. Decide whether to combine lexical and vector candidates with your own query set rather than assuming one method is always sufficient.

Try it next

Choose ten questions and retrieve evidence with two chunking configurations. Compare whether correct passages appear, remain complete, and introduce duplicated context.

Keep the inputs, chunking version, and result identifiers. The next guide turns these observations into repeatable retrieval evaluations.