RAG: Chunking & Embedding

- Chunking splits a document into chunks. A good chunk keeps related information together, fits inside the model context window and is small enough for accurate retrieval.

- The hard part is that the splitter does not always know where human meaning begins and ends. If it cuts in the wrong place like cutting a policy statement or a code function or an API example, LLM can lose the context needed to answer correctly.

Fixed-size chunking splits documents after a fixed number of characters or tokens. For example, chunk size might be 500 characters, 10 words, or page-wise chunks.

Chunk size = 8 words

chunk 1: Leave Policy: Employee gets 10 paid leaves per
chunk 2: year. No carry forward. Laptop Reimbursement Policy: Up
chunk 3: to $1000 allowed. Only MAC allowed.

Overlap chunking is fixed-size chunking where chunks share some text. For example, chunk size = 8 words and overlap = 4 words. This increases the chance that related text falls in one chunk.

chunk size = 8 words, overlap = 4 words

chunk 1: Leave Policy: Employee gets 10 paid leaves per year. No carry forward.
chunk 2: year. No carry forward. Laptop Reimbursement Policy: Up to $1000 allowed. Only
chunk 3: to $1000 allowed. Only MAC allowed

Sentence-based chunking keeps full sentences together.

Paragraph-based chunking splits by paragraphs.

- Section/header-based chunking splits documents using section headings.
- Use section-based chunking for documents with fixed stucture like markdown files, technical docs, API docs, HR policies, legal docs.

# Leave Policy -> chunk 1
# Travel Policy -> chunk 2
# Laptop Refresh Policy -> chunk 3

Recursive chunking tries to split a document using the best natural separator first, then falls back to smaller separators.

- Semantic chunking splits a document based on meaning. It tries to keep related topics together.
- Works well for systems where high accuracy is needed in responses.

para 1: Leave Policy
para 2: Carry-forward Rules
para 3: Sick Leave Process
para 4: Laptop Reimbursement

chunk 1 = para 1 + 2 + 3, as all 3 are semantically similar
chunk 2 = para 4

- Chunk based on the document type or document structure.
- It is mostly used in production systems where higher accuracy is needed.

- Chunks should also store metadata.
- Metadata helps with citation, filtering, debugging, and source tracking.

{
  "id": "employee-policy-leave-001",
  "text": "Employees get 20 paid leaves per year",
  "embedding": [0.12, -0.44, 0.9],
  "metadata": {
    "source": "employee-handbook.pdf",
    "section": "HR - Leave Policy",
    "page": 12,
    "chunkIndex": 1
  }
}

Embedding converts text to vectors.

Which model to choose?

• - No single model is best. Choose one that fits your document types and use case.
• - Test embedding models with your documents and retrieval results.
• - The MTEB leaderboard on Hugging Face compares embedding models.
• - Trade-offs to consider: latency, dimensions and quality. Dimensions such as 256, 768, or 1024 mean the text is converted into that many vector numbers.
• - Always use the same model to embed chunks and the user query.
• - If your document has legal jargon, choose a model trained for legal language. If your document has images, choose a model that can embed images and compare their vectors.

- For documents with code or complex structure, you need a parser before chunking. A parser understands the document shape so chunks are not created from broken pieces.
- For PDFs, you might use a PDF parser. For codebases, an AST parser creates an abstract syntax tree representation so the chunker can split by class, module, function, or component.

Function Declaration
    |-- name: sum
    |-- params:
    |     |-- a
    |     |-- b
    |-- body
    |     |-- Return Statement
    |     |       |-- Binary Expression
    |     |       |       | -- left: a
    |     |       |       | -- operator: +
    |     |       |       | -- right: a