RAG: Storage & Retrieval

• Vector DB in RAG = Store Embeddings + Search Engine
• Examples of vector databases include Pinecone, pgvector, Chroma DB.
• Real models can have 384, 768, 1536, or 3072 dimensions. So each chunk embedding can become a large number of vectors.
• Vector DB stores:
  • Vector
  • Original text chunk
  • Metadata (Your app decides the metadata. For example, if you store HR data, you may add metadata as policy-name: "Leave" and policy-type: "HR".)

{
  "id": "employee-handbook-001",
  "embedding": [0.12, -0.44, 0.93, ..., 0.7, 0.19],
  "text": "Employees get 20 paid leaves per year",
  "metadata": {
    "source": "employee-handbook.pdf",
    "section": "leave-policy",
    "page": 12,
    "chunk-index": 1
  }
}

The vector DB compares the user query vector with chunk vectors using cosine similarity.

• For every user query, compare the user query vector with each chunk vector.
• Brute-force search works well if there are few chunks, say 100 or 1,000. But in production you mostly will have a lot more: 10,000, 1 lakh, or millions of chunks.
• Trade-Off: Brute force is accurate but slow.

• Vector indexing, like a SQL index, helps avoid scanning every record or chunk in the DB.
• A vector index does approximate nearest-neighbor search. Results after using these indexing methods may not be as accurate as brute force, but we accept approximate results so that we save latency and compute.
• Types of vector indexes:
  1. HNSW (Hierarchical Navigable Small World)
  2. IVF (Inverted File)
  3. Product Quantization (PQ)
  4. IVF-PQ

• It is an indexing technique that finds nearest vectors quickly without comparing the user query vector with every stored chunk vector.
• HNSW is used in production where speed and accuracy are important.

Hierarchical Graph Creation

• HNSW creates a multi-layer graph where each vector is a node and each node is connected to nearby vectors (semantically close vectors) using cosine similarity.
• Search begins at a random node, then moves to the neighbor of that node that is closest to the user query vector, then to neighbors of that neighbor, and so on.

How HNSW search moves through layers?

• Search starts at a random node in the top layer (Layer 2 above). Find top layer node which is nearest to the user query vector.
• Then find its nearest neigbour (node) in the layer below (Layer 1 above)
• And so on ... moving down to layer 0, thus coming closer and closer to nearby, similar vectors.

Who goes in the top layer?

• Ideally the vector that is semantically the root should be on top.
• But with millions of nodes, this would be costly because each time we store a new vector, the root and its subtree may change. Also, it is not necessary that there is a clear winner i.e. a vector that is semantically root of all other vectors.
• So, HNSW randomly assigns a level to each node. Every node is always in level 0. Only some nodes randomly get level > 0.

Example Graph

Sample Nodes:

A - paid leave policy              Layer 0
B - sick leave policy              Layer 1
C - laptop reimbursement policy    Layer 1
D - travel reimbursement policy    Layer 0
E - Accounting department          Layer 2
F - HR department                  Layer 2
G - Hiring process                 Layer 1

Example Search

User query: How many paid leaves can I take?

1. Layer 2: Start search. Compare user query vector with E & F. The user query vector is semantically closer to node F.
2. Layer 1: Go to the sub-tree of node F to reach its neighbors. Compare with neighbours B and G. The user query vector is semantically closer to node B.
3. Layer 0: Go to the sub-tree of node B to reach its neighbors. Compare with neighbour node A which is semantically closest to the user query vector.
4. Return Top K = 3 closest nodes: A, B & F.

Implementation

To implement HNSW, data structures like skip lists or Delaunay graphs are used.

HSNW Parameters

• Inverted File Index (IVF) groups similar vectors into buckets, or clusters, and then searches only within the clusters that are closest to the user query vector.
• It uses the K-Means clustering algorithm
• Suppose you have these document chunks:
  (A) paid leave, (B) sick leave, (C) hardware request, (D) laptop reimbursement and (E) travel reimbursement.
• If you ask to group them into K = 3 clusters:
  1. Cluster 1: paid leave, sick leave
  2. Cluster 2: laptop reimbursement, travel reimbursement
  3. Cluster 3: hardware request
• Each cluster has a centroid. A centroid is the numerical average of the group of vectors.
  For example, if A = [1, 2] and B = [2, 2], then
  Centroid C1 = [(1 + 2) / 2, (2 + 2) / 2] = [1.5, 2].

How are clusters created?

1. Say we decided to build K clusters. Then for each cluster, randomly select a vector as its centroid.
2. Assign each remaining vector to the cluster whose centroid is closest to this vector.
3. Recalculate each centroid as the average of vectors within its cluster.
4. Reassign all vectors to the nearest centroids.
5. Repeat until clusters stabilize.

A centroid is an average number. It does not necessarily represent an actual chunk vector.

Example
1) Pick centroids: C1 = [1, 1], C2 = [8, 8]
2) Assign vectors:
   Cluster 1: A = [1, 1], B = [1, 2], C = [2, 1]
   Cluster 2: D = [8, 8], E = [6, 9], F = [7, 8]
3) Recalculate centroids:
   C1 = [(1 + 1 + 2) / 3, (1 + 2 + 1) / 3] = [1.33, 1.33]
   C2 = [(8 + 6 + 7) / 3, (8 + 9 + 8) / 3] = [7, 8.33]
4) Re-check if any chunks need to move to another cluster.
5) If no chunks move, clusters are stabilized.

How vectors are stored inside IVF?

Vectors are stored in Inverted File Pattern

Search

1. Take the user query vector (UQ)
2. Find the nearest centroid
3. Search vectors within the cluster of the nearest centroid.

Step 1:
  UQ vector vs C1 = 0.91 similarity
  UQ vector vs C2 = 0.35
  UQ vector vs C3 = 0.62

Step 2:
  Search within cluster of nearest centriods C1 and C3

Step 3:
  Inside Cluster 1:
    UQ vector vs chunk 1 = 0.94
    UQ vector vs chunk 2 = 0.72
  Inside Cluster 3:
    UQ vector vs chunk 3 = 0.88

Response Top 2 closest: chunk 1 and chunk 3

IVF Parameters

Risk

If a user query is not properly formed, or if the query vector is closer to C1 but the answer is in cluster C2, searching only C1 can miss the right chunk.

e.g. user query: Engineering deparment needs 30 Apple MacBook Air each with 10 GB RAM, 10 Apple keyboards and 10 Apple mice. Will it fit budget of Q4?

- Closest Chunk: Chunk E of Cluster 2
- But since the user query has a lot text relating to Apple, the query vector is closer to C1 than C2. If you search only cluster 1 (with nprobe = 1), you will not find the right answer chunk E which is in cluster 2.

• Product Quantization (PQ) is a vector compression technique.
• PQ compresses large vectors into smaller codes, so vector search becomes cheaper and faster.
• A common production stack is IVF + PQ. IVF narrows where you search, and PQ shrinks the vectors.

Floating point quantization

Converts floating point values into smaller integer values.

Vector = [0.12, -0.44, 2.37, 10.8, ..., 1.8, 0.19]
Compressed vector = [0, 0, 2, 10, ..., 1, 0]

Code quantization

Splits a vector into parts and replaces each part with a code.

Vector = [0.12, -0.44, 0.87, 0.10, 0.55, -0.20]

Split:
    Part 1 = [0.12, -0.44] -> code 7
    Part 2 = [0.87, 0.10]  -> code 2
    Part 3 = [0.55, -0.20] -> code 13
Compressed representation = [7, 2, 13]

Codebook:
  Id 7  = [0.12, -0.44]
  Id 2  = [0.87, 0.10]
  Id 13 = [0.55, -0.20]

It depends on the use case. In general,

• pgvector: when you already run Postgres and want SQL metadata filters plus vector search.
• Qdrant: metadata filtering, low latency, and cost.
• Weaviate: built-in vectorization and GraphRAG-style retrieval.
• Milvus: disaggregated compute/storage and scalability.

Things to consider when selecting a vector DB:

1. Filtering support:
   - Filter by metadata to increase accuracy of chunk retreival.
   - Each chunk should store metadata, e.g. doc name, page number, section header.
   - When the LLM gives a response, this helps us figure out which chunks were picked and helps us check accuracy.
2. Hybrid search support.
3. Support your existing stack, such as Postgres or SQL apps using pgvector.
4. Benchmark on your own data: Evaluate your RAG system and tweak it to get best results

e.g. user query: Can employee 007 take 10 days leave?

Employee data is in SQL DB. HR leave policy is in documents we ingest inside RAG.

Offline step

{
  "id": "chunk-01",
  "text": "Contract employees in India get 20 paid leaves per year",
  "embedding": [0.12, -0.44, 0.91, 0.75, 0.53],
  "metadata": {
    "doc_type": "policy",
    "policy_type": "leave",
    "country": "India",
    "emp_type": "contract"
  }
}

Online step

select employee_id, country, employment_type, status, available_leave_balance
from emp_leave_table
where emp_id = '001';

Result JSON:
{
  "emp_id": "001",
  "country": "India",
  "emp_type": "contract",
  "status": "active",
  "avail_leave_bal": "10 days"
}

Search process

e.g. user query: Can employee 007 take 10 days leave?

1. Hit SQL DB to get employee details: id, country, emp_type, avail_leave_bal
2. Use this data as filter on chunk metadata. Tell vector DB to search only chunks where metadata value = emp data
3. Send data from steps 1 & 2 along with user query vector to LLM

Search only chunks where:
emp_id = "007"
policy_type = "leave"
country = "India"
emp_type = "contract"

Faster search because fewer chunks are searched.