RAG: Architecture

LLMs have two common failures:

1. Knowledge cutoff:
- Once the LLM has completed its training, it will not learn anything new.
- For example, if an LLM was trained on web development knowledge up to June 2025, it will not know features from a new React version released after June 2025.
- It also will not know your private component library and CSS styles because that data was not part of training.

2. Hallucination:
- If you ask an LLM something it does not know, it may guess. It will produce an answer, that too confidently. But the answer may be wrong, unrelated, or irrelevant.
- Since the model has to answer something, it can hallucinate instead of saying, “I do not know this.”

- RAG stands for Retrieval Augmented Generation.
- Your RAG application is not part of the LLM. You build the RAG app to provide extra relevant information to the LLM, so when it answers, it can answer from data your RAG gave, its own training, the user prompt and the retrieved context.

RAG combines two memories

Example:

User asks: “What is the leave policy?”

- Provide the RAG app, extra relevant information like the company documents, codebase, design principles, user feedback etc.
- These could be PDFs, Word documents, web pages, code, GitHub links, support tickets, support logs, images, videos, or other internal sources.
- e.g. HR policies, GitHub repository, or support logs.

- Split documents into smaller chunks
- If company data is large, not all of it will fit in the LLM context window. So when answering, the LLM may use only part of the company data or use summarized data, which can cause hallucination.
- Chunking helps limit the LLM context window.

- For example, an HR policies document may become separate chunks for leave policy, travel policy, holiday calendar, and work-from-home policy.
- If the user asks “What is the leave policy?”, your RAG can retrieve only the relevant leave policy chunk, not the whole document.

- Convert chunks to vectors.
- LLMs do not understand text directly; they understand vectors.
- The user query is also converted to an embedding. Use the same embedding model for chunks and the user query so similar meaning tokens lands close together in the same vector space.

- Store embeddings in a vector database, such as Chroma DB, Pinecone, or Postgres with pgvector.
- We cannot use a regular SQL database since we are not doing text comparison or keyword search. We do vector comparison, hence we need vector DB.

Example of a chunk stored in DB:

{
  "id": "employee-policy-leave-001",
  "text": "Employees get 20 paid leaves per year",
  "embedding": [0.12, -0.44, 0.9],
  "metadata": {
    "source": "employee-handbook.pdf",
    "section": "HR - Leave Policy",
    "page": 12,
    "chunkIndex": 1
  }
}

- Retrieval performs vector similarity search to compare the user query embedding vector with chunk vectors.
- It retrieves the top K chunks. These are the most relevant chunks likely to have the right data to feed the LLM.
- K is decided when designing the RAG system.

The LLM gets the user query vector & relevant chunks from RAG. The LLM answers using this additional context.