BS
Back to blog posts

AI Engineering / LLM 101

LLM 101: prompts, tokens and context windows

Updated Jul 2026 5 min read Built from handwritten notes

What is an LLM?

LLM stands for Large Language Model. It is a system that is trained on a large dataset like the stock market history or all the languages in the world or the entire text on the web.

What is it trained for? LLM's job Given an input, predict a response based on your training.

How does it respond?

LLM does not think. It does not know the future.

It only knows the dataset and generates a response based on the patterns it learned during training plus the context you give in the prompt.

e.g. ChatGPT, Claude, Gemini, DeepSeek, Llama

PromptPost-trained dataPatternsLLMPredicts a response

Token

A token is the smallest unit an LLM processes. It can be a word or a part of the word. Roughly 1 token is ~ 4 characters, but every model has its own tokenizer which decides how to break the text into tokens.

  • Billing often depends on the number of input and output tokens consumed.
  • Each token receives an internal ID before the model processes it.
  • Tokenization is model-specific: car may stay one token car, while a longer word like automobile may split into 3 tokens auto, mob and ile.

Prompt

Prompt is the instruction given to the model. 2 types: the user prompt and the system prompt.

User prompt

The message written by the user i.e. the question you ask or task you give to ChatGPT. e.g.
"Explain RAG to a beginner."
"Modify my resume so that it is ATS compliant and has high ATS score."

System prompt

Instruction given to the LLM on what it needs to do. These are set when designing AI apps. e.g.
“You are a resume writer that modifies resumes to get high ATS scores."
"Do not add any skills or experience that the user has not provided in the resume.”

A good prompt has five parts

1. Role2. Task3. Context4. Rules5. Output format

Example prompt structure

You are a senior AI engineer. Explain RAG to a beginner. The student is a Product Manager. Use simple language, avoid unnecessary math and give examples. Include the definition, why it matters and one example.

System prompt: Key to Success

System prompts are especially important as they define behavior, safety boundaries, and source-of-truth rules for the LLM before user input arrives. e.g.

  • Answer only from provided or retrieved documents.
  • If the answer is not in the documents, say “I don’t know.”
  • Do not hallucinate; cite sources when possible.
  • Use only retrieved documents, not the open internet, unless the app explicitly allows it.
  • Ask a clarifying question when the request is ambiguous.

Context window

Context window is how much data the model can look at in one request i.e. the maximum no. of tokens the model can attend to, at once.
Once the window fills up, LLM compacts (summarizes) the data within the context window.

Context window contents

User promptSystem promptChat historyRetrieved documentsTool resultsAI response
All of these compete for the same token budget. If the request is too large, LLM compacts the context window and tries to keep the most relevant information, but that can cause accuracy loss causing LLM to hallucinate.

Takeaway

Key to AI engineering is giving the right context to the LLM at the right time in the right amount.
And that starts with a disciplined context design: choosing the right instructions, passing only relevant documents, providing necessary tools and modelling the response.