AI Engineering / Embedding & Training

Embedding & Training

Updated Jul 2026 8 min read Built from handwritten notes

Mental Model

An LLM does not directly compare English words the way humans do. It turns text into numbers, learns patterns in those numbers during training and then uses those learned patterns to predict useful answers.

Vector

In Math, a vector has a magnitude and a direction.
Magnitude means a list of numbers.

If two pieces of text mean similar things, their vectors should point in a similar direction.

Embedding

In LLM, each token is represented by a vector.
Each LLM gives its own vector to a token. Claude, Gemini, and other LLMs can give different vectors for the same token.
Embedding is a vector that represents LLM's meaning of the text.

Example embedding table

A 2-dimensional vector can use Dimension 1 for text with vehicle-like meaning and Dimension 2 text with for fruit-like / yellow color like meaning.

Text	Vector	Meaning
Car	[0.95, 0.05]	very vehicle-like, not fruit-like
Automobile	[0.96, 0.04]	very vehicle-like, not fruit-like
Banana	[0.08, 0.96]	not vehicle-like, very fruit-like
Yellow vehicle	[0.81, 0.45]	part vehicle-like, part yellow-like

2D Vector Space

Direction of car and automobile is similar. Banana is a different direction. Similar meaning means close vectors and similar direction. Different meaning means far vectors and different direction.

How does LLM know car is vehicle-like but banana is not?

During training you teach LLM that “car” and “automobile” are related and are vehicle-like. Hence during embedding you give close vectors to them.

Training relationship → embedding closeness

Cosine similarity

Cosine similarity asks: how close two vectors are? Ideally if 2 vecors are in:

Identical Direction → cosine similarity = 1
Opposite Direction → cosine similarity = -1
Unrelated → cosine similarity = 0

Why is it needed?

Let's take 3 sentences:

The car won't start
My automobile broke down
I love chocolate shake

If we only compare English words without meaning, sentences “The car won't start” and “My automobile broke down” do not repeat any words, so plain word comparison will say they are not similar. This is why embedding is needed: embedding tells the LLM that both sentences are similar meaning and should have close vectors and high cosine similarity.

Text → Vector

LLM understands vectors, not English words.

Text becomes vector

Text ‘car’

TokensToken IDsEmbedding vectorVcar

Text ‘bus’

TokensToken IDsEmbedding vectorVbus

Pre-training Stage

Gives the Base Model.
In pre-training, you give huge datasets to LLM, e.g. whole internet, whole English library.
Then LLM is asked: given this text, what is the likely next token? If prediction is right, move on. If wrong, LLM re-trains itself.
Here LLM learns associations, reasoning patterns, code patterns, style patterns, language, grammar and facts.
Example: LLM sees multiple texts referring to France as capital of Paris. So when we ask “What is the capital of France?” it predicts next token = "Paris".

Pre-training loop

Ingest Huge datasets

Predict next token

Compare prediction

Update parameters

How does it learn?

Example: take vector of car and vector of bus. LLM sees multiple texts where car and bus have been used in similar context or both have similar patterns. It has seen this hundreds / millions of times.

Car and bus become semantically close

Post-training Stage

Base model knows relations and patterns, but it cannot yet reliably chat in Q/A format. Base model is like a person who knows entire internet but does not know how to talk or assist. It is not trained in customer support or teaching style yet.

Post-training turns a knowledgeable base model into a more helpful assistant.

1. Supervised Fine Tuning (SFT)

After pre-training, we want LLM to follow instructions. Humans show the model tons of high quality Q/A pairs and instruct model to answer in a particular fashion.
For example, User Prompt: “Explain cosine similarity in AI”
Base Model may answer:
1. Cosine similarity is a measure used in mathematics. This article desribes ....
2. OR
3. AI uses cosine similariy. It involves vector ...
In SFT, you instruct the LLM to answer: Cosine Similarity checks whether 2 vectors point in the same direction.

2. Reinforcement Learning / Preference Tuning

Give the user 3 answer variations - A) Concise Style B) Verbose Style C) Bullet Points. Each user may choose A or B or C to be most useful. This is stored as user metadata. Next time AI answers it will choose answer based on your preference.

3. Safety Tuning

Teach LLM what not to do.

Do not leak data.
Do not give harmful instructions.
Do not give malware.
Do not give misleading advice.
Teach LLM to say “I can't help with this”.

4. Domain specific fine tuning

Optional.
Teach LLM legal contracts, C++ coding, financial research.
Teach ideal response format, e.g. answer as { ticket_id - 1234, category - HW issue, resolution - restart laptop }

Parameter

We can think of parameters as the patterns models have learnt during training.

Example: model Llama 70B means Llama model has been trained with 70B parameters.
Parameters are like connections or weights.
They help LLM decide: given this text, what token is likely next? What concepts are related? How should grammar work? What does this sentence mean?

A model parameter is a learned internal number that helps the AI decide what output is most useful / likely for a given input.