Failed Star / fs

Learning 04 · what “embedding” actually names

“Embedding model” ≠ the token-embedding table

Context: untangling what “embeddings” means · Status: background — not on our inference path, but a common confusion worth pinning down

The word “embedding” gets attached to at least three different things. The confusion that matters: “when I use an embedding model for search/RAG, am I just reading rows out of the learned table inside an LLM?” Short answer — no. This note pins down why.

This is background. Our engine only ever uses sense #1 below — the input lookup table (studied at M2). Embedding models (sense #3) are a separate kind of model we are not building. Knowing the difference just keeps the vocabulary straight.
📖 Inference Engineering · §2.1 — token embeddings as the input layer 🔧 ds4 · token_embd.weight (ds4.c:3033), gathered by metal/get_rows.metal 🧭 Lineage · word2vec/GloVe → sentence-transformers / BGE / E5 / text-embedding-3

One word, three referents

Three things called “embedding”

Drawn apart, the three are easy to keep straight — the split that matters is lookup vs activation vs output, and context-free vs context-aware:

1 · Token embeddings — the lookup table

Shape [vocab, d_model], one vector per token. A static lookup: token 15339 → row 15339, the same vector every time, regardless of neighbors — context-free. It is the network's input layer. The only sense our engine uses.

2 · Contextual embeddings — the hidden states

The vectors between layers, after attention has mixed in surrounding context. Context-dependent: “bank” gets a different vector in “river bank” vs “bank account.” They are activations computed by a forward pass — not stored anywhere.

3 · Text / sentence embeddings — the model's output

One fixed-size vector for an entire sentence or document — for semantic search, RAG, clustering, dedup. It is the output of running text through a whole network and then pooling.

When you call an “embedding model” (text-embedding-3, BGE, E5, sentence-transformers, …) you get #3 — not the table from #1.

Not a lookup — a forward pass

What an embedding model actually does

For each input text it runs a full forward pass. The learned token table is in there — it's the front door — but the vector you receive is the product of the entire network, squashed into one:

sense #1 — a lookup (context-free) token 15339 row 15339 of [vocab, d_model] same vector every time sense #3 — a forward pass + pooling (context-aware) text tokenize + table gather transformer attention mixes ctx pool mean / CLS / last one vector (#3) + L2-normalize the #1 lookup, as the INPUT layer meaning comes from the whole forward pass — not from the raw input rows.
input / output stages of the network
The token table is the front door of the embedding model — not the room you walk out with.

Two more things distinguish a real embedding model from a raw LLM's table:

Different training objective

A generative LLM trains on next-token prediction. An embedding model is trained with a contrastive objective (InfoNCE / cosine loss): shown (query, relevant doc, irrelevant doc), it pushes relevant vectors together and irrelevant ones apart. That is what makes its output geometry good for cosine comparison — a raw LLM's token rows are not, off the shelf.

Often a different architecture

Many embedding models are smaller, encoder-only / bidirectional (BERT-style), specialized for this job — not decoder LLMs at all.

Where the “pull the table out” intuition is right

That intuition describes the previous generation

“Train a net, throw it away, keep the table” perfectly describes word2vec and GloVe (~2013). Their entire output was a static word→vector table: train a shallow net on a context-prediction task, discard the net, keep the table. A lookup was a pure table read — context-free, exactly sense #1. Modern sentence-embedding models replaced that because the static table has hard limits:

word2vec / GloVe (old)embedding model (modern)
What you usethe table itself (a lookup)the output of a forward pass
Context-aware?No — one vector per word, foreverYes — same word, different vector by context
Word orderignoredcaptured (attention)
Granularityper wordper sentence / document
Trained forpredicting nearby wordssimilarity (contrastive)
Practical upshot. You could build a cheap text embedding by averaging the token-embedding rows of a sentence (a “bag of embeddings”) — people did exactly that with word2vec. It's a weak baseline: it discards word order and context (“dog bites man” = “man bites dog”). Modern models win because they run the full contextual network and are trained for similarity.

The one to keep

Mental model

Token embedding

A lookup in the model's input table; context-free; one per token; it's the input. The only sense our inference engine touches.

Embedding model

A forward pass + pooling over a separate, similarity-trained network; context-aware; one per text; it's the output.

Using an embedding model means running a neural network over your text and pooling its output — not reading rows from the token table we study at M2. That table is merely the first layer of the machine doing the work.