Learning 10 · what each tensor is, and how we know
Learning 05 told us the shapes. This note answers the two questions it leaves open — the ones you hit the moment you try to list a model's tensors: what is each one? and where does that knowledge come from?
Failed Star lm_head. The source of
truth is the file's header and the reference implementation.
Part 1 · where the architecture comes from
The tempting shortcut is "I know transformers, I'll write the blocks
from memory." Two problems. Memory is a lossy cache and it's already been
wrong — it said "tied embeddings ⟹ no lm_head.weight"; the real
file has one (Part 3). And you cannot derive an architecture from
config.json — the thesis of
learning 09: a config parameterizes a structure
that lives in code. Nothing in it announces q_norm.
So where does it come from? From the model's creators, down a chain of trust — each link verified against the one above by reproducing its numbers:
modeling_qwen3.py
opens with Copyright 2025 The Qwen team, Alibaba Group and the HuggingFace Inc.
team. — the creator and the reference library named as co-authors. The
chain of trust isn't a metaphor; it's a copyright line. (The file is even auto-generated
from modular_qwen3.py, which the model team often contributes directly.)
transformers is for us — three roles, kept distinctRead it to learn what tensors exist
and how they're wired (the names are the PyTorch attribute paths:
self.q_proj → …self_attn.q_proj.weight). Run
it at M2 to dump golden logits and assert our output matches.
We copy the architecture, never the code. The weights are only meaningful under Qwen3's exact computation, so our math must match — but we express it in our own tight Rust and prove equivalence with numbers, not by diffing source.
| source | answers | for fs inspect |
|---|---|---|
| the file header | what tensors exist + shapes | ground truth we cross-check against |
| the reference code | how each tensor is used (the forward pass) | how we know what to implement (M2) |
| the config | the dims that size everything | learning 09 |
So expected_tensors derives the spec from the config,
then diffs it against the header. Any disagreement is a loud failure at
load time — not quiet garbage in M2.
Part 2 · tensor by tensor
Straight from the real header: 311 tensors = 3 global + 11 per
block × 28 layers. A block is two sub-layers — attention, then
an MLP — each wrapped as norm → sub-layer → add back to the residual stream.
That wrapping is why you see two layernorms per block plus two more norms inside
attention:
| tensor | shape | what it is |
|---|---|---|
model.embed_tokens.weight | [V, H] | the vocabulary table — a token id is a row index (a gather, not a matmul). See learning 04. |
model.norm.weight | [H] | one final RMSNorm on the output stream before the head. |
lm_head.weight | [V, H] | the output projection: final H-vector → V logits. Tied to embed_tokens — Part 3. |
| tensor | shape | role |
|---|---|---|
input_layernorm | [H] | RMSNorm the bus before attention (a scale vector, not a matrix). |
self_attn.q_proj | [2048, 1024] | the query: "what am I looking for?" (16 heads × 128) |
self_attn.k_proj | [1024, 1024] | the key: "what do I offer as a match?" (8 kv heads × 128) |
self_attn.v_proj | [1024, 1024] | the value: "what do I carry if matched?" |
self_attn.q_norm | [128] | QK-norm on each query vector before scoring — the one you can't guess. |
self_attn.k_norm | [128] | QK-norm on each key vector. |
self_attn.o_proj | [1024, 2048] | project the concatenated head outputs back onto the H-wide bus. |
post_attention_layernorm | [H] | RMSNorm the bus before the MLP. |
mlp.gate_proj | [3072, 1024] | SwiGLU gate: H → I, then squashed by SiLU. |
mlp.up_proj | [3072, 1024] | SwiGLU value: H → I, multiplied by the gated activation. |
mlp.down_proj | [1024, 3072] | I → H, back onto the bus. |
q_norm.weight / k_norm.weight. No config field announces them.
self.q_norm = Qwen3RMSNorm(self.head_dim, eps=config.rms_norm_eps) # and k_norm
...
query_states = self.q_norm(self.q_proj(hidden_states)...) # applied before attentionSwiGLU is why there are two "up" projections. A
vanilla FFN is down(act(up(x))); SwiGLU splits the up into a gate and a value:
MLP(x) = down_proj( SiLU(gate_proj(x)) · up_proj(x) )
└── the gate ──┘ └─ the value ─┘ (· = element-wise)Two [I,H] tensors + one [H,I] is the SwiGLU
signature; a single up-projection would be a plain FFN.
Part 3 · read, don't assume
lm_head surpriseconfig.json has tie_word_embeddings: true.
"Tied" means the output projection is the embedding table — the
same weights, used twice. It's a statement about the math, not the file.
Memory said "tied ⟹ the file omits lm_head.weight." The header
says otherwise — lm_head.weight [151936, 1024] is present, and its raw bytes
are byte-for-byte identical to embed_tokens. Qwen3-0.6B is
tied and ships a redundant copy. The consequences:
| count | value |
|---|---|
| stored tensors | 311 |
| stored params (naive sum) | 751,632,384 (~751M) |
| logical params (dedup the tied copy) | 596,049,920 — the "0.6B" |
| embedding share of logical | 26.1% |
A quarter of the model is the vocabulary table — stored twice. So
expected_tensors marks lm_head optional when tied,
and cross_check reports stored and logical separately,
flagging the redundant copy as a note, not a failure. Had we coded from the
memory-assumption, the cross-check would have called a real tensor an "extra" and reported
a false failure on the real model.
fs inspect models/qwen3-0.6b now prints exactly this
reconciliation — the deduped 596M, the redundant-copy note, the
26.1% share — all derived from config.json and checked
against the file, none of it hard-coded. See the abridged output in
M1 · load the weights.
[out,in], GQA, tied-embedding math).embed_tokens as a gather, and the three senses of "embedding."fs inspect as the config↔weights handshake.modeling_qwen3.py
— the reference forward pass (spec + M2 oracle) ·
📄 Qwen3 Technical Report — the design
rationale · Full note:
docs/learnings/10-transformer-block-anatomy.md.