The one rule

Inner dimensions must match

A matrix multiply A · B is legal iff the inner dimensions match. The shared axis is contracted (summed away); the outer dims survive:

A: (m × k)   B: (k × n)   ⟹   A·B: (m × n)
        └────────┘
        these two k's (the "inner" dims) must be equal — k is then
        "contracted" (summed away); the outer dims m and n survive.

That's it. Every shape question below is an application of this one rule. When a forward pass breaks, 9 times out of 10 it's two ks that didn't match — so we make the ks visible and assert them.

The #1 gotcha

Weights are stored [out, in]

A linear layer computes y = x · Wᵀ. In PyTorch — and therefore in the safetensors file — the weight is stored transposed relative to the data flow, as [out_features, in_features]. The stored shape looks reversed from the arrow because of that Wᵀ:

The residual stream

A fixed-width bus through every block

A transformer block doesn't change the width of the thing flowing through it. Tokens enter as H-wide vectors, every sub-layer reads from and writes back to that same H-wide residual stream, and the block outputs H-wide vectors. For Qwen3-0.6B, H = hidden_size = 1024. Attention and the FFN fan out to wider working widths internally, then project back to H before rejoining the bus:

The legend

Qwen3-0.6B's named dimensions

Every shape in the model is built from these seven numbers, straight from its config.json:

Two of these deserve their own diagram because they're the usual sources of confusion.

Decoupled dimension

head_dim is decoupled from hidden_size

Naively you'd expect num_heads × head_dim == hidden_size. It doesn't here: 16 × 128 = 2048 ≠ 1024. Qwen3 lets the attention working width differ from the residual width. So q_proj projects H ──▶ 16·d, and after attention o_proj projects back 16·d ──▶ H:

Grouped-Query Attention

Query heads share key/value heads

GQA uses fewer key/value heads than query heads. Here: 16 query heads, 8 kv heads → a group size of 16 / 8 = 2 (every 2 query heads share one kv head). That's why q_proj is width 2048 but k_proj/v_proj are width 1024 = 8·d:

Width by width

Walking one block

The full tensor set of a block reads as a chain of in ──▶ out arrows, every arrow obeying the one rule. (norm weights are 1-D scale vectors, not matrices — they multiply element-wise, so their width just states which thing they scale.)

Notice the rhythm: everything leaves the H-wide bus, does work at 16·d / 8·d / I, and comes back to H. If you can see that, you can read any dense transformer's weights.

Where it shows up

How this reads in fs inspect

fs inspect makes all of the above legible at a glance. It prints three parts:

• a dimension legend (the table above), so every shape has a named source;
• a grouped-by-layer tensor table whose last column is the in ──▶ out arrow — one representative block labelled × L, not 28 copies;
• a cross-check that derives the expected tensor set from config.json and diffs it against the file (q/k/v ← H, o ← 16·d, gate/up ← H, down ← I, embed = [V, H], lm_head tied), reporting stored vs logical params and any mismatch.

Run against the real model, the table + verdict read (abridged):

── tensors ─────────────────────────────────────────────────────────────────
  TENSOR                          DTYPE SHAPE            PARAMS   in ──▶ out
  global
    model.embed_tokens.weight     BF16  [151936, 1024]  155,582,464  id ──▶ H  (row gather)
  each block  × 28   (shown: layer 0)
    self_attn.q_proj.weight       BF16  [2048, 1024]      2,097,152  1024 ──▶ 2048
    self_attn.k_proj.weight       BF16  [1024, 1024]      1,048,576  1024 ──▶ 1024
    self_attn.o_proj.weight       BF16  [1024, 2048]      2,097,152  2048 ──▶ 1024
    mlp.gate_proj.weight          BF16  [3072, 1024]      3,145,728  1024 ──▶ 3072
    mlp.down_proj.weight          BF16  [1024, 3072]      3,145,728  3072 ──▶ 1024
    …                                                                (11 per block)
  final
    lm_head.weight   (tied)       BF16  [151936, 1024]  155,582,464  1024 ──▶ 151936

── verdict ─────────────────────────────────────────────────────────────────
  ✓ all 311 expected tensors present, shapes match the config
  note: lm_head.weight present but tied — a redundant byte-identical copy of embed_tokens
  params: 751,632,384 stored · 596,049,920 logical (the "0.6B")
  embeddings: 155,582,464 = 26.1% of logical

Every number here is derived — the GQA asymmetry (2048 vs 1024), the deduped 596M, the 26.1% — nothing hard-coded. That cross-check is the M1 verification, and its asserts carry into M2 so a mis-wired matmul fails loudly instead of producing quiet garbage.