M1 — Load the weights

Milestone M1 · the 1.4 GB shows up

Load the weights: prove the file matches the architecture

M0 turned text into token IDs with no weights at all. M1 is where the pile of numbers finally arrives — and the whole milestone is one question: does this file contain exactly the tensors, at exactly the shapes, the architecture implies?

Status: ✅ done. fs inspect models/qwen3-0.6b loads config.json + model.safetensors, derives the expected tensor set from the config, cross-checks the file against it, and prints a shape-first report. The real model checks clean: 311 tensors, 596M logical params. 51 unit + 2 golden tests, clippy clean.

$ fs inspect models/qwen3-0.6b
  ✓ all 311 expected tensors present, shapes match the config
  params: 751,632,384 stored · 596,049,920 logical (the "0.6B")

No forward pass yet — that's M2. This is the handshake that makes M2 safe to write. Its companion is learning 10 · block anatomy (what each tensor is, and how we know); this page is what we built. Previous milestone: M0 · tokenizer.

Useless apart

A model is two files — and you need both

This is the weights-side echo of learning 09. model.safetensors alone is a flat blob — 596M numbers with no grammar; nothing in it says a [2048,1024] matrix is q_proj, or where one layer ends. config.json alone is a recipe with no ingredients. fs inspect is that handshake made executable.

config — the shape weights — the values verdict — they line up

Only the config says what "correct" means — and where that definition comes from (the reference implementation, not memory) is learning 10 §1.

Bottom-up, one helper at a time

The loading stack

Built as small, separately-tested helpers (the M0 cadence). Each reads through the one below it.

Mmap::open

File → memory

Maps the whole file zero-copy via raw POSIX mmap FFI (no libc crate); munmap on Drop. The 1.4 GB never lands on our heap.

SafeTensors::load

Bytes → tensor directory

Reads [u64 len][JSON header][blob] into validated Tensor entries — each a borrowed [s,e) slice of the blob. No weight is ever copied.

Config::load

JSON → dims

Parses the 7 named dimensions + M2/M3 scalars, with no silent defaults — a model we can't fully read fails here, not with wrong shapes in M2.

expected_tensors

Config → the spec

Emits the tensor set the architecture implies: 3 global + 11 per block × L = 311, each with its [out,in] shape.

cross_check

Spec vs. file

Diffs expected against actual: missing / mis-shaped / unexpected tensors become problems; totals the params (stored vs logical).

render_* + run

The report

Prints the dimension legend, the grouped × L table with the in ──▶ out column, and the ✓/✗ verdict.

The safetensors format — three regions

"Reading" the file is: read 8 bytes, parse N bytes of JSON, and now every tensor is a slice of the rest.

length prefix JSON header weight blob (mmap'd, lazy)

parse_tensor_entry validates each entry — end ≤ blob and end−start == shape·dtype.size() — so a self-inconsistent header fails at load, not as a mis-slice in M2. Detail: learning 01.

mmap, not read

The OS maps the file into our address space and pages it in lazily; the weights never hit our heap. Raw FFI, to match the no-hidden-abstraction ethos — learning 06.

bf16 stays lazy

Tensors keep raw bf16 bytes; bf16_to_f32 exists but is only called in M2. Eager conversion would copy 1.4 GB → ~2.8 GB and defeat the mapping — learning 07.

Deriving what should be there

The cross-check, and the gotcha the file taught us

expected_tensors(cfg) is learning 05's shape spec written as code — global embed_tokens [V,H], model.norm [H], lm_head [V,H], plus the 11-tensor block × 28 (the GQA asymmetry [2048,1024] vs [1024,1024], QK-norms, SwiGLU). cross_check diffs it against the file: missing / mis-shaped / unexpected tensors become problems, each naming the offending dim.

Tied ≠ absent. The plan assumed tie_word_embeddings: true ⟹ no lm_head.weight in the file. The real file has one, byte-for-byte identical to embed_tokens. "Tied" is about the math, not the file: a tied export may omit lm_head or ship a redundant copy (Qwen3-0.6B does the latter). Coding from the assumption would have flagged a real tensor as an "extra" and reported a false failure. We caught it only by reading the header — the lesson of learning 10.

So lm_head is optional when tied, and the verdict reports two param counts:

count	value	meaning
stored	`751,632,384`	naive sum over every file tensor — the vocab table stored twice
logical	`596,049,920`	the redundant copy deduped — the "0.6B"
embeddings	`155,582,464`	26.1% of logical — a quarter of the model is the vocabulary

What it prints

The report, on the real model

Every number is derived — the GQA asymmetry, the deduped 596M, the 26.1% — nothing hard-coded. A shape mismatch makes run return not-clean, which the CLI turns into a non-zero exit (so fs inspect can gate a build).

── dimensions (from config.json) ───────────────────────────────────────────
  V  vocab_size             151936   distinct tokens
  H  hidden_size              1024   residual-stream width (the bus)
  L  num_hidden_layers          28   transformer blocks
  d  head_dim                  128   width of one attention head
     num_attention_heads        16   query heads → q width = 16·128 = 2048
     num_key_value_heads         8   kv heads → kv width = 8·128 = 1024   (GQA group 2)
  I  intermediate_size        3072   FFN inner width
  weights are stored [out, in]; read a row as   in ──▶ out   (y = x·Wᵀ)

── tensors ─────────────────────────────────────────────────────────────────
  TENSOR                          DTYPE SHAPE            PARAMS   in ──▶ out
  global
    model.embed_tokens.weight     BF16  [151936, 1024]  155,582,464  id ──▶ H   (row gather)
  each block  × 28   (shown: layer 0)
    input_layernorm.weight        BF16  [1024]                1,024  scale 1024
    self_attn.q_proj.weight       BF16  [2048, 1024]      2,097,152  1024 ──▶ 2048
    self_attn.k_proj.weight       BF16  [1024, 1024]      1,048,576  1024 ──▶ 1024
    self_attn.v_proj.weight       BF16  [1024, 1024]      1,048,576  1024 ──▶ 1024
    self_attn.q_norm.weight       BF16  [128]                   128  scale 128
    self_attn.k_norm.weight       BF16  [128]                   128  scale 128
    self_attn.o_proj.weight       BF16  [1024, 2048]      2,097,152  2048 ──▶ 1024
    post_attention_layernorm.weight BF16 [1024]                1,024  scale 1024
    mlp.gate_proj.weight          BF16  [3072, 1024]      3,145,728  1024 ──▶ 3072
    mlp.up_proj.weight            BF16  [3072, 1024]      3,145,728  1024 ──▶ 3072
    mlp.down_proj.weight          BF16  [1024, 3072]      3,145,728  3072 ──▶ 1024
  final
    model.norm.weight             BF16  [1024]                1,024  scale 1024
    lm_head.weight   (tied)       BF16  [151936, 1024]  155,582,464  1024 ──▶ 151936

── verdict ─────────────────────────────────────────────────────────────────
  ✓ all 311 expected tensors present, shapes match the config
  note: lm_head.weight present but tied — a redundant byte-identical copy of embed_tokens (155,582,464 params counted once)
  params: 751,632,384 stored · 596,049,920 logical (the "0.6B")
  embeddings: 155,582,464 = 26.1% of logical

The milestone's "done" gate

How we know it's right

51 unit + 2 golden integration tests; cargo clippy --all-targets clean. The verification is layered:

Per-helper units hit reality — Mmap::open round-trips raw bytes through the live syscall; SafeTensors::load parses the real 1.4 GB header; Config::load reads the shipped config.json; expected_tensors derives exactly 311 from it.
cross_check synthetic cases (built through the real SafeTensors::load) cover clean/dedup, shape-mismatch, missing-required, unexpected-extra, tied-head-omitted, and tied-head-differs.
run end-to-end on a synthetic model (no assets, fresh-checkout safe) proves both the clean exit and the shape-mismatch exit-1 path.
The reality anchor — the real model cross-checks clean at 311 / 751,632,384 / 596,049,920 / 26.1%, every figure derived from config.json.

Reality checks skip gracefully when the (git-ignored) assets aren't fetched, so a fresh checkout stays green. Next: M2 — forward pass → logits, where these tensors finally compute.

📖 Inference Engineering (Kiely) · §4.2.2 (p.103) 🔧 ds4 · GGUF reference; fs keeps safetensors native, with GGUF optional post-core 🧭 Raschka · report → config → reference implementation

See the anatomy companion learning 10 · block anatomy, the shapes primer learning 05, and the full writeup docs/m1-weights.md.