The why behind the build
Bite-sized notes on things we figured out along the way — the reasoning
behind a decision, and background worth keeping. Each one cross-links the three sources
(the book, ds4, Raschka). They complement the milestone writeups and the
big-picture map.
Failed Star docs/learnings/*.md notes — same content, with the ASCII diagrams promoted
to real SVG. tools/sync-check.sh flags when a page drifts from its source.
The notes so far
LEARNING 01
Where a model's weights actually live on disk. Why we parse safetensors now (trivial: length + JSON + bytes); GGUF remains optional interoperability after core.
2026-06-13 · M1 format choice
LEARNING 02
What a radix tree is — and why ds4's tokenizer is actually a hash table,
not a trie. A data-structure detour worth taking once.
2026-06-14 · M0 background
LEARNING 03
BPE learned once (training) vs replayed (inference), and what it means to “match a model.” The heart of M0's tokenizer.
2026-06-14 · M0 tokenizer
LEARNING 04
The three senses of “embedding,” and why an embedding model is a full forward pass, not a lookup.
2026-06-14 · concept
LEARNING 05
How the dimensions line up: the [out, in] convention, the residual
stream, head_dim decoupled from hidden_size, and GQA.
2026-06-24 · M1 weights
LEARNING 06
Turning a 1.4 GB file into memory, zero-copy — the mmap/munmap
FFI without a wrapper crate, contained in one RAII type.
2026-06-24 · M1 weights
LEARNING 07
The weights' number format: bf16 = fp32's top 16 bits, so widening is a free shift — and why we do it lazily to avoid doubling memory.
2026-07-06 · M1 weights
LEARNING 09
The model's blueprint: why you need it and the weights — and the twist that a config doesn't describe a network, it parameterizes one whose structure lives in code.
2026-07-14 · M1 weights
LEARNING 10
What each tensor is — q/k/v/o, QK-norm, SwiGLU, lm_head — and the
deeper question: where architecture knowledge comes from. The provenance chain, and why
we read the header and reference, not memory.
2026-07-14 · M1 weights
LEARNING 08
How shape, layout, and strides become flat offsets — from a real embedding gather
through matmul and the no-physical-transpose payoff of [out,in].
2026-07-26 · M2 forward pass
The Markdown sources live in
docs/learnings/;
the big-picture map is the home page.