Learning 09 — config.json

Learning 09 · the model's blueprint

config.json: a dozen numbers that stand next to 1.4 GB

Context: M1, reading config.json into Config · Status: living

We just wrote Config::load. It reads a ~1 KB text file and hands back thirteen numbers — a strange thing to sit beside a 1.4 GB pile of weights and call an equal partner. But it is one, and the reason is more surprising than it looks.

The question this note exists to answer: how can a config file possibly describe a whole neural network? The honest answer — the spine of everything below — is it can't, and it doesn't. It parameterizes one.

📖 Inference Engineering (Kiely) · §4.2 (p.103) 🔧 ds4 · GGUF has no separate config — hyperparams live inside the file 🧭 HF PretrainedConfig · architectures → AutoModel

Useless apart

A model is (at least) two files, and you need both

model.safetensors alone is a flat blob: 596M numbers with no grammar. Nothing in it says a [2048,1024] matrix is q_proj, or that there are 28 blocks, or where one layer ends. config.json alone is the opposite — a recipe with no ingredients: the shape of a network with no learned values in it. Fold one into the other and you have something that runs.

config — the shape weights — the values

M1's fs inspect is this handshake: it proves the tensors in the .safetensors file are exactly the ones the config implies, at the shapes it implies. A checksum can't do that — only the config says what “correct” means.

Twenty-six keys, three piles

What's actually in Qwen3-0.6B's config

The fields sort into three groups — and the sorting itself is the lesson.

Pile A — the dimensions we build every shape from

These are the learning 05 legend; every weight matrix's shape is a product of them. Get one wrong and fs inspect lights up.

field	value	symbol
`vocab_size`	151936	`V`
`hidden_size`	1024	`H`
`num_hidden_layers`	28	`L`
`head_dim`	128	`d`
`num_attention_heads`	16	—
`num_key_value_heads`	8	—
`intermediate_size`	3072	`I`

Pile B — scalars we parse and spend later

field	value	where it's spent
`rms_norm_eps`	1e-6	RMSNorm's divide-by-zero guard (M2)
`rope_theta`	1000000	RoPE base frequency (M2)
`tie_word_embeddings`	true	reuse embeddings as `lm_head` — changes the expected tensor set (M1)
`bos_token_id` / `eos_token_id`	151643 / 151645	sequence markers (M3)
`max_position_embeddings`	40960	context-length ceiling

Pile C — fields we ignore, and exactly why that's safe

A field is safe to skip for one of two reasons — it's training-only, or its value happens to match what we hardcoded:

field	value	why we can skip it
`architectures`	`["Qwen3ForCausalLM"]`	names the code class to build. We are that class — see below.
`model_type`	`"qwen3"`	HF registry key that resolves to the same class.
`hidden_act`	`"silu"`	we hardcode SwiGLU; if this said `"gelu"` our engine would be wrong, not reconfigurable.
`torch_dtype`	`"bfloat16"`	a hint; we read each tensor's dtype from the safetensors header instead (authoritative).
`attention_bias`	false	q/k/v/o have no bias — matches our bias-free `Linear`.
`attention_dropout` · `initializer_range`	0.0 · 0.02	training-only; inference never touches them.
`rope_scaling`	null	no long-context frequency scaling; if set we'd bend RoPE.
`sliding_window` · `use_sliding_window`	null · false	sliding-window attention off; full causal attention is correct.
`use_cache` · `transformers_version`	true · "4.51.0"	an M4 decision regardless · pure metadata.

A quiet trap. Several Pile-C fields are safe only because of their value. Flip attention_bias, rope_scaling, or sliding_window and the architecture silently changes under us while our code keeps computing the old one. The fail-loud move — a good future hardening of Config::load — is to assert these, not skip them. Ignoring a field and asserting a field look identical until the day the value differs.

The spine

How can ~13 numbers describe a network? They don't.

They configure code that already knows the network. Look at the field doing the heavy lifting:

"architectures": ["Qwen3ForCausalLM"]

That's not data — it's a pointer to a class, a ~1500-line file in the transformers library whose body is the architecture: embed → (RMSNorm → RoPE → GQA attention → residual → RMSNorm → SwiGLU → residual) × L → RMSNorm → tied lm_head. The wiring, the mask, the order of operations, which norm, which activation — all of that lives in code. The config only fills in the free numbers of a structure that's already fixed.

config.json is not a blueprint of the network. It's the arguments to a constructor whose body is code. architectures names the constructor; the other fields are its dials.

So the “two files” story is really three things, and one is usually invisible:

code — what we're building config — the dials weights — the values

Most people never see the code column — everyone shares the same transformers — so “a model” collapses to config + weights. We don't get that luxury: fs is the code column. That's why we can ignore hidden_act: we baked SwiGLU into our own source.

Then why does it feel like enough?

Because the field standardized. Modern LLMs converged, hard, on one template: the decoder-only transformer block, stacked L times. Within it, models differ almost entirely in (1) the ~7 dimension numbers and (2) a handful of “which variant” switches — norm type, activation, positional scheme, attention flavor (MHA → GQA → MLA). A dozen numbers span the whole current family because the family is a monoculture — a fact about this moment in the field, not a law of neural nets.

So is the schema flexible enough for new architectures?

Variants inside the family

MoE, MLA, a new RoPE scaling, a sliding window: new config fields + new code branches. HF configs are open JSON dicts, so adding num_experts or q_lora_rank is trivial — but the field is inert until code reads it. The schema grows by accretion, one field per capability.

Genuinely new families

State-space models (Mamba), diffusion LMs, RNN revivals: a different vocabulary entirely — Mamba has d_state, d_conv, expand and no num_attention_heads. The JSON container is universal; the words inside are architecture-specific and mean nothing without that architecture's code.

It's flexible because it's barely a schema at all — an open key/value dict that delegates every structural decision to code. That's not the config being powerful; it's the config being humble. A config can never describe an architecture nobody has coded yet. It's “fill in the blanks,” never “describe anything.” The surprise you felt is the right instinct: the code explains the network; the config just says how big.

Mental model. Config = the dials. Code = the wiring. Weights = the learned values. You need all three to run a model; config.json + model.safetensors are only two of them, and they're only enough because the third — the architecture code — is assumed. On this project we're building that third thing, so we see the whole machine. architectures names it; everything else in config.json is just its arguments.

Cross-links

Learning 01 · safetensors vs GGUF — the “a model is two things” split; note GGUF folds the config into the weight file as metadata.
Learning 05 · reading shapes — Pile A in depth: how the seven dimensions build every tensor's shape ([out,in], head_dim decoupling, GQA).
Milestone M1 · load the weights — fs inspect as the config↔weights handshake; docs/m1-weights.md (owed) states “need both” from the weights side.
➡ M2 · forward pass — where Pile B's rms_norm_eps / rope_theta and the hardcoded SwiGLU / causal mask (the code half) finally run.
🔧 src/config.rs — Config, the typed extractors, and the fail-loud “no silent defaults” stance this note argues for extending to Pile C. · Full note: docs/learnings/09-config.md.