Failed Star / fs

Learning 01 · where weights live on disk

Model file formats: safetensors vs GGUF

Context: deciding M1's on-disk format · Status: decided

This is the first thing we learned: where do a model's weights actually live on disk, and in what shape? A model is, physically, two things — a pile of numbers (the weights) and a description of how they're arranged (names, shapes, dtypes). A file format is just an agreed way to write those two down. Two formats dominate, built for different jobs.

📖 Inference Engineering · §4.2.2 “Model File Formats” (p.103) 🔧 ds4 · ds4.c “owns GGUF loading”, gguf-tools/ 🧭 Raschka · “Workflow for Understanding LLMs” — working code doesn't lie

HuggingFace's native format

safetensors — three parts, no surprises

This is what Qwen3-0.6B ships as on Hugging Face. It is almost shockingly simple — which is exactly why it's a great first format to parse by hand. The whole file is three concatenated regions:

8 bytes N bytes the rest of the file u64 (LE) = N JSON header name → dtype·shape·offsets raw tensor bytes row-major, contiguous length of →
header length JSON header → the shape table raw tensor blob → the numbers
The entire format: an 8-byte little-endian u64 length, then that many bytes of JSON describing every tensor, then one contiguous blob of raw bytes.
  1. First 8 bytes: a little-endian u64 giving the length N of the header.
  2. Next N bytes: a UTF-8 JSON object. Each key is a tensor name; each value is { "dtype": …, "shape": […], "data_offsets": [start, end] }. (An optional __metadata__ key holds arbitrary string key/values.)
  3. Everything after: one contiguous blob of raw tensor data. Each tensor lives at [start, end) within that blob, row-major (C-contiguous).

That's the entire format. The properties that matter to us:

Weights in original dtype

Qwen3 is bf16 — no quantization, no decoding scheme: the bytes are the numbers. We convert bf16→f32 when we compute; that's trivial.

It is “safe”

Unlike Python pickle/.bin checkpoints there's no embedded code to execute — just data. Hence the name.

Zero-copy / mmap-friendly

Map the file and point tensors straight at the bytes. (ds4 mmaps too; the idea is the same.)

No tokenizer or config

Those live in sibling files: config.json (layers, dims, heads, vocab) and tokenizer.json (vocab + merges, our M0). We need those anyway.

llama.cpp's format · what ds4 uses

GGUF — one self-contained file

GGUF is built for local inference distribution: get a model running from one self-contained file. It carries everything — weights, tokenizer, config, chat template. Its layout is a stack of sections rather than three flat regions:

sectionwhat it holds
header magic "GGUF" (0x46554747) · version (u32) · n_tensors · n_metadata_kv
metadata (key/value) general.architecture, hyperparameters, and the full tokenizer/vocab, chat template, etc. (~13 typed value kinds, arrays supported)
tensor info per tensor: name · n_dims · dims[] · ggml_type (quant kind) · offset
padding pad to alignment
tensor data quantized blocks (Q4_K, Q2_K, IQ2_XXS, …) or f16/f32

The key differences from safetensors:

One file holds it all

Weights and tokenizer and config and chat template. Download one .gguf, run.

Quantization is native

Tensors aren't plain arrays; they're block quant formats — a block of N weights shares a scale (sometimes a min / secondary scale), packed into far fewer bits. Q4_K ≈ 4 bits/weight; Q2_K/IQ2_XXS ≈ 2. This is the reason a huge model fits a laptop — a whole topic of its own (our M5).

More complex to parse

A binary key/value metadata section with many value types, plus the per-format block layouts you must decode to get real numbers.

Evidence: ds4 is GGUF, end to end.
  • ds4.c:5 — “This file is deliberately vertical: it owns GGUF loading…”
  • ds4.c:11 — “Loading is mmap based. The loader parses only the GGUF header, metadata…”
  • ds4.c:579 — #define DS4_GGUF_MAGIC 0x46554747u /* "GGUF", little endian. */
  • ds4.c:1514 — the GGUF_VALUE_* type enum (UINT8…FLOAT32…).
  • download_model.sh pulls only *.gguf; gguf-tools/ builds/quantizes GGUF.
So ds4 has no safetensors path at all — it's a GGUF engine.

The tradeoff, for our goals

safetensors vs GGUF, side by side

safetensorsGGUF
Qwen3-0.6B ships it?✅ natively❌ needs conversion (llama.cpp)
Parse-from-scratch efforttrivial (len + JSON + bytes)meatier (binary KV + quant blocks)
Weights for a first forward passclean bf16, no decodingfine if f16, but quant blocks loom
Includes tokenizer/config?❌ (sibling files)✅ (all in one)
Quantizationnone (add later)native, the whole point
Matches ds4 directly❌✅

Our decision (and why)

safetensors native; GGUF optional

The reasoning is sequencing complexity to the milestone that needs it:

M1 · weights are bytes + a shape table

safetensors makes that idea visceral in an afternoon, and Qwen ships it natively — so there's no tooling detour before we've loaded a single tensor.

The core correctness path stays clean

Real bf16 weights mean our first-ever forward pass isn't also fighting dequantization math while we hunt numerical bugs against the golden vector.

Quantization and GGUF are separate decisions

M7 starts with a benchmark-driven low-bit go/no-go. GGUF may later earn an optional interoperability experiment beside ds4's parser and gguf-tools/; it is not a core-roadmap promise.

Two formats = two distinct lessons — “load raw tensors” vs “interoperate with GGUF” — not wasted work. ds4 remains a GGUF-only reference; adoption by fs is not promised.

Mental model to keep

Numbers + a table describing the numbers

A model on disk = (numbers) + (a table describing the numbers).

  • safetensors writes the numbers plainly. Simple; the bytes are the values.
  • GGUF writes the numbers compressed (quantized) and bundles the tokenizer and config alongside. Self-contained; the bytes need decoding to become values.

The leap from the first to the second — “the bytes need decoding” — is the door into quantization, which is most of what makes local inference possible. We walk through that door deliberately at M5.

To revisit at M5