# Testing and verification

Correctness is the backbone of this project. Each milestone should become a
small runnable artifact whose output is checked against a known-good reference.

## The rule

A milestone is not done because the code “looks right.” It is done when:

1. the artifact runs,
2. its output is verified against an oracle or invariant,
3. the verification can be rerun locally,
4. and the result is documented in the milestone notes.

## Kinds of checks

### Unit tests

Use these for small deterministic pieces:

- byte/unicode maps,
- parsers,
- tensor shape calculations,
- math kernels on toy inputs,
- sampling helper functions.

These should be small enough that a failure points directly at one idea. For
Rust modules, prefer light inline tests next to the helper when private details
are the thing being checked.

### Golden-vector tests

Use these when `fs` must match an official implementation:

- tokenizer token IDs,
- logits for fixed prompts,
- greedy continuations,
- CPU vs GPU output.

Golden data should be generated by a one-shot oracle, then committed when it is
small and license-safe. Normal Rust tests should not need Python, network access,
or Hugging Face credentials.

Default CI is deliberately **model-free**: after Cargo fetches locked dependencies,
`cargo fmt --check`, build, tests, and clippy need no model assets, Python, or
runtime network access. Tests requiring git-ignored model assets are explicit
`#[ignore]`d local-Mac checks. Future Metal correctness and performance checks are
also local-Mac only; orb checks are non-authoritative, and standard GitHub runners
are not promised to execute Metal.

Run the complete asset-backed suite on the target Mac with:

```sh
uv run --directory scripts --frozen fetch_model.py --weights
uv run --directory scripts --frozen verify_golden.py
cargo test --locked --release -- --ignored
```

To check only the committed forward fixtures (no downloaded model required), run
`uv run --directory scripts --frozen verify_golden.py --fixtures-only`. The
verifier is read-only: it checks manifest shape/byte invariants and SHA-256 hashes
but never regenerates or rewrites fixtures.

### CLI smoke tests

Use these for milestone artifacts:

- `fs tokenize "hello world"`,
- `fs inspect model/`,
- `fs logits "The capital of France is"`,
- `fs generate "..."` (and optional later `fs chat "..."`).

These checks verify that the thin CLI path still reaches the engine, but most
correctness should live in library tests.

### Benchmarks / measurements

Use these when a milestone promises speed or memory improvements:

- M4 KV cache: prefill/decode tokens per second, cached vs uncached,
- M6 Metal optimization: profiled before/after speed with output agreement,
- M7 quantization (if the go decision is made): memory and quality deltas.

Benchmarks should report enough context to be meaningful: model, prompt length,
hardware, dtype/quantization, and relevant settings.

## M0 tokenizer plan

The tokenizer should be tested bottom-up:

1. Inline unit tests: `build_byte_encoder` / decoder inverse.
2. Inline unit tests: `build_vocab` and `build_merges` parse `tokenizer.json`'s
   `model.vocab` / `model.merges` (array form + legacy `"left right"` form).
3. Inline unit tests: `bpe` matches expected pieces/IDs for controlled chunks.
4. Inline unit tests: `pretokenize` matches Qwen's regex behavior on tricky strings.
5. Inline unit tests: special-token carving in `encode` + decode of special ids.
6. Light integration test: `encode` matches [`tests/golden/tokenizer.json`](../tests/golden/tokenizer.json).
7. Light integration test: `decode(encode(text)) == text` for all golden cases.
8. Light integration test: special tokens (`<|im_start|>` → 151644, carving, round-trip).
9. CLI smoke test once the library path is correct.

The golden tokenizer fixture is generated by:

```sh
uv run --directory scripts --frozen gen_golden.py
```

That script uses Hugging Face's `tokenizers` library as the official oracle and
writes committed JSON so `cargo test` can stay Rust-only.

## Later milestones

- **M1 weights:** config values, tensor names, shapes, dtypes, counts, and a few
  checksums match the downloaded model.
- **M2 logits:** the fixed prompt `"The capital of France is"` matches the official
  implementation at four layered fp32 checkpoints: embedding output, block-0
  output, final-norm output, and last-position logits. The manifest records exact
  shapes, axis meanings, checkpoint boundaries, pinned model revision, source-
  asset hashes, and canonical runtime; raw little-endian f32 files keep the
  complete vectors compact. Compare with `atol = 1e-4`, `rtol = 1e-4` so failures
  bisect to the first divergent stage. Regenerate with:

  ```sh
  uv run --directory scripts --frozen fetch_model.py --weights
  uv run --directory scripts --frozen gen_forward_golden.py
  ```
- **M3 generation:** `fs generate` greedy decoding reproduces a reference
  continuation before sampling is introduced.
- **M4 KV cache:** cached and uncached decode produce the same tokens; cached
  decode is measurably faster.
- **M5 Metal:** end-to-end GPU and CPU paths agree within tolerance on the local Mac.
- **M6 Metal optimization:** profile first; optimized/fused and baseline paths
  agree, and benchmarks describe where speed was gained.
- **M7 quantization:** first record a benchmark-driven go/no-go; if go, output
  quality stays within a documented budget while memory drops.

## Open testing decisions

- Whether later, much larger golden tensors need chunked comparison instead of
  M2's manifest + raw-f32 format.