Failed Star / fs

Interactive previews

Diagrams

Live, in-browser diagrams. Each one previews a concept the engine will implement.

Concept explorers, not model output. M0–M2 are implemented in Rust; sampling (M3) and KV caching (M4) are still ahead. These browser toys illustrate selected operations, not the full Qwen computation.

Milestone M0 · the first and last thing every prompt touches

Tokenizer: text becomes numbers

Before the model sees anything, text is split into integer token IDs from a learned vocabulary — common chunks become one token, rare ones split into pieces. Type below and watch it tokenize. This is a character-level merge toy with invented IDs and simplified whitespace. It uses JavaScript UTF-16 units, not Qwen's UTF-8 byte mapping, regex, or vocabulary. For real IDs, use the M0 CLI.

What you’re seeing

  • Each chip is one token; the small number is its illustrative ID, not an actual Qwen vocabulary index.
  • Spaces are tokens too (the dim ␣ chips). Whitespace and casing change the tokens, which is why they matter.
  • The toy replays a fixed merge-priority list. Real byte-level BPE starts from encoded bytes, not JavaScript characters, and also replays learned priorities; it does not count pair frequency in your prompt.
📖 Inference Engineering (Kiely) · §2.2 (p.46) 🔧 ds4 · ds4.c (hash-table vocab)

Milestone M3 · the last step of every token

From logits to a token: sampling

The model’s forward pass ends with a logit (a raw score) for every token in the vocabulary. To sample the next token we turn those scores into probabilities with softmax, then choose. Three knobs shape that choice — drag them and watch the distribution move. The math here is the real softmax; only the logits are a fixed toy example.

The cat sat on the ___

What each knob does

  • Temperature divides the logits before softmax. Low (→0) sharpens toward the single top token (greedy); high (→2) flattens the distribution so unlikely tokens get a real chance. Greedy selection itself is argmax, with no division by zero or softmax.
  • Top-k keeps only the k highest-probability tokens and discards the rest (dimmed). A hard cap on how many candidates survive.
  • Top-p (nucleus) keeps the smallest set of tokens whose probabilities reach at least p, after top-k renormalization — an adaptive cutoff that keeps more candidates when the model is unsure, fewer when it’s confident.

Order here: temperature → top-k → top-p → final renormalization. The bars and percentages show final sampling probabilities (100% before display rounding); cut tokens have probability zero. Sample a token draws from that distribution. M3 will explicitly test the engine's filtering contract rather than treating this toy as an oracle.

📖 Inference Engineering (Kiely) · §2.2 (p.46) 🔧 ds4 · softmax.metal, argsort.metal

Milestone M2 · the heart of the transformer

Attention: every token looks back

Inside each transformer block, a token builds its next representation by looking at the tokens before it and taking a weighted average of them. The weights come from how well a token’s query matches each other token’s key (Q·Kᵀ → softmax). Pick a token to see what it attends to. The softmax is real; the token vectors are a fixed toy example.

query token →

What you’re seeing

  • The chosen query token is compared against every token’s key; softmax turns those scores into weights that sum to 100% — shown as the bar under each chip and in the list.
  • The token’s output is the weighted sum of the value vectors: it has now mixed in information from whatever it attended to.
  • Turn off the causal mask to let a token look at later words. During generation we keep it on — token t can’t depend on words that don’t exist yet.

Work through the actual vectors, matrices, and numbers →

📖 Inference Engineering (Kiely) · §2.2.3 (p.52) 🔧 ds4 · flash_attn.metal, softmax.metal

Milestone M4 · why decode isn’t quadratic

The KV cache: don’t recompute the past

To produce the next token, attention needs the keys and values of every earlier token. Recomputing them at every step would be quadratic waste — so we cache each token’s K/V once and reuse it. Step through generation and watch the difference.

What you’re seeing

  • Each cell is one token’s cached K/V. The glowing cells are the ones actually computed this step.
  • With the cache on, each decode step computes K/V for just one new token. With it off, every step recomputes the whole sequence.
  • Watch the totals diverge: cached work grows linearly, uncached work grows quadratically. That gap is what keeps long generations practical.
📖 Inference Engineering (Kiely) · §5.3 Caching (p.136) 🔧 ds4 · ds4_kvstore.c, dsv4_kv.metal

On the way

Diagrams we want to build next

These arrive as their milestones land — see the abstraction ladder for where each one sits.