microgpt in MLPL
Andrej Karpathy's microgpt.py trains a GPT on 32,033 names and invents new ones, in ~200 lines of dependency-free Python: "the complete algorithm; everything else is just efficiency." This site compares it with a Rust port and three ports to sw-MLPL, an array language with autograd built in. Each port has a literate program you can read with its output and equations.
One algorithm, five programs
Python: microgpt.py
A scalar autograd class, a 1-layer, 4-head GPT, Adam and a sampler, over lists of floats.
149 code lines · 64.4 s · loss 2.28
Rust: microgpt-rs
The same algorithm and shape: a tape-based scalar autograd, no crates, compiled.
390 code lines · 0.589 s · loss 2.36
microgpt.mlpl
microgpt.py line by line with hand-written layers. With --rs-parity it replays
microgpt-rs's random stream and its output is byte-identical to the Rust.
230 code lines · 0.718 s · loss 2.37
aline, garien, anisn, alilia, thayn
microgpt-idiomatic.mlpl
The same data regime, built from the Model DSL: embed, causal_attention,
residual, chain, adam over models, sample.
Faster than compiled Rust.
48 code lines · 0.468 s · loss 2.47
jiafini, tamesin, jali, kair, tasen
microgpt-compact.mlpl
The smallest honest version: the corpus as one token stream in 16-token windows, one DSL chain, KV-cached sampling.
32 code lines · 0.539 s · loss 2.56*
an, cren, yn, arialilin
Wall time for the whole program (load, train 1000 steps, sample 20 names) on an Apple M1 Max, median of 7 runs (CPython: one run). Loss: mean over the last 100 training steps. *The compact variant trains on 16-token windows, so its loss is not strictly comparable. Code lines exclude comments, blank lines and docstrings.
Results
| implementation | code lines | wall time | vs Rust | params | loss |
|---|---|---|---|---|---|
| microgpt.py (CPython 3.14) | 149 | 64.4 s | 109x | 4,192 | 2.28 |
| microgpt-rs (release) | 390 | 0.589 s | 1.00x | 4,192 | 2.36 |
| MLPL faithful | 230 | 0.718 s | 1.22x | 4,192 | 2.37 |
MLPL faithful --rs-parity | 344 | 1.406 s | 2.39x | 4,192 | = Rust, byte for byte |
| MLPL idiomatic | 48 | 0.468 s | 0.79x | 4,299 | 2.47 |
| MLPL compact | 32 | 0.539 s | 0.92x | 4,043 | 2.56* |
The idea in one expression
Where microgpt.py spends a class on autograd and loops over scalars, the idiomatic MLPL model is one expression, and training is one call per step:
body = chain(rms_norm(d),
residual(chain(rms_norm(d), causal_attention(d, 4, 3))),
residual(chain(rms_norm(d), linear(d, 4 * d, 4), relu_layer(), linear(4 * d, d, 5))),
linear(d, V, 6));
train 1000 {
adam(cross_entropy(u:logits(inp), tgt), [tok, pos, body], 0.01 * (1 - step / 1000), 0.85, 0.99, 1e-8)
};
Literate programs
Each is an Org document run through ob-mlpl: every block's output is real, each section
states its math, and the model's functions carry their equations as @formula annotations. The
program blocks tangle to a script whose output is checked against that variant's baseline, so the prose
cannot drift from the code.
1. Compact
Start here: the shape of an MLPL language model in 32 lines.
2. Idiomatic
How the Model DSL composes, and what its defaults change.
3. Faithful
Every equation spelled out and checked: masked attention equals the KV-cache loop; gradients equal finite differences; parity with Rust.
What we learned
- Array primitives are the efficiency. The idiomatic MLPL interpreter run beats compiled Rust because a DSL layer is one native array op, while the Rust port keeps microgpt's per-scalar tape.
- Exact cross-language parity is possible. microgpt-rs's SplitMix64 RNG, reimplemented in pure MLPL with 16-bit limbs, makes the faithful port's output byte-identical to the Rust.
- Readability and speed trade off case by case. Hand-written equations vs DSL layers, a mask built inside the loss vs passed in, per-step reads vs pre-encoding: each choice is measured, so it can be made on the numbers (comparison, section 5).
- The interpreter shapes the code. Every
u:call copies the globals and large reads copy arrays, so the data is pre-encoded and the corpus expunged before training. - What would make MLPL shorter still:
format,gather/slice, destructuring, param records, copy-on-write values, a cacheable position layer, and compiled inference. See the requests and the idioms guide.