programdemos/micro-gpt/batch.xtl
The tiny GPT over a batch of names: the first 1,000 names of its training file (makemore's names.txt) scored by the forward pass of micro-gpt.xtl, a name per call under its causal mask; the mean loss per token (the cross-entropy the trainer reports) and how many next letters it predicts right, both as the trainer's Rust model gives them. The batch run for a GPU (docs/gpu-asks.md): a thousand calls of one forward pass, the token ids all that goes in.
The model, read from data/ (just micro-gpt-data), as micro-gpt.xtl reads it
The forward pass, as micro-gpt.xtl has it
ᵘr̲ms : Float -> Float
RMS norm without a gain (microgpt's).
ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
ᵘl̲inear : Num a => a -> a -> a
x times a matrix kept a row per output.
ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
ᵘh̲ead : Int -> (Float, Float) -> Float
Attention head j (of 4): rows 4j-3 .. 4j of the projections.
ᵘh̲ead ← { j hM → r ← (4 × j − 1) + r̲ange 4 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM }
ᵘg̲pt : Int -> Float
Every position's scores over the 27 tokens, for the tokens ids.
ᵘg̲pt ← { ids → n ← t̲ally ids x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n) heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM x1 ← x0 + heads ᵘl̲inear Wo x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2 x2 ᵘl̲inear lm }
A name's score
names : Int
The names as rows of 17 tokens: the start token, the letters, the end token (numbered from 1), then zeros.
names ← 1000 17 r̲eshape f̲loor n̲umbers ⎕N̲GET "data/names.txt"
ᵘs̲core : Int -> Float
For a name's row r: the scores at every token but the last, the tokens that follow as the targets; the sum of -log of each target's probability, the targets predicted right (the likeliest, the first on a tie) and the tokens scored.
ᵘs̲core ← { r → ids ← ('+ r̲/ r > 0) t̲ake r S ← ᵘg̲pt -1 d̲rop ids t ← 1 d̲rop ids ('+ r̲/ r̲avel (v ⁿⁿo̲neHot t) × n̲eg l̲og ⁿⁿs̲oftmax S) c̲at (f̲loat '+ r̲/ (ⁿⁿa̲rgmax S) = t) c̲at f̲loat t̲ally t }
The thousand names: the mean loss per token, the next letters right; the Rust model's numbers
totals : Float
The scores summed over the names, one call each.
(totals, _) ← 1000 '{ (T, i) → (T + ᵘs̲core i s̲elect names, i + 1) } p̲ower (0.0 0.0 0.0, 1)