sourcedemos/micro-gpt/batch.xtl

1⍝!/usr/bin/env xetal 2⍝│ The tiny GPT over a batch of names: the first 1,000 names of its 3⍝│ training file (makemore's names.txt) scored by the forward pass of 4⍝│ micro-gpt.xtl, a name per call under its causal mask; the mean loss 5⍝│ per token (the cross-entropy the trainer reports) and how many next 6⍝│ letters it predicts right, both as the trainer's Rust model gives 7⍝│ them. The batch run for a GPU (docs/gpu-asks.md): a thousand calls 8⍝│ of one forward pass, the token ids all that goes in. 9 10"nn:" u̲se< "NN" 11"nm:" u̲se< "Norm" 12"at:" u̲se< "Attention" 13 14⍝━━ The model, read from data/ (just micro-gpt-data), as micro-gpt.xtl reads it 15chars ← -1 d̲rop ⎕N̲GET "data/chars.txt" 16v ← 1 + t̲ally chars 17wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt" 18wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt" 19lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt" 20Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt" 21Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt" 22Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt" 23Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt" 24fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt" 25fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt" 26 27⍝━━ The forward pass, as micro-gpt.xtl has it 28⍝│ RMS norm without a gain (microgpt's). 29ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x } 30⍝│ x times a matrix kept a row per output. 31ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W } 32⍝│ Attention head j (of 4): rows 4j-3 .. 4j of the projections. 33ᵘh̲ead ← { j hM → 34 r ← (4 × j − 1) + r̲ange 4 35 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM 36} 37⍝│ Every position's scores over the 27 tokens, for the tokens ids. 38ᵘg̲pt ← { ids → 39 n ← t̲ally ids 40 x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe 41 hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n) 42 heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM 43 x1 ← x0 + heads ᵘl̲inear Wo 44 x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2 45 x2 ᵘl̲inear lm 46} 47 48⍝━━ A name's score 49⍝│ The names as rows of 17 tokens: the start token, the letters, the 50⍝│ end token (numbered from 1), then zeros. 51names ← 1000 17 r̲eshape f̲loor n̲umbers ⎕N̲GET "data/names.txt" 52⍝│ For a name's row r: the scores at every token but the last, the 53⍝│ tokens that follow as the targets; the sum of -log of each target's 54⍝│ probability, the targets predicted right (the likeliest, the first 55⍝│ on a tie) and the tokens scored. 56ᵘs̲core ← { r → 57 ids ← ('+ r̲/ r > 0) t̲ake r 58 S ← ᵘg̲pt -1 d̲rop ids 59 t ← 1 d̲rop ids 60 ('+ r̲/ r̲avel (v ⁿⁿo̲neHot t) × n̲eg l̲og ⁿⁿs̲oftmax S) c̲at (f̲loat '+ r̲/ (ⁿⁿa̲rgmax S) = t) c̲at f̲loat t̲ally t 61} 62⍝ -- end of the core ------------------------------------------ 63 64⍝━━ The thousand names: the mean loss per token, the next letters right; the Rust model's numbers 65⍝│ The scores summed over the names, one call each. 66(totals, _) ← 1000 '{ (T, i) → (T + ᵘs̲core i s̲elect names, i + 1) } p̲ower (0.0 0.0 0.0, 1) 67mean ← (1 s̲elect totals) ÷ 3 s̲elect totals 68(f̲loat f̲loor 0.5 + 10000.0 × mean) ÷ 10000.0 69f̲loor 2 3 s̲elect totals 70expected ← n̲umbers ⎕N̲GET "data/names-loss.txt" 711e-9 > a̲bs mean − 1 s̲elect expected 72(f̲loor 2 s̲elect totals) = f̲loor 2 s̲elect expected