sourcedemos/micro-gpt/batch.xtl
1⍝!/usr/bin/env xetal
2⍝│ The tiny GPT over a batch of names: the first 1,000 names of its
3⍝│ training file (makemore's names.txt) scored by the forward pass of
4⍝│ micro-gpt.xtl, a name per call under its causal mask; the mean loss
5⍝│ per token (the cross-entropy the trainer reports) and how many next
6⍝│ letters it predicts right, both as the trainer's Rust model gives
7⍝│ them. The batch run for a GPU (docs/gpu-asks.md): a thousand calls
8⍝│ of one forward pass, the token ids all that goes in.
9
10"nn:" u̲se< "NN"
11"nm:" u̲se< "Norm"
12"at:" u̲se< "Attention"
13
14⍝━━ The model, read from data/ (just micro-gpt-data), as micro-gpt.xtl reads it
15chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
16v ← 1 + t̲ally chars
17wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
18wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
19lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt"
20Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"
21Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"
22Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
23Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"
24fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt"
25fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt"
26
27⍝━━ The forward pass, as micro-gpt.xtl has it
28⍝│ RMS norm without a gain (microgpt's).
29ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
30⍝│ x times a matrix kept a row per output.
31ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
32⍝│ Attention head j (of 4): rows 4j-3 .. 4j of the projections.
33ᵘh̲ead ← { j hM →
34 r ← (4 × j − 1) + r̲ange 4
35 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM
36}
37⍝│ Every position's scores over the 27 tokens, for the tokens ids.
38ᵘg̲pt ← { ids →
39 n ← t̲ally ids
40 x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe
41 hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n)
42 heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM
43 x1 ← x0 + heads ᵘl̲inear Wo
44 x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2
45 x2 ᵘl̲inear lm
46}
47
48⍝━━ A name's score
49⍝│ The names as rows of 17 tokens: the start token, the letters, the
50⍝│ end token (numbered from 1), then zeros.
51names ← 1000 17 r̲eshape f̲loor n̲umbers ⎕N̲GET "data/names.txt"
52⍝│ For a name's row r: the scores at every token but the last, the
53⍝│ tokens that follow as the targets; the sum of -log of each target's
54⍝│ probability, the targets predicted right (the likeliest, the first
55⍝│ on a tie) and the tokens scored.
56ᵘs̲core ← { r →
57 ids ← ('+ r̲/ r > 0) t̲ake r
58 S ← ᵘg̲pt -1 d̲rop ids
59 t ← 1 d̲rop ids
60 ('+ r̲/ r̲avel (v ⁿⁿo̲neHot t) × n̲eg l̲og ⁿⁿs̲oftmax S) c̲at (f̲loat '+ r̲/ (ⁿⁿa̲rgmax S) = t) c̲at f̲loat t̲ally t
61}
62⍝ -- end of the core ------------------------------------------
63
64⍝━━ The thousand names: the mean loss per token, the next letters right; the Rust model's numbers
65⍝│ The scores summed over the names, one call each.
66(totals, _) ← 1000 '{ (T, i) → (T + ᵘs̲core i s̲elect names, i + 1) } p̲ower (0.0 0.0 0.0, 1)
67mean ← (1 s̲elect totals) ÷ 3 s̲elect totals
68(f̲loat f̲loor 0.5 + 10000.0 × mean) ÷ 10000.0
69f̲loor 2 3 s̲elect totals
70expected ← n̲umbers ⎕N̲GET "data/names-loss.txt"
711e-9 > a̲bs mean − 1 s̲elect expected
72(f̲loor 2 s̲elect totals) = f̲loor 2 s̲elect expected