sourcedemos/micro-gpt/micro-gpt.xtl
1⍝!/usr/bin/env xetal
2⍝│ A tiny GPT in X_eTaL: Karpathy's microgpt (1 layer, 16 numbers wide,
3⍝│ 4 attention heads, 4,192 weights), trained offline on 32,033 baby
4⍝│ names, inventing new ones a letter at a time. The forward pass reads
5⍝│ a whole name at once under a causal mask, on the Norm, Attention and
6⍝│ Sample libraries; it gives the same scores as the Rust model it was
7⍝│ trained in.
8
9"nn:" u̲se< "NN"
10"nm:" u̲se< "Norm"
11"at:" u̲se< "Attention"
12"sm:" u̲se< "Sample"
13"t:" u̲se< "Strings"
14
15⍝━━ The model, read from data/ (just micro-gpt-data)
16⍝│ The letters, then the token 27 that starts and ends a name.
17chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
18v ← 1 + t̲ally chars
19⍝│ Each matrix a row per output, as the model keeps them.
20wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
21wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
22lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt"
23Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"
24Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"
25Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
26Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"
27fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt"
28fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt"
29
30⍝━━ The forward pass: scores for what comes after each token
31⍝│ RMS norm without a gain (microgpt's).
32ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
33⍝│ x times a matrix kept a row per output.
34ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
35⍝│ Attention head j (of 4): rows 4j-3 .. 4j of the projections.
36ᵘh̲ead ← { j hM →
37 r ← (4 × j − 1) + r̲ange 4
38 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM
39}
40⍝│ Every position's scores over the 27 tokens, for the tokens ids.
41ᵘg̲pt ← { ids →
42 n ← t̲ally ids
43 x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe
44 hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n)
45 heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM
46 x1 ← x0 + heads ᵘl̲inear Wo
47 x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2
48 x2 ᵘl̲inear lm
49}
50
51⍝━━ Generating: a letter at a time
52⍝│ From the tokens s (the start token, then any letters given), up to 16
53⍝│ tokens in all, each drawn at temperature t by the uniform numbers u
54⍝│ (one per letter); a name ends at token 27.
55ᵘn̲ame ← { (t, s) u →
56 (ids, _) ← (16 − t̲ally s) '{ (ids, done) →
57 done ? (ids, done)
58 p ← t ˢᵐp̲robs (t̲ally ids) s̲elect ᵘg̲pt ids
59 tok ← f̲irst (1 r̲eshape (t̲ally ids) s̲elect u) ˢᵐd̲raw (1 c̲at t̲ally p) r̲eshape p
60 (ids c̲at tok, tok = v)
61 } p̲ower (s, 0 = 1)
62 ((ids < v) r̲eplicate ids) s̲elect chars
63}
64⍝ -- end of the core ------------------------------------------
65
66⍝━━ The check
67⍝│ The scores for the start token then "emma", position by position,
68⍝│ equal the Rust model's (data/expected.txt, written by the trainer).
69expected ← 5 27 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"
70e ← chars i̲ndexOf "emma"
711e-9 > 'm̲ax r̲/ r̲avel a̲bs expected − ᵘg̲pt v c̲at e
72
73⍝━━ Twenty names at temperature 0.5 (microgpt's), the uniform numbers from r_oll! (seeded)
74U ← (f̲loat r̲oll! 20 16 r̲eshape 100000) ÷ 100000.0
75" " ᵗj̲oin '{ i → (0.5, 1 r̲eshape v) ᵘn̲ame i s̲elect U } m̲ap r̲ange 20
76⍝│ Five more that start with "ma".
77" " ᵗj̲oin '{ i → (0.5, v c̲at chars i̲ndexOf "ma") ᵘn̲ame i s̲elect U } m̲ap 1 + r̲ange 5