sourcedemos/micro-gpt/micro-gpt.xtl

1⍝!/usr/bin/env xetal 2⍝│ A tiny GPT in X_eTaL: Karpathy's microgpt (1 layer, 16 numbers wide, 3⍝│ 4 attention heads, 4,192 weights), trained offline on 32,033 baby 4⍝│ names, inventing new ones a letter at a time. The forward pass reads 5⍝│ a whole name at once under a causal mask, on the Norm, Attention and 6⍝│ Sample libraries; it gives the same scores as the Rust model it was 7⍝│ trained in. 8 9"nn:" u̲se< "NN" 10"nm:" u̲se< "Norm" 11"at:" u̲se< "Attention" 12"sm:" u̲se< "Sample" 13"t:" u̲se< "Strings" 14 15⍝━━ The model, read from data/ (just micro-gpt-data) 16⍝│ The letters, then the token 27 that starts and ends a name. 17chars ← -1 d̲rop ⎕N̲GET "data/chars.txt" 18v ← 1 + t̲ally chars 19⍝│ Each matrix a row per output, as the model keeps them. 20wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt" 21wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt" 22lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt" 23Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt" 24Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt" 25Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt" 26Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt" 27fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt" 28fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt" 29 30⍝━━ The forward pass: scores for what comes after each token 31⍝│ RMS norm without a gain (microgpt's). 32ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x } 33⍝│ x times a matrix kept a row per output. 34ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W } 35⍝│ Attention head j (of 4): rows 4j-3 .. 4j of the projections. 36ᵘh̲ead ← { j hM → 37 r ← (4 × j − 1) + r̲ange 4 38 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM 39} 40⍝│ Every position's scores over the 27 tokens, for the tokens ids. 41ᵘg̲pt ← { ids → 42 n ← t̲ally ids 43 x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe 44 hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n) 45 heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM 46 x1 ← x0 + heads ᵘl̲inear Wo 47 x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2 48 x2 ᵘl̲inear lm 49} 50 51⍝━━ Generating: a letter at a time 52⍝│ From the tokens s (the start token, then any letters given), up to 16 53⍝│ tokens in all, each drawn at temperature t by the uniform numbers u 54⍝│ (one per letter); a name ends at token 27. 55ᵘn̲ame ← { (t, s) u → 56 (ids, _) ← (16 − t̲ally s) '{ (ids, done) → 57 done ? (ids, done) 58 p ← t ˢᵐp̲robs (t̲ally ids) s̲elect ᵘg̲pt ids 59 tok ← f̲irst (1 r̲eshape (t̲ally ids) s̲elect u) ˢᵐd̲raw (1 c̲at t̲ally p) r̲eshape p 60 (ids c̲at tok, tok = v) 61 } p̲ower (s, 0 = 1) 62 ((ids < v) r̲eplicate ids) s̲elect chars 63} 64⍝ -- end of the core ------------------------------------------ 65 66⍝━━ The check 67⍝│ The scores for the start token then "emma", position by position, 68⍝│ equal the Rust model's (data/expected.txt, written by the trainer). 69expected ← 5 27 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt" 70e ← chars i̲ndexOf "emma" 711e-9 > 'm̲ax r̲/ r̲avel a̲bs expected − ᵘg̲pt v c̲at e 72 73⍝━━ Twenty names at temperature 0.5 (microgpt's), the uniform numbers from r_oll! (seeded) 74U ← (f̲loat r̲oll! 20 16 r̲eshape 100000) ÷ 100000.0 75" " ᵗj̲oin '{ i → (0.5, 1 r̲eshape v) ᵘn̲ame i s̲elect U } m̲ap r̲ange 20 76⍝│ Five more that start with "ma". 77" " ᵗj̲oin '{ i → (0.5, v c̲at chars i̲ndexOf "ma") ᵘn̲ame i s̲elect U } m̲ap 1 + r̲ange 5