programdemos/micro-gpt/micro-gpt.xtl
A tiny GPT in X_eTaL: Karpathy's microgpt (1 layer, 16 numbers wide, 4 attention heads, 4,192 weights), trained offline on 32,033 baby names, inventing new ones a letter at a time. The forward pass reads a whole name at once under a causal mask, on the Norm, Attention and Sample libraries; it gives the same scores as the Rust model it was trained in.
The model, read from data/ (just micro-gpt-data)
chars : Char
The letters, then the token 27 that starts and ends a name.
chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
wte : Float
Each matrix a row per output, as the model keeps them.
wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
The forward pass: scores for what comes after each token
ᵘr̲ms : Float -> Float
RMS norm without a gain (microgpt's).
ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
ᵘl̲inear : Num a => a -> a -> a
x times a matrix kept a row per output.
ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
ᵘh̲ead : Int -> (Float, Float) -> Float
Attention head j (of 4): rows 4j-3 .. 4j of the projections.
ᵘh̲ead ← { j hM → r ← (4 × j − 1) + r̲ange 4 (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM }
ᵘg̲pt : Int -> Float
Every position's scores over the 27 tokens, for the tokens ids.
ᵘg̲pt ← { ids → n ← t̲ally ids x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n) heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM x1 ← x0 + heads ᵘl̲inear Wo x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2 x2 ᵘl̲inear lm }
Generating: a letter at a time
ᵘn̲ame : (Float, Int) -> Float -> Char
From the tokens s (the start token, then any letters given), up to 16 tokens in all, each drawn at temperature t by the uniform numbers u (one per letter); a name ends at token 27.
ᵘn̲ame ← { (t, s) u → (ids, _) ← (16 − t̲ally s) '{ (ids, done) → done ? (ids, done) p ← t ˢᵐp̲robs (t̲ally ids) s̲elect ᵘg̲pt ids tok ← f̲irst (1 r̲eshape (t̲ally ids) s̲elect u) ˢᵐd̲raw (1 c̲at t̲ally p) r̲eshape p (ids c̲at tok, tok = v) } p̲ower (s, 0 = 1) ((ids < v) r̲eplicate ids) s̲elect chars }
The check
expected : Float
The scores for the start token then "emma", position by position, equal the Rust model's (data/expected.txt, written by the trainer).
expected ← 5 27 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"