programdemos/micro-gpt/micro-gpt.xtl

A tiny GPT in X_eTaL: Karpathy's microgpt (1 layer, 16 numbers wide, 4 attention heads, 4,192 weights), trained offline on 32,033 baby names, inventing new ones a letter at a time. The forward pass reads a whole name at once under a causal mask, on the Norm, Attention and Sample libraries; it gives the same scores as the Rust model it was trained in.

source · imports nn: libs/NN/src/NN.xtl; nm: libs/Norm/src/Norm.xtl; at: libs/Attention/src/Attention.xtl; sm: libs/Sample/src/Sample.xtl; t: work/libs/Strings/src/Strings.xtl

The model, read from data/ (just micro-gpt-data)

chars : Char

value · line 17

The letters, then the token 27 that starts and ends a name.

chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"

v : Int

value · line 18
v ← 1 + t̲ally chars

wte : Float

value · line 20

Each matrix a row per output, as the model keeps them.

wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
Used in: ᵘg̲pt

wpe : Float

value · line 21
wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
Used in: ᵘg̲pt

lm : Float

value · line 22
lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt"
Used in: ᵘg̲pt

Wq : Float

value · line 23
Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"
Used in: ᵘh̲ead

Wk : Float

value · line 24
Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"
Used in: ᵘh̲ead

Wv : Float

value · line 25
Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
Used in: ᵘh̲ead

Wo : Float

value · line 26
Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"
Used in: ᵘg̲pt

fc1 : Float

value · line 27
fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt"
Used in: ᵘg̲pt

fc2 : Float

value · line 28
fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt"
Used in: ᵘg̲pt

The forward pass: scores for what comes after each token

ᵘr̲ms : Float -> Float

function · line 32

RMS norm without a gain (microgpt's).

ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
Used in: ᵘg̲pt

ᵘl̲inear : Num a => a -> a -> a

function · line 34

x times a matrix kept a row per output.

ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
Used in: ᵘg̲pt

ᵘh̲ead : Int -> (Float, Float) -> Float

function · line 36

Attention head j (of 4): rows 4j-3 .. 4j of the projections.

ᵘh̲ead ← { j hM →
  r ← (4 × j − 1) + r̲ange 4
  (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM
}
Used in: ᵘg̲pt

ᵘg̲pt : Int -> Float

function · line 41

Every position's scores over the 27 tokens, for the tokens ids.

ᵘg̲pt ← { ids →
  n ← t̲ally ids
  x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe
  hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n)
  heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM
  x1 ← x0 + heads ᵘl̲inear Wo
  x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2
  x2 ᵘl̲inear lm
}

Generating: a letter at a time

ᵘn̲ame : (Float, Int) -> Float -> Char

function · line 55

From the tokens s (the start token, then any letters given), up to 16 tokens in all, each drawn at temperature t by the uniform numbers u (one per letter); a name ends at token 27.

ᵘn̲ame ← { (t, s) u →
  (ids, _) ← (16 − t̲ally s) '{ (ids, done) →
    done ? (ids, done)
    p ← t ˢᵐp̲robs (t̲ally ids) s̲elect ᵘg̲pt ids
    tok ← f̲irst (1 r̲eshape (t̲ally ids) s̲elect u) ˢᵐd̲raw (1 c̲at t̲ally p) r̲eshape p
    (ids c̲at tok, tok = v)
  } p̲ower (s, 0 = 1)
  ((ids < v) r̲eplicate ids) s̲elect chars
}

The check

expected : Float

value · line 69

The scores for the start token then "emma", position by position, equal the Rust model's (data/expected.txt, written by the trainer).

expected ← 5 27 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"

e : Int

value · line 70
e ← chars i̲ndexOf "emma"

Twenty names at temperature 0.5 (microgpt's), the uniform numbers from r_oll! (seeded)

U : Float

value · line 74
U ← (f̲loat r̲oll! 20 16 r̲eshape 100000) ÷ 100000.0