programdemos/micro-gpt/batch.xtl

The tiny GPT over a batch of names: the first 1,000 names of its training file (makemore's names.txt) scored by the forward pass of micro-gpt.xtl, a name per call under its causal mask; the mean loss per token (the cross-entropy the trainer reports) and how many next letters it predicts right, both as the trainer's Rust model gives them. The batch run for a GPU (docs/gpu-asks.md): a thousand calls of one forward pass, the token ids all that goes in.

source · imports nn: libs/NN/src/NN.xtl; nm: libs/Norm/src/Norm.xtl; at: libs/Attention/src/Attention.xtl

The model, read from data/ (just micro-gpt-data), as micro-gpt.xtl reads it

chars : Char

value · line 15
chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
Used in: v

v : Int

value · line 16
v ← 1 + t̲ally chars
Used in: wte, lm, ᵘs̲core

wte : Float

value · line 17
wte ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
Used in: ᵘg̲pt

wpe : Float

value · line 18
wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
Used in: ᵘg̲pt

lm : Float

value · line 19
lm ← (v c̲at 16) r̲eshape n̲umbers ⎕N̲GET "data/lm_head.txt"
Used in: ᵘg̲pt

Wq : Float

value · line 20
Wq ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"
Used in: ᵘh̲ead

Wk : Float

value · line 21
Wk ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"
Used in: ᵘh̲ead

Wv : Float

value · line 22
Wv ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
Used in: ᵘh̲ead

Wo : Float

value · line 23
Wo ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"
Used in: ᵘg̲pt

fc1 : Float

value · line 24
fc1 ← 64 16 r̲eshape n̲umbers ⎕N̲GET "data/fc1.txt"
Used in: ᵘg̲pt

fc2 : Float

value · line 25
fc2 ← 16 64 r̲eshape n̲umbers ⎕N̲GET "data/fc2.txt"
Used in: ᵘg̲pt

The forward pass, as micro-gpt.xtl has it

ᵘr̲ms : Float -> Float

function · line 29

RMS norm without a gain (microgpt's).

ᵘr̲ms ← { x → (16 r̲eshape 1.0) ⁿᵐr̲ms x }
Used in: ᵘg̲pt

ᵘl̲inear : Num a => a -> a -> a

function · line 31

x times a matrix kept a row per output.

ᵘl̲inear ← { x W → x '+ '× i̲nner o̲\ W }
Used in: ᵘg̲pt

ᵘh̲ead : Int -> (Float, Float) -> Float

function · line 33

Attention head j (of 4): rows 4j-3 .. 4j of the projections.

ᵘh̲ead ← { j hM →
  r ← (4 × j − 1) + r̲ange 4
  (o̲\ r s̲elect Wq, o̲\ r s̲elect Wk, o̲\ r s̲elect Wv) ᵃᵗh̲ead hM
}
Used in: ᵘg̲pt

ᵘg̲pt : Int -> Float

function · line 38

Every position's scores over the 27 tokens, for the tokens ids.

ᵘg̲pt ← { ids →
  n ← t̲ally ids
  x0 ← ᵘr̲ms (ids s̲elect wte) + (r̲ange n) s̲elect wpe
  hM ← (ᵘr̲ms x0, ᵃᵗc̲ausal n)
  heads ← (1 ᵘh̲ead hM) c̲at₂ (2 ᵘh̲ead hM) c̲at₂ (3 ᵘh̲ead hM) c̲at₂ 4 ᵘh̲ead hM
  x1 ← x0 + heads ᵘl̲inear Wo
  x2 ← x1 + (0.0 m̲ax (ᵘr̲ms x1) ᵘl̲inear fc1) ᵘl̲inear fc2
  x2 ᵘl̲inear lm
}
Used in: ᵘs̲core

A name's score

names : Int

value · line 51

The names as rows of 17 tokens: the start token, the letters, the end token (numbered from 1), then zeros.

names ← 1000 17 r̲eshape f̲loor n̲umbers ⎕N̲GET "data/names.txt"
Used in: totals

ᵘs̲core : Int -> Float

function · line 56

For a name's row r: the scores at every token but the last, the tokens that follow as the targets; the sum of -log of each target's probability, the targets predicted right (the likeliest, the first on a tie) and the tokens scored.

ᵘs̲core ← { r →
  ids ← ('+ r̲/ r > 0) t̲ake r
  S ← ᵘg̲pt -1 d̲rop ids
  t ← 1 d̲rop ids
  ('+ r̲/ r̲avel (v ⁿⁿo̲neHot t) × n̲eg l̲og ⁿⁿs̲oftmax S) c̲at (f̲loat '+ r̲/ (ⁿⁿa̲rgmax S) = t) c̲at f̲loat t̲ally t
}
Used in: totals

The thousand names: the mean loss per token, the next letters right; the Rust model's numbers

totals : Float

value · line 66

The scores summed over the names, one call each.

(totals, _) ← 1000 '{ (T, i) → (T + ᵘs̲core i s̲elect names, i + 1) } p̲ower (0.0 0.0 0.0, 1)

mean : Float

value · line 67
mean ← (1 s̲elect totals) ÷ 3 s̲elect totals

expected : Float

value · line 70
expected ← n̲umbers ⎕N̲GET "data/names-loss.txt"