programdemos/micro-bert/micro-bert.xtl
A tiny BERT in X_eTaL: the encoder half of a transformer (Devlin et al. 2019), 2 layers, 16 numbers wide, 2 attention heads, 7,594 weights, trained offline to fill a blank letter in baby names. Every letter sees every other, before it and after it, which a GPT's causal mask forbids. One encoder layer is a pipeline (Flow), applied L times by p_ower; the forward pass gives the same scores as the Rust model it was trained in, and the same as the layer written out, and the attention as Equations writes it.
The model, read from data/ (just micro-bert-data)
chars : Char
The 26 letters; the token 27 is the mask, the blank to fill.
chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
wte : Float
Each matrix a row per input, as x times it is written; a dense layer with its bias as one more row (NN's d_ense); a layer norm as two rows, its gain and its bias; a layer's weights stacked, layer 1 first.
wte ← 27 16 r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
The forward pass
ᵘh̲ead : (Int, Int) -> Float -> Float
Head j of layer l: its eight columns of the layer's projections; every letter sees every letter (at:a_ll, not a causal mask).
ᵘh̲ead ← { (l, j) H → r ← (8 × j − 1) + r̲ange 8 (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk, r ᵘc̲ols l s̲elect Wv) ᵃᵗh̲ead (H, ᵃᵗa̲ll t̲ally H) }
ᵘl̲ooks : (Int, Int) -> Float -> Float
Where head j of layer l looks: its attention weights, n x n.
ᵘl̲ooks ← { (l, j) H → r ← (8 × j − 1) + r̲ange 8 (ᵃᵗa̲ll t̲ally H) ᵃᵗw̲eights (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk) ᵃᵗs̲cores H }
ᵘa̲ttention : Int -> Float -> Float
Layer l's attention: its two heads side by side, times Wo.
ᵘa̲ttention ← { l H → (((l, 1) ᵘh̲ead H) c̲at₂ (l, 2) ᵘh̲ead H) '+ '× i̲nner l s̲elect Wo }
ᵘn̲orm : Float -> Float -> Float
Layer norm by its two rows.
ᵘn̲orm ← { G X → (1 s̲elect G, 2 s̲elect G) ⁿᵐl̲ayer X }
ᵘl̲ayer : (Float, Int) -> (Float, Int)
Encoder layer l as a pipeline: attention, added back ($h) and normed, the GELU feed-forward layer, added back ($a) and normed; the shapes checked. It takes and gives the layer number too, for p_ower.
ᵘl̲ayer ← { (H, l) → n ← t̲ally H y ← @ ᶠˡᵒʷp̲ipe< c" H ▹h ▷ l ᵘa̲ttention it ▷ (l s̲elect ln1) ᵘn̲orm it + h ▹a [n 16] ▷ ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1 ▷ it ⁿⁿd̲ense l s̲elect W2 ▷ (l s̲elect ln2) ᵘn̲orm it + a ▹out [n 16]" (y, l + 1) }
ᶠˡᵒʷp̲ipe< expands to
({ @ →
h ← H
g1:stage ← { @ →
it ← h
l ᵘa̲ttention it
} @
a ← { @ →
it ← g1:stage
(l s̲elect ln1) ᵘn̲orm it + h
} @
_ ← { @ →
((s̲hape a) m̲atch r̲avel n c̲at 16) ? @
"shape" ⎕S̲IGNAL "stage 3 (a): shape " c̲at (f̲ormat s̲hape a) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
} @
g2:stage ← { @ →
it ← a
ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
} @
g3:stage ← { @ →
it ← g2:stage
it ⁿⁿd̲ense l s̲elect W2
} @
out ← { @ →
it ← g3:stage
(l s̲elect ln2) ᵘn̲orm it + a
} @
_ ← { @ →
((s̲hape out) m̲atch r̲avel n c̲at 16) ? @
"shape" ⎕S̲IGNAL "stage 6 (out): shape " c̲at (f̲ormat s̲hape out) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
} @
out
} @)ᵘe̲mbed : Int -> Float
The letters' embeddings and their places', normed.
ᵘe̲mbed ← { ids → ln0 ᵘn̲orm (ids s̲elect wte) + (r̲ange t̲ally ids) s̲elect wpe }
ᵘb̲ert : Int -> Float
The scores over the 26 letters at every place of the tokens ids: L layers, then the dense layer.
ᵘb̲ert ← { ids → (hL, _) ← L 'ᵘl̲ayer p̲ower (ᵘe̲mbed ids, 1) hL ⁿⁿd̲ense lm }
ᵘt̲okens : Char -> Int
A name with a blank, "ma_y", as tokens: the blank is the mask (27, past the letters).
ᵘt̲okens ← { s → chars i̲ndexOf s }
ᵘf̲ill : Char -> Float
The probability of each letter for the blank.
ᵘf̲ill ← { s → ⁿⁿs̲oftmax (f̲irst w̲here 27 = ᵘt̲okens s) s̲elect ᵘb̲ert ᵘt̲okens s }
Four ways to write it
ids : Int
2. The attention of head 1 of layer 1 as Equations reads it, for the blank of "em_a": the scores, the softmax written out (exp over the row's sum: not shifted by the row's largest, fine at these sizes), the values weighted. GELU, the layer and its L repeats stay outside the strings (asks E6, E3, E2 in docs/xetal-libraries-asks.md).
ids ← ᵘt̲okens "em_a"
S : Float
S ← ᵉe̲q< S_{i,j} = \frac{\sum_{k=0}^{7} Q_{i,k} \, K_{j,k}}{\sqrt{8.0}}
ᵉe̲q< expands to
(({ @ →
g6:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g5:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g4:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g7:grid ← n c̲at n
g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
g7:grid r̲eshape ((({ @ →
g9:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g10:g̲rid ← { k → g9:grid r̲eshape (o̲ffsets k s̲elect g9:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g9:grid) r̲eshape 0 }
'+ r̲/₃ g9:grid r̲eshape ((
((g4:r̲ank (Q, "Q", 2, "Q[i,k]")) g6:a̲t ((
((Q, "Q", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape Q) ×
((Q, "Q", 1, "i") g5:c̲heck (0 + g10:g̲rid 1))))) ×
((g4:r̲ank (K, "K", 2, "K[j,k]")) g6:a̲t ((
((K, "K", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape K) ×
((K, "K", 1, "j") g5:c̲heck (0 + g10:g̲rid 2)))))))
} @) ÷ (8.0 ^ 0.5)))
} @))A : Float
A ← ᵉe̲q< A_{i,j} = \frac{\exp(S_{i,j})}{\sum_{m=0}^{n - 1} \exp(S_{i,m})}
ᵉe̲q< expands to
(({ @ →
g13:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g12:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g11:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g14:grid ← n c̲at n
g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
g14:grid r̲eshape (((e̲xp
((g11:r̲ank (S, "S", 2, "S[i,j]")) g13:a̲t ((
((S, "S", 2, "j") g12:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape S) ×
((S, "S", 1, "i") g12:c̲heck (0 + g15:g̲rid 1)))))) ÷ ({ @ →
g16:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
g17:g̲rid ← { k → g16:grid r̲eshape (o̲ffsets k s̲elect g16:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g16:grid) r̲eshape 0 }
'+ r̲/₃ g16:grid r̲eshape ((e̲xp
((g11:r̲ank (S, "S", 2, "S[i,m]")) g13:a̲t ((
((S, "S", 2, "m") g12:c̲heck (0 + g17:g̲rid 3)) + (2 s̲elect s̲hape S) ×
((S, "S", 1, "i") g12:c̲heck (0 + g17:g̲rid 1)))))))
} @)))
} @))Y : Float
Y ← ᵉe̲q< Y_{i,d} = \sum_{j=0}^{n - 1} A_{i,j} \, V_{j,d}
ᵉe̲q< expands to
(({ @ →
g20:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g19:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g18:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g21:grid ← n c̲at 8
g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
g21:grid r̲eshape (({ @ →
g23:grid ← n c̲at 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
g24:g̲rid ← { k → g23:grid r̲eshape (o̲ffsets k s̲elect g23:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g23:grid) r̲eshape 0 }
'+ r̲/₃ g23:grid r̲eshape ((
((g18:r̲ank (A, "A", 2, "A[i,j]")) g20:a̲t ((
((A, "A", 2, "j") g19:c̲heck (0 + g24:g̲rid 3)) + (2 s̲elect s̲hape A) ×
((A, "A", 1, "i") g19:c̲heck (0 + g24:g̲rid 1))))) ×
((g18:r̲ank (V, "V", 2, "V[j,d]")) g20:a̲t ((
((V, "V", 2, "d") g19:c̲heck (0 + g24:g̲rid 2)) + (2 s̲elect s̲hape V) ×
((V, "V", 1, "j") g19:c̲heck (0 + g24:g̲rid 3)))))))
} @))
} @))ᵘp̲lain : (Float, Int) -> (Float, Int)
4. As plain X_eTaL: the same layer written out.
ᵘp̲lain ← { (H, l) → a ← (l s̲elect ln1) ᵘn̲orm H + l ᵘa̲ttention H ((l s̲elect ln2) ᵘn̲orm a + (ᵃᶜᵗg̲elu a ⁿⁿd̲ense l s̲elect W1) ⁿⁿd̲ense l s̲elect W2, l + 1) }
The checks
expected : Float
The Rust model's scores for "em_a" at every place (data/expected.txt) and its heads' attention, layer by layer (data/expected-looks.txt), written by the trainer.
expected ← 4 26 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"
mine : Float
mine ← r̲avel ((1, 1) ᵘl̲ooks X) c̲at ((1, 2) ᵘl̲ooks X) c̲at ((2, 1) ᵘl̲ooks h1) c̲at (2, 2) ᵘl̲ooks h1
r1 : Char
r1 ← "the attention as Equations reads it is head 1's" ᵏt̲est Y ᵏn̲ear (1, 1) ᵘh̲ead X
r2 : Char
r2 ← "the pipeline, L times, is the layer written out" ᵏt̲est hF ᵏn̲ear hP
r3 : Char
r3 ← "the scores are the Rust model's, within 1e-9" ᵏt̲est expected ᵘc̲lose ᵘb̲ert ids
r4 : Char
r4 ← "and so is where each head looks" ᵏt̲est (r̲avel looks) ᵘc̲lose mine