sourcedemos/micro-bert/micro-bert.xtl
1⍝!/usr/bin/env xetal
2⍝│ A tiny BERT in X_eTaL: the encoder half of a transformer (Devlin et
3⍝│ al. 2019), 2 layers, 16 numbers wide, 2 attention heads, 7,594
4⍝│ weights, trained offline to fill a blank letter in baby names. Every
5⍝│ letter sees every other, before it and after it, which a GPT's causal
6⍝│ mask forbids. One encoder layer is a pipeline (Flow), applied L times
7⍝│ by p_ower; the forward pass gives the same scores as the Rust model it
8⍝│ was trained in, and the same as the layer written out, and the
9⍝│ attention as Equations writes it.
10
11"nn:" u̲se< "NN"
12"act:" u̲se< "Activations"
13"nm:" u̲se< "Norm"
14"at:" u̲se< "Attention"
15"flow:" u̲se< "Flow"
16"e:" u̲se< "Equations"
17"k:" u̲se< "Check"
18"t:" u̲se< "Strings"
19
20⍝━━ The model, read from data/ (just micro-bert-data)
21⍝│ The 26 letters; the token 27 is the mask, the blank to fill.
22chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"
23L ← 2
24⍝│ Each matrix a row per input, as x times it is written; a dense layer
25⍝│ with its bias as one more row (NN's d_ense); a layer norm as two rows,
26⍝│ its gain and its bias; a layer's weights stacked, layer 1 first.
27wte ← 27 16 r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
28wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
29ln0 ← 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln0.txt"
30Wq ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"
31Wk ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"
32Wv ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
33Wo ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"
34ln1 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln1.txt"
35W1 ← 2 17 64 r̲eshape n̲umbers ⎕N̲GET "data/w1.txt"
36W2 ← 2 65 16 r̲eshape n̲umbers ⎕N̲GET "data/w2.txt"
37ln2 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln2.txt"
38lm ← 17 26 r̲eshape n̲umbers ⎕N̲GET "data/lm.txt"
39
40⍝━━ The forward pass
41⍝│ The columns r of a matrix.
42ᵘc̲ols ← { r W → o̲\ r s̲elect o̲\ W }
43⍝│ Head j of layer l: its eight columns of the layer's projections; every
44⍝│ letter sees every letter (at:a_ll, not a causal mask).
45ᵘh̲ead ← { (l, j) H →
46 r ← (8 × j − 1) + r̲ange 8
47 (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk, r ᵘc̲ols l s̲elect Wv) ᵃᵗh̲ead (H, ᵃᵗa̲ll t̲ally H)
48}
49⍝│ Where head j of layer l looks: its attention weights, n x n.
50ᵘl̲ooks ← { (l, j) H →
51 r ← (8 × j − 1) + r̲ange 8
52 (ᵃᵗa̲ll t̲ally H) ᵃᵗw̲eights (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk) ᵃᵗs̲cores H
53}
54⍝│ Layer l's attention: its two heads side by side, times Wo.
55ᵘa̲ttention ← { l H → (((l, 1) ᵘh̲ead H) c̲at₂ (l, 2) ᵘh̲ead H) '+ '× i̲nner l s̲elect Wo }
56⍝│ Layer norm by its two rows.
57ᵘn̲orm ← { G X → (1 s̲elect G, 2 s̲elect G) ⁿᵐl̲ayer X }
58⍝│ Encoder layer l as a pipeline: attention, added back ($h) and normed,
59⍝│ the GELU feed-forward layer, added back ($a) and normed; the shapes
60⍝│ checked. It takes and gives the layer number too, for p_ower.
61ᵘl̲ayer ← { (H, l) →
62 n ← t̲ally H
63 y ← @ ᶠˡᵒʷp̲ipe< c"
64 H ▹h
65 ▷ l ᵘa̲ttention it
66 ▷ (l s̲elect ln1) ᵘn̲orm it + h ▹a [n 16]
67 ▷ ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
68 ▷ it ⁿⁿd̲ense l s̲elect W2
69 ▷ (l s̲elect ln2) ᵘn̲orm it + a ▹out [n 16]"
70 (y, l + 1)
71}
72⍝│ The letters' embeddings and their places', normed.
73ᵘe̲mbed ← { ids → ln0 ᵘn̲orm (ids s̲elect wte) + (r̲ange t̲ally ids) s̲elect wpe }
74⍝│ The scores over the 26 letters at every place of the tokens ids: L layers, then the dense layer.
75ᵘb̲ert ← { ids →
76 (hL, _) ← L 'ᵘl̲ayer p̲ower (ᵘe̲mbed ids, 1)
77 hL ⁿⁿd̲ense lm
78}
79⍝│ A name with a blank, "ma_y", as tokens: the blank is the mask (27,
80⍝│ past the letters).
81ᵘt̲okens ← { s → chars i̲ndexOf s }
82⍝│ The probability of each letter for the blank.
83ᵘf̲ill ← { s → ⁿⁿs̲oftmax (f̲irst w̲here 27 = ᵘt̲okens s) s̲elect ᵘb̲ert ᵘt̲okens s }
84⍝ -- end of the core ------------------------------------------
85
86⍝━━ Four ways to write it
87⍝│ 1. As a paper prints it, one line: L layers, each attention then the
88⍝│ feed-forward layer, each added back (I +) and layer-normed, on the
89⍝│ letters' and places' embeddings. Equations has no composition or
90⍝│ power of functions, so this line is written as LaTeX by hand; the
91⍝│ pieces below are read by Equations.
92r"\mathrm{BERT}(x) = \big(\mathrm{LN} \circ (I + \mathrm{FFN}) \circ \mathrm{LN} \circ (I + \mathrm{MHA})\big)^{L} \, \mathrm{LN}(E_{tok}[x] + E_{pos})"
93
94⍝│ 2. The attention of head 1 of layer 1 as Equations reads it, for the
95⍝│ blank of "em_a": the scores, the softmax written out (exp over the
96⍝│ row's sum: not shifted by the row's largest, fine at these sizes),
97⍝│ the values weighted. GELU, the layer and its L repeats stay outside
98⍝│ the strings (asks E6, E3, E2 in docs/xetal-libraries-asks.md).
99ids ← ᵘt̲okens "em_a"
100n ← t̲ally ids
101X ← ᵘe̲mbed ids
102c ← r̲ange 8
103Q ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wq
104K ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wk
105V ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wv
106S ← ᵉe̲q< S_{i,j} = \frac{\sum_{k=0}^{7} Q_{i,k} \, K_{j,k}}{\sqrt{8.0}}
107A ← ᵉe̲q< A_{i,j} = \frac{\exp(S_{i,j})}{\sum_{m=0}^{n - 1} \exp(S_{i,m})}
108Y ← ᵉe̲q< Y_{i,d} = \sum_{j=0}^{n - 1} A_{i,j} \, V_{j,d}
109"S[i,j] | n n" ᵉt̲ex< "(sum k:=0..7 | Q[i,k] * K[j,k]) / sqrt(8.0)"
110"A[i,j] | n n" ᵉt̲ex< "exp(S[i,j]) / (sum m:=0..n-1 | exp(S[i,m]))"
111"Y[i,d] | n 8" ᵉt̲ex< "sum j:=0..n-1 | A[i,j] * V[j,d]"
112
113⍝│ 3. As a pipeline: u:l_ayer above, L times by p_ower (u:b_ert).
114
115⍝│ 4. As plain X_eTaL: the same layer written out.
116ᵘp̲lain ← { (H, l) →
117 a ← (l s̲elect ln1) ᵘn̲orm H + l ᵘa̲ttention H
118 ((l s̲elect ln2) ᵘn̲orm a + (ᵃᶜᵗg̲elu a ⁿⁿd̲ense l s̲elect W1) ⁿⁿd̲ense l s̲elect W2, l + 1)
119}
120
121⍝━━ The checks
122⍝│ The Rust model's scores for "em_a" at every place (data/expected.txt)
123⍝│ and its heads' attention, layer by layer (data/expected-looks.txt),
124⍝│ written by the trainer.
125expected ← 4 26 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"
126looks ← 4 16 r̲eshape n̲umbers ⎕N̲GET "data/expected-looks.txt"
127(h1, _) ← ᵘl̲ayer (X, 1)
128(hP, _) ← L 'ᵘp̲lain p̲ower (X, 1)
129(hF, _) ← L 'ᵘl̲ayer p̲ower (X, 1)
130mine ← r̲avel ((1, 1) ᵘl̲ooks X) c̲at ((1, 2) ᵘl̲ooks X) c̲at ((2, 1) ᵘl̲ooks h1) c̲at (2, 2) ᵘl̲ooks h1
131r1 ← "the attention as Equations reads it is head 1's" ᵏt̲est Y ᵏn̲ear (1, 1) ᵘh̲ead X
132r2 ← "the pipeline, L times, is the layer written out" ᵏt̲est hF ᵏn̲ear hP
133ᵘc̲lose ← { a b → ᵏt̲rue 1e-9 > 'm̲ax r̲/ r̲avel a̲bs a − b }
134r3 ← "the scores are the Rust model's, within 1e-9" ᵏt̲est expected ᵘc̲lose ᵘb̲ert ids
135r4 ← "and so is where each head looks" ᵏt̲est (r̲avel looks) ᵘc̲lose mine
136ᵏr̲eport r1 ᵏa̲nd r2 ᵏa̲nd r3 ᵏa̲nd r4
137
138⍝━━ Blanks filled: the three likeliest letters and their probabilities
139ᵘt̲op ← { s →
140 p ← ᵘf̲ill s
141 o ← 3 t̲ake r̲ev g̲rade p
142 s c̲at " " c̲at " " ᵗj̲oin '{ i → (i s̲elect chars) c̲at " " c̲at f̲ormat (f̲loat f̲loor 0.5 + 1000.0 × i s̲elect p) ÷ 1000.0 } m̲ap o
143}
144ᵘt̲op "em_a"
145ᵘt̲op "ma_y"
146ᵘt̲op "_lice"
147ᵘt̲op "sophi_"
148ᵘt̲op "jo_n"
149ᵘt̲op "_"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
h ← H
g1:stage ← { @ →
it ← h
l ᵘa̲ttention it
} @
a ← { @ →
it ← g1:stage
(l s̲elect ln1) ᵘn̲orm it + h
} @
_ ← { @ →
((s̲hape a) m̲atch r̲avel n c̲at 16) ? @
"shape" ⎕S̲IGNAL "stage 3 (a): shape " c̲at (f̲ormat s̲hape a) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
} @
g2:stage ← { @ →
it ← a
ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
} @
g3:stage ← { @ →
it ← g2:stage
it ⁿⁿd̲ense l s̲elect W2
} @
out ← { @ →
it ← g3:stage
(l s̲elect ln2) ᵘn̲orm it + a
} @
_ ← { @ →
((s̲hape out) m̲atch r̲avel n c̲at 16) ? @
"shape" ⎕S̲IGNAL "stage 6 (out): shape " c̲at (f̲ormat s̲hape out) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
} @
out
} @)ᵉe̲q< expands to
(({ @ →
g6:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g5:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g4:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g7:grid ← n c̲at n
g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
g7:grid r̲eshape ((({ @ →
g9:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g10:g̲rid ← { k → g9:grid r̲eshape (o̲ffsets k s̲elect g9:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g9:grid) r̲eshape 0 }
'+ r̲/₃ g9:grid r̲eshape ((
((g4:r̲ank (Q, "Q", 2, "Q[i,k]")) g6:a̲t ((
((Q, "Q", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape Q) ×
((Q, "Q", 1, "i") g5:c̲heck (0 + g10:g̲rid 1))))) ×
((g4:r̲ank (K, "K", 2, "K[j,k]")) g6:a̲t ((
((K, "K", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape K) ×
((K, "K", 1, "j") g5:c̲heck (0 + g10:g̲rid 2)))))))
} @) ÷ (8.0 ^ 0.5)))
} @))ᵉe̲q< expands to
(({ @ →
g13:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g12:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g11:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g14:grid ← n c̲at n
g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
g14:grid r̲eshape (((e̲xp
((g11:r̲ank (S, "S", 2, "S[i,j]")) g13:a̲t ((
((S, "S", 2, "j") g12:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape S) ×
((S, "S", 1, "i") g12:c̲heck (0 + g15:g̲rid 1)))))) ÷ ({ @ →
g16:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
g17:g̲rid ← { k → g16:grid r̲eshape (o̲ffsets k s̲elect g16:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g16:grid) r̲eshape 0 }
'+ r̲/₃ g16:grid r̲eshape ((e̲xp
((g11:r̲ank (S, "S", 2, "S[i,m]")) g13:a̲t ((
((S, "S", 2, "m") g12:c̲heck (0 + g17:g̲rid 3)) + (2 s̲elect s̲hape S) ×
((S, "S", 1, "i") g12:c̲heck (0 + g17:g̲rid 1)))))))
} @)))
} @))ᵉe̲q< expands to
(({ @ →
g20:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g19:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g18:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g21:grid ← n c̲at 8
g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
g21:grid r̲eshape (({ @ →
g23:grid ← n c̲at 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
g24:g̲rid ← { k → g23:grid r̲eshape (o̲ffsets k s̲elect g23:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g23:grid) r̲eshape 0 }
'+ r̲/₃ g23:grid r̲eshape ((
((g18:r̲ank (A, "A", 2, "A[i,j]")) g20:a̲t ((
((A, "A", 2, "j") g19:c̲heck (0 + g24:g̲rid 3)) + (2 s̲elect s̲hape A) ×
((A, "A", 1, "i") g19:c̲heck (0 + g24:g̲rid 1))))) ×
((g18:r̲ank (V, "V", 2, "V[j,d]")) g20:a̲t ((
((V, "V", 2, "d") g19:c̲heck (0 + g24:g̲rid 2)) + (2 s̲elect s̲hape V) ×
((V, "V", 1, "j") g19:c̲heck (0 + g24:g̲rid 3)))))))
} @))
} @))ᵉt̲ex< expands to
"S_{i,j} = \\frac{\\sum_{k=0}^{7} Q_{i,k} \\, K_{j,k}}{\\sqrt{8.0}}"ᵉt̲ex< expands to
"A_{i,j} = \\frac{\\exp(S_{i,j})}{\\sum_{m=0}^{n - 1} \\exp(S_{i,m})}"ᵉt̲ex< expands to
"Y_{i,d} = \\sum_{j=0}^{n - 1} A_{i,j} \\, V_{j,d}"