sourcedemos/micro-bert/micro-bert.xtl

1⍝!/usr/bin/env xetal 2⍝│ A tiny BERT in X_eTaL: the encoder half of a transformer (Devlin et 3⍝│ al. 2019), 2 layers, 16 numbers wide, 2 attention heads, 7,594 4⍝│ weights, trained offline to fill a blank letter in baby names. Every 5⍝│ letter sees every other, before it and after it, which a GPT's causal 6⍝│ mask forbids. One encoder layer is a pipeline (Flow), applied L times 7⍝│ by p_ower; the forward pass gives the same scores as the Rust model it 8⍝│ was trained in, and the same as the layer written out, and the 9⍝│ attention as Equations writes it. 10 11"nn:" u̲se< "NN" 12"act:" u̲se< "Activations" 13"nm:" u̲se< "Norm" 14"at:" u̲se< "Attention" 15"flow:" u̲se< "Flow" 16"e:" u̲se< "Equations" 17"k:" u̲se< "Check" 18"t:" u̲se< "Strings" 19 20⍝━━ The model, read from data/ (just micro-bert-data) 21⍝│ The 26 letters; the token 27 is the mask, the blank to fill. 22chars ← -1 d̲rop ⎕N̲GET "data/chars.txt" 23L ← 2 24⍝│ Each matrix a row per input, as x times it is written; a dense layer 25⍝│ with its bias as one more row (NN's d_ense); a layer norm as two rows, 26⍝│ its gain and its bias; a layer's weights stacked, layer 1 first. 27wte ← 27 16 r̲eshape n̲umbers ⎕N̲GET "data/wte.txt" 28wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt" 29ln0 ← 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln0.txt" 30Wq ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt" 31Wk ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt" 32Wv ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt" 33Wo ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt" 34ln1 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln1.txt" 35W1 ← 2 17 64 r̲eshape n̲umbers ⎕N̲GET "data/w1.txt" 36W2 ← 2 65 16 r̲eshape n̲umbers ⎕N̲GET "data/w2.txt" 37ln2 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln2.txt" 38lm ← 17 26 r̲eshape n̲umbers ⎕N̲GET "data/lm.txt" 39 40⍝━━ The forward pass 41⍝│ The columns r of a matrix. 42ᵘc̲ols ← { r W → o̲\ r s̲elect o̲\ W } 43⍝│ Head j of layer l: its eight columns of the layer's projections; every 44⍝│ letter sees every letter (at:a_ll, not a causal mask). 45ᵘh̲ead ← { (l, j) H → 46 r ← (8 × j − 1) + r̲ange 8 47 (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk, r ᵘc̲ols l s̲elect Wv) ᵃᵗh̲ead (H, ᵃᵗa̲ll t̲ally H) 48} 49⍝│ Where head j of layer l looks: its attention weights, n x n. 50ᵘl̲ooks ← { (l, j) H → 51 r ← (8 × j − 1) + r̲ange 8 52 (ᵃᵗa̲ll t̲ally H) ᵃᵗw̲eights (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk) ᵃᵗs̲cores H 53} 54⍝│ Layer l's attention: its two heads side by side, times Wo. 55ᵘa̲ttention ← { l H → (((l, 1) ᵘh̲ead H) c̲at₂ (l, 2) ᵘh̲ead H) '+ '× i̲nner l s̲elect Wo } 56⍝│ Layer norm by its two rows. 57ᵘn̲orm ← { G X → (1 s̲elect G, 2 s̲elect G) ⁿᵐl̲ayer X } 58⍝│ Encoder layer l as a pipeline: attention, added back ($h) and normed, 59⍝│ the GELU feed-forward layer, added back ($a) and normed; the shapes 60⍝│ checked. It takes and gives the layer number too, for p_ower. 61ᵘl̲ayer ← { (H, l) → 62 n ← t̲ally H 63 y ← @ ᶠˡᵒʷp̲ipe< c"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  h ← H
  g1:stage ← { @ →
    it ← h
    l ᵘa̲ttention it
  } @
  a ← { @ →
    it ← g1:stage
    (l s̲elect ln1) ᵘn̲orm it + h
  } @
  _ ← { @ →
    ((s̲hape a) m̲atch r̲avel n c̲at 16) ? @
    "shape" ⎕S̲IGNAL "stage 3 (a): shape " c̲at (f̲ormat s̲hape a) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
  } @
  g2:stage ← { @ →
    it ← a
    ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
  } @
  g3:stage ← { @ →
    it ← g2:stage
    it ⁿⁿd̲ense l s̲elect W2
  } @
  out ← { @ →
    it ← g3:stage
    (l s̲elect ln2) ᵘn̲orm it + a
  } @
  _ ← { @ →
    ((s̲hape out) m̲atch r̲avel n c̲at 16) ? @
    "shape" ⎕S̲IGNAL "stage 6 (out): shape " c̲at (f̲ormat s̲hape out) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
  } @
  out
} @)
64 H ▹h 65 ▷ l ᵘa̲ttention it 66 ▷ (l s̲elect ln1) ᵘn̲orm it + h ▹a [n 16] 67 ▷ ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1 68 ▷ it ⁿⁿd̲ense l s̲elect W2 69 ▷ (l s̲elect ln2) ᵘn̲orm it + a ▹out [n 16]" 70 (y, l + 1) 71} 72⍝│ The letters' embeddings and their places', normed. 73ᵘe̲mbed ← { ids → ln0 ᵘn̲orm (ids s̲elect wte) + (r̲ange t̲ally ids) s̲elect wpe } 74⍝│ The scores over the 26 letters at every place of the tokens ids: L layers, then the dense layer. 75ᵘb̲ert ← { ids → 76 (hL, _) ← L 'ᵘl̲ayer p̲ower (ᵘe̲mbed ids, 1) 77 hL ⁿⁿd̲ense lm 78} 79⍝│ A name with a blank, "ma_y", as tokens: the blank is the mask (27, 80⍝│ past the letters). 81ᵘt̲okens ← { s → chars i̲ndexOf s } 82⍝│ The probability of each letter for the blank. 83ᵘf̲ill ← { s → ⁿⁿs̲oftmax (f̲irst w̲here 27 = ᵘt̲okens s) s̲elect ᵘb̲ert ᵘt̲okens s } 84⍝ -- end of the core ------------------------------------------ 85 86⍝━━ Four ways to write it 87⍝│ 1. As a paper prints it, one line: L layers, each attention then the 88⍝│ feed-forward layer, each added back (I +) and layer-normed, on the 89⍝│ letters' and places' embeddings. Equations has no composition or 90⍝│ power of functions, so this line is written as LaTeX by hand; the 91⍝│ pieces below are read by Equations. 92r"\mathrm{BERT}(x) = \big(\mathrm{LN} \circ (I + \mathrm{FFN}) \circ \mathrm{LN} \circ (I + \mathrm{MHA})\big)^{L} \, \mathrm{LN}(E_{tok}[x] + E_{pos})" 93 94⍝│ 2. The attention of head 1 of layer 1 as Equations reads it, for the 95⍝│ blank of "em_a": the scores, the softmax written out (exp over the 96⍝│ row's sum: not shifted by the row's largest, fine at these sizes), 97⍝│ the values weighted. GELU, the layer and its L repeats stay outside 98⍝│ the strings (asks E6, E3, E2 in docs/xetal-libraries-asks.md). 99ids ← ᵘt̲okens "em_a" 100n ← t̲ally ids 101X ← ᵘe̲mbed ids 102c ← r̲ange 8 103Q ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wq 104K ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wk 105V ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wv 106S ← ᵉe̲q< S_{i,j} = \frac{\sum_{k=0}^{7} Q_{i,k} \, K_{j,k}}{\sqrt{8.0}}
ᵉe̲q< expands to
(({ @ →
  g6:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g5:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g4:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g7:grid ← n c̲at n
  g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
  g7:grid r̲eshape ((({ @ →
  g9:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g10:g̲rid ← { k → g9:grid r̲eshape (o̲ffsets k s̲elect g9:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g9:grid) r̲eshape 0 }
  '+ r̲/₃ g9:grid r̲eshape ((
    ((g4:r̲ank (Q, "Q", 2, "Q[i,k]")) g6:a̲t ((
      ((Q, "Q", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape Q) × 
      ((Q, "Q", 1, "i") g5:c̲heck (0 + g10:g̲rid 1))))) × 
    ((g4:r̲ank (K, "K", 2, "K[j,k]")) g6:a̲t ((
      ((K, "K", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape K) × 
      ((K, "K", 1, "j") g5:c̲heck (0 + g10:g̲rid 2)))))))
} @) ÷ (8.0 ^ 0.5)))
} @))
107A ← ᵉe̲q< A_{i,j} = \frac{\exp(S_{i,j})}{\sum_{m=0}^{n - 1} \exp(S_{i,m})}
ᵉe̲q< expands to
(({ @ →
  g13:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g12:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g11:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g14:grid ← n c̲at n
  g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
  g14:grid r̲eshape (((e̲xp 
    ((g11:r̲ank (S, "S", 2, "S[i,j]")) g13:a̲t ((
      ((S, "S", 2, "j") g12:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape S) × 
      ((S, "S", 1, "i") g12:c̲heck (0 + g15:g̲rid 1)))))) ÷ ({ @ →
  g16:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
  g17:g̲rid ← { k → g16:grid r̲eshape (o̲ffsets k s̲elect g16:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g16:grid) r̲eshape 0 }
  '+ r̲/₃ g16:grid r̲eshape ((e̲xp 
    ((g11:r̲ank (S, "S", 2, "S[i,m]")) g13:a̲t ((
      ((S, "S", 2, "m") g12:c̲heck (0 + g17:g̲rid 3)) + (2 s̲elect s̲hape S) × 
      ((S, "S", 1, "i") g12:c̲heck (0 + g17:g̲rid 1)))))))
} @)))
} @))
108Y ← ᵉe̲q< Y_{i,d} = \sum_{j=0}^{n - 1} A_{i,j} \, V_{j,d}
ᵉe̲q< expands to
(({ @ →
  g20:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g19:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g18:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g21:grid ← n c̲at 8
  g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
  g21:grid r̲eshape (({ @ →
  g23:grid ← n c̲at 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
  g24:g̲rid ← { k → g23:grid r̲eshape (o̲ffsets k s̲elect g23:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g23:grid) r̲eshape 0 }
  '+ r̲/₃ g23:grid r̲eshape ((
    ((g18:r̲ank (A, "A", 2, "A[i,j]")) g20:a̲t ((
      ((A, "A", 2, "j") g19:c̲heck (0 + g24:g̲rid 3)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "i") g19:c̲heck (0 + g24:g̲rid 1))))) × 
    ((g18:r̲ank (V, "V", 2, "V[j,d]")) g20:a̲t ((
      ((V, "V", 2, "d") g19:c̲heck (0 + g24:g̲rid 2)) + (2 s̲elect s̲hape V) × 
      ((V, "V", 1, "j") g19:c̲heck (0 + g24:g̲rid 3)))))))
} @))
} @))
109"S[i,j] | n n" ᵉt̲ex< "(sum k:=0..7 | Q[i,k] * K[j,k]) / sqrt(8.0)"
ᵉt̲ex< expands to
"S_{i,j} = \\frac{\\sum_{k=0}^{7} Q_{i,k} \\, K_{j,k}}{\\sqrt{8.0}}"
110"A[i,j] | n n" ᵉt̲ex< "exp(S[i,j]) / (sum m:=0..n-1 | exp(S[i,m]))"
ᵉt̲ex< expands to
"A_{i,j} = \\frac{\\exp(S_{i,j})}{\\sum_{m=0}^{n - 1} \\exp(S_{i,m})}"
111"Y[i,d] | n 8" ᵉt̲ex< "sum j:=0..n-1 | A[i,j] * V[j,d]"
ᵉt̲ex< expands to
"Y_{i,d} = \\sum_{j=0}^{n - 1} A_{i,j} \\, V_{j,d}"
112 113⍝│ 3. As a pipeline: u:l_ayer above, L times by p_ower (u:b_ert). 114 115⍝│ 4. As plain X_eTaL: the same layer written out. 116ᵘp̲lain ← { (H, l) → 117 a ← (l s̲elect ln1) ᵘn̲orm H + l ᵘa̲ttention H 118 ((l s̲elect ln2) ᵘn̲orm a + (ᵃᶜᵗg̲elu a ⁿⁿd̲ense l s̲elect W1) ⁿⁿd̲ense l s̲elect W2, l + 1) 119} 120 121⍝━━ The checks 122⍝│ The Rust model's scores for "em_a" at every place (data/expected.txt) 123⍝│ and its heads' attention, layer by layer (data/expected-looks.txt), 124⍝│ written by the trainer. 125expected ← 4 26 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt" 126looks ← 4 16 r̲eshape n̲umbers ⎕N̲GET "data/expected-looks.txt" 127(h1, _) ← ᵘl̲ayer (X, 1) 128(hP, _) ← L 'ᵘp̲lain p̲ower (X, 1) 129(hF, _) ← L 'ᵘl̲ayer p̲ower (X, 1) 130mine ← r̲avel ((1, 1) ᵘl̲ooks X) c̲at ((1, 2) ᵘl̲ooks X) c̲at ((2, 1) ᵘl̲ooks h1) c̲at (2, 2) ᵘl̲ooks h1 131r1 ← "the attention as Equations reads it is head 1's" ᵏt̲est Y ᵏn̲ear (1, 1) ᵘh̲ead X 132r2 ← "the pipeline, L times, is the layer written out" ᵏt̲est hF ᵏn̲ear hP 133ᵘc̲lose ← { a b → ᵏt̲rue 1e-9 > 'm̲ax r̲/ r̲avel a̲bs a − b } 134r3 ← "the scores are the Rust model's, within 1e-9" ᵏt̲est expected ᵘc̲lose ᵘb̲ert ids 135r4 ← "and so is where each head looks" ᵏt̲est (r̲avel looks) ᵘc̲lose mine 136ᵏr̲eport r1 ᵏa̲nd r2 ᵏa̲nd r3 ᵏa̲nd r4 137 138⍝━━ Blanks filled: the three likeliest letters and their probabilities 139ᵘt̲op ← { s → 140 p ← ᵘf̲ill s 141 o ← 3 t̲ake r̲ev g̲rade p 142 s c̲at " " c̲at " " ᵗj̲oin '{ i → (i s̲elect chars) c̲at " " c̲at f̲ormat (f̲loat f̲loor 0.5 + 1000.0 × i s̲elect p) ÷ 1000.0 } m̲ap o 143} 144ᵘt̲op "em_a" 145ᵘt̲op "ma_y" 146ᵘt̲op "_lice" 147ᵘt̲op "sophi_" 148ᵘt̲op "jo_n" 149ᵘt̲op "_"