programdemos/micro-bert/micro-bert.xtl

A tiny BERT in X_eTaL: the encoder half of a transformer (Devlin et al. 2019), 2 layers, 16 numbers wide, 2 attention heads, 7,594 weights, trained offline to fill a blank letter in baby names. Every letter sees every other, before it and after it, which a GPT's causal mask forbids. One encoder layer is a pipeline (Flow), applied L times by p_ower; the forward pass gives the same scores as the Rust model it was trained in, and the same as the layer written out, and the attention as Equations writes it.

source · imports nn: libs/NN/src/NN.xtl; act: libs/Activations/src/Activations.xtl; nm: libs/Norm/src/Norm.xtl; at: libs/Attention/src/Attention.xtl; flow: libs/Flow/src/Flow.xtlm; e: work/libs/Equations/src/Equations.xtl work/libs/Equations/src/Equations.xtlm; k: work/libs/Check/src/Check.xtl work/libs/Check/src/Check.xtlm; t: work/libs/Strings/src/Strings.xtl

The model, read from data/ (just micro-bert-data)

chars : Char

value · line 22

The 26 letters; the token 27 is the mask, the blank to fill.

chars ← -1 d̲rop ⎕N̲GET "data/chars.txt"

L : Int

value · line 23
L ← 2
Used in: ᵘb̲ert, hP, hF

wte : Float

value · line 27

Each matrix a row per input, as x times it is written; a dense layer with its bias as one more row (NN's d_ense); a layer norm as two rows, its gain and its bias; a layer's weights stacked, layer 1 first.

wte ← 27 16 r̲eshape n̲umbers ⎕N̲GET "data/wte.txt"
Used in: ᵘe̲mbed

wpe : Float

value · line 28
wpe ← 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wpe.txt"
Used in: ᵘe̲mbed

ln0 : Float

value · line 29
ln0 ← 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln0.txt"
Used in: ᵘe̲mbed

Wq : Float

value · line 30
Wq ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wq.txt"

Wk : Float

value · line 31
Wk ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wk.txt"

Wv : Float

value · line 32
Wv ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wv.txt"
Used in: ᵘh̲ead, V

Wo : Float

value · line 33
Wo ← 2 16 16 r̲eshape n̲umbers ⎕N̲GET "data/wo.txt"

ln1 : Float

value · line 34
ln1 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln1.txt"

W1 : Float

value · line 35
W1 ← 2 17 64 r̲eshape n̲umbers ⎕N̲GET "data/w1.txt"

W2 : Float

value · line 36
W2 ← 2 65 16 r̲eshape n̲umbers ⎕N̲GET "data/w2.txt"

ln2 : Float

value · line 37
ln2 ← 2 2 16 r̲eshape n̲umbers ⎕N̲GET "data/ln2.txt"

lm : Float

value · line 38
lm ← 17 26 r̲eshape n̲umbers ⎕N̲GET "data/lm.txt"
Used in: ᵘb̲ert

The forward pass

ᵘc̲ols : Int -> a -> a

function · line 42

The columns r of a matrix.

ᵘc̲ols ← { r W → o̲\ r s̲elect o̲\ W }
Used in: ᵘh̲ead, ᵘl̲ooks, Q, K, V

ᵘh̲ead : (Int, Int) -> Float -> Float

function · line 45

Head j of layer l: its eight columns of the layer's projections; every letter sees every letter (at:a_ll, not a causal mask).

ᵘh̲ead ← { (l, j) H →
  r ← (8 × j − 1) + r̲ange 8
  (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk, r ᵘc̲ols l s̲elect Wv) ᵃᵗh̲ead (H, ᵃᵗa̲ll t̲ally H)
}
Used in: ᵘa̲ttention, r1

ᵘl̲ooks : (Int, Int) -> Float -> Float

function · line 50

Where head j of layer l looks: its attention weights, n x n.

ᵘl̲ooks ← { (l, j) H →
  r ← (8 × j − 1) + r̲ange 8
  (ᵃᵗa̲ll t̲ally H) ᵃᵗw̲eights (r ᵘc̲ols l s̲elect Wq, r ᵘc̲ols l s̲elect Wk) ᵃᵗs̲cores H
}
Used in: mine

ᵘa̲ttention : Int -> Float -> Float

function · line 55

Layer l's attention: its two heads side by side, times Wo.

ᵘa̲ttention ← { l H → (((l, 1) ᵘh̲ead H) c̲at₂ (l, 2) ᵘh̲ead H) '+ '× i̲nner l s̲elect Wo }

ᵘn̲orm : Float -> Float -> Float

function · line 57

Layer norm by its two rows.

ᵘn̲orm ← { G X → (1 s̲elect G, 2 s̲elect G) ⁿᵐl̲ayer X }

ᵘl̲ayer : (Float, Int) -> (Float, Int)

function · line 61

Encoder layer l as a pipeline: attention, added back ($h) and normed, the GELU feed-forward layer, added back ($a) and normed; the shapes checked. It takes and gives the layer number too, for p_ower.

ᵘl̲ayer ← { (H, l) →
  n ← t̲ally H
  y ← @ ᶠˡᵒʷp̲ipe< c"
     H ▹h
  ▷ l ᵘa̲ttention it
  ▷ (l s̲elect ln1) ᵘn̲orm it + h ▹a [n 16]
  ▷ ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
  ▷ it ⁿⁿd̲ense l s̲elect W2
  ▷ (l s̲elect ln2) ᵘn̲orm it + a ▹out [n 16]"
  (y, l + 1)
}
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  h ← H
  g1:stage ← { @ →
    it ← h
    l ᵘa̲ttention it
  } @
  a ← { @ →
    it ← g1:stage
    (l s̲elect ln1) ᵘn̲orm it + h
  } @
  _ ← { @ →
    ((s̲hape a) m̲atch r̲avel n c̲at 16) ? @
    "shape" ⎕S̲IGNAL "stage 3 (a): shape " c̲at (f̲ormat s̲hape a) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
  } @
  g2:stage ← { @ →
    it ← a
    ᵃᶜᵗg̲elu it ⁿⁿd̲ense l s̲elect W1
  } @
  g3:stage ← { @ →
    it ← g2:stage
    it ⁿⁿd̲ense l s̲elect W2
  } @
  out ← { @ →
    it ← g3:stage
    (l s̲elect ln2) ᵘn̲orm it + a
  } @
  _ ← { @ →
    ((s̲hape out) m̲atch r̲avel n c̲at 16) ? @
    "shape" ⎕S̲IGNAL "stage 6 (out): shape " c̲at (f̲ormat s̲hape out) c̲at ", expected " c̲at f̲ormat r̲avel n c̲at 16
  } @
  out
} @)
Used in: ᵘb̲ert, h1, hF

ᵘe̲mbed : Int -> Float

function · line 73

The letters' embeddings and their places', normed.

ᵘe̲mbed ← { ids → ln0 ᵘn̲orm (ids s̲elect wte) + (r̲ange t̲ally ids) s̲elect wpe }
Used in: ᵘb̲ert, X

ᵘb̲ert : Int -> Float

function · line 75

The scores over the 26 letters at every place of the tokens ids: L layers, then the dense layer.

ᵘb̲ert ← { ids →
  (hL, _) ← L 'ᵘl̲ayer p̲ower (ᵘe̲mbed ids, 1)
  hL ⁿⁿd̲ense lm
}
Used in: ᵘf̲ill, r3

ᵘt̲okens : Char -> Int

function · line 81

A name with a blank, "ma_y", as tokens: the blank is the mask (27, past the letters).

ᵘt̲okens ← { s → chars i̲ndexOf s }
Used in: ᵘf̲ill, ids

ᵘf̲ill : Char -> Float

function · line 83

The probability of each letter for the blank.

ᵘf̲ill ← { s → ⁿⁿs̲oftmax (f̲irst w̲here 27 = ᵘt̲okens s) s̲elect ᵘb̲ert ᵘt̲okens s }
Used in: ᵘt̲op

Four ways to write it

ids : Int

value · line 99

2. The attention of head 1 of layer 1 as Equations reads it, for the blank of "em_a": the scores, the softmax written out (exp over the row's sum: not shifted by the row's largest, fine at these sizes), the values weighted. GELU, the layer and its L repeats stay outside the strings (asks E6, E3, E2 in docs/xetal-libraries-asks.md).

ids ← ᵘt̲okens "em_a"
Used in: n, X, r3

n : Int

value · line 100
n ← t̲ally ids
Used in: ᵘl̲ayer, S, A, Y

X : Float

value · line 101
X ← ᵘe̲mbed ids
Used in: Q, K, V, h1, hP, hF, mine, r1

c : Int

value · line 102
c ← r̲ange 8
Used in: Q, K, V

Q : Float

value · line 103
Q ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wq
Used in: S

K : Float

value · line 104
K ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wk
Used in: S

V : Float

value · line 105
V ← X '+ '× i̲nner c ᵘc̲ols 1 s̲elect Wv
Used in: Y

S : Float

value · line 106
S ←  ᵉe̲q< S_{i,j} = \frac{\sum_{k=0}^{7} Q_{i,k} \, K_{j,k}}{\sqrt{8.0}}
ᵉe̲q< expands to
(({ @ →
  g6:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g5:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g4:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g7:grid ← n c̲at n
  g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
  g7:grid r̲eshape ((({ @ →
  g9:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g10:g̲rid ← { k → g9:grid r̲eshape (o̲ffsets k s̲elect g9:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g9:grid) r̲eshape 0 }
  '+ r̲/₃ g9:grid r̲eshape ((
    ((g4:r̲ank (Q, "Q", 2, "Q[i,k]")) g6:a̲t ((
      ((Q, "Q", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape Q) × 
      ((Q, "Q", 1, "i") g5:c̲heck (0 + g10:g̲rid 1))))) × 
    ((g4:r̲ank (K, "K", 2, "K[j,k]")) g6:a̲t ((
      ((K, "K", 2, "k") g5:c̲heck (0 + g10:g̲rid 3)) + (2 s̲elect s̲hape K) × 
      ((K, "K", 1, "j") g5:c̲heck (0 + g10:g̲rid 2)))))))
} @) ÷ (8.0 ^ 0.5)))
} @))
Used in: A

A : Float

value · line 107
A ←  ᵉe̲q< A_{i,j} = \frac{\exp(S_{i,j})}{\sum_{m=0}^{n - 1} \exp(S_{i,m})}
ᵉe̲q< expands to
(({ @ →
  g13:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g12:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g11:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g14:grid ← n c̲at n
  g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
  g14:grid r̲eshape (((e̲xp 
    ((g11:r̲ank (S, "S", 2, "S[i,j]")) g13:a̲t ((
      ((S, "S", 2, "j") g12:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape S) × 
      ((S, "S", 1, "i") g12:c̲heck (0 + g15:g̲rid 1)))))) ÷ ({ @ →
  g16:grid ← n c̲at n c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
  g17:g̲rid ← { k → g16:grid r̲eshape (o̲ffsets k s̲elect g16:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g16:grid) r̲eshape 0 }
  '+ r̲/₃ g16:grid r̲eshape ((e̲xp 
    ((g11:r̲ank (S, "S", 2, "S[i,m]")) g13:a̲t ((
      ((S, "S", 2, "m") g12:c̲heck (0 + g17:g̲rid 3)) + (2 s̲elect s̲hape S) × 
      ((S, "S", 1, "i") g12:c̲heck (0 + g17:g̲rid 1)))))))
} @)))
} @))
Used in: S, A, Y

Y : Float

value · line 108
Y ←  ᵉe̲q< Y_{i,d} = \sum_{j=0}^{n - 1} A_{i,j} \, V_{j,d}
ᵉe̲q< expands to
(({ @ →
  g20:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g19:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g18:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g21:grid ← n c̲at 8
  g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
  g21:grid r̲eshape (({ @ →
  g23:grid ← n c̲at 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..n-1" } (1 + (n − 1) − 0))
  g24:g̲rid ← { k → g23:grid r̲eshape (o̲ffsets k s̲elect g23:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g23:grid) r̲eshape 0 }
  '+ r̲/₃ g23:grid r̲eshape ((
    ((g18:r̲ank (A, "A", 2, "A[i,j]")) g20:a̲t ((
      ((A, "A", 2, "j") g19:c̲heck (0 + g24:g̲rid 3)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "i") g19:c̲heck (0 + g24:g̲rid 1))))) × 
    ((g18:r̲ank (V, "V", 2, "V[j,d]")) g20:a̲t ((
      ((V, "V", 2, "d") g19:c̲heck (0 + g24:g̲rid 2)) + (2 s̲elect s̲hape V) × 
      ((V, "V", 1, "j") g19:c̲heck (0 + g24:g̲rid 3)))))))
} @))
} @))
Used in: r1

ᵘp̲lain : (Float, Int) -> (Float, Int)

function · line 116

4. As plain X_eTaL: the same layer written out.

ᵘp̲lain ← { (H, l) →
  a ← (l s̲elect ln1) ᵘn̲orm H + l ᵘa̲ttention H
  ((l s̲elect ln2) ᵘn̲orm a + (ᵃᶜᵗg̲elu a ⁿⁿd̲ense l s̲elect W1) ⁿⁿd̲ense l s̲elect W2, l + 1)
}
Used in: hP

The checks

expected : Float

value · line 125

The Rust model's scores for "em_a" at every place (data/expected.txt) and its heads' attention, layer by layer (data/expected-looks.txt), written by the trainer.

expected ← 4 26 r̲eshape n̲umbers ⎕N̲GET "data/expected.txt"
Used in: r3

looks : Float

value · line 126
looks ← 4 16 r̲eshape n̲umbers ⎕N̲GET "data/expected-looks.txt"
Used in: r4

h1 : Float

value · line 127
(h1, _) ← ᵘl̲ayer (X, 1)
Used in: h1, mine

hP : Float

value · line 128
(hP, _) ← L 'ᵘp̲lain p̲ower (X, 1)
Used in: hP, r2

hF : Float

value · line 129
(hF, _) ← L 'ᵘl̲ayer p̲ower (X, 1)
Used in: hF, r2

mine : Float

value · line 130
mine ← r̲avel ((1, 1) ᵘl̲ooks X) c̲at ((1, 2) ᵘl̲ooks X) c̲at ((2, 1) ᵘl̲ooks h1) c̲at (2, 2) ᵘl̲ooks h1
Used in: r4

r1 : Char

value · line 131
r1 ← "the attention as Equations reads it is head 1's" ᵏt̲est Y ᵏn̲ear (1, 1) ᵘh̲ead X

r2 : Char

value · line 132
r2 ← "the pipeline, L times, is the layer written out" ᵏt̲est hF ᵏn̲ear hP

ᵘc̲lose : Float -> Float -> Char

function · line 133
ᵘc̲lose ← { a b → ᵏt̲rue 1e-9 > 'm̲ax r̲/ r̲avel a̲bs a − b }
Used in: r3, r4

r3 : Char

value · line 134
r3 ← "the scores are the Rust model's, within 1e-9" ᵏt̲est expected ᵘc̲lose ᵘb̲ert ids

r4 : Char

value · line 135
r4 ← "and so is where each head looks" ᵏt̲est (r̲avel looks) ᵘc̲lose mine

Blanks filled: the three likeliest letters and their probabilities

ᵘt̲op : Char -> Char

function · line 139
ᵘt̲op ← { s →
  p ← ᵘf̲ill s
  o ← 3 t̲ake r̲ev g̲rade p
  s c̲at "  " c̲at " " ᵗj̲oin '{ i → (i s̲elect chars) c̲at " " c̲at f̲ormat (f̲loat f̲loor 0.5 + 1000.0 × i s̲elect p) ÷ 1000.0 } m̲ap o
}