librarylibs/Attention/src/Attention.xtl

Attention: scaled dot-product scores, masked softmax weights, causal masks, a head

Import it with an alias of your choice: "at:" u_se< "Attention". Put libs/Attention/src on XETAL_PATH ("just path"); the reference is libs/Attention/docs. Names with l: are exported; h: names are private to this file.

Tokens X are a matrix, one token per row. A head's projections are matrices with a row per feature of X: Wq and Wk give queries and keys of d numbers each, Wv the values. A mask is n x n: 1 where a token may look at another, 0 where not. Attention is three array expressions: S = Q K^T / sqrt d, A = softmax of each row of S, Y = A V.

source · imports nn: libs/NN/src/NN.xtl

Scores and weights

ˡs̲cores : (Float, Float) -> Float -> Float

function · line 25

(Wq, Wk) s̲cores X: every token's query against every token's key, over the square root of their length: n x n.

      "at:" u̲se< "Attention"
      X ← 2 2 r̲eshape 1.0 0.0 0.0 1.0
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᵗs̲cores X
1.414213562373095               0.0
              0.0 1.414213562373095
ˡs̲cores ← { (Wq, Wk) X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ (f̲loat 2 s̲elect s̲hape Wq) ^ 0.5 }

ˡw̲eights : Float -> Float -> Float

function · line 33

M w̲eights S: each row of the scores S into weights summing to 1 (softmax), where the mask M allows; a masked place gets 0.

      "at:" u̲se< "Attention"
      (ᵃᵗc̲ausal 2) ᵃᵗw̲eights 2 2 r̲eshape 0.0 5.0 1.0 1.0
1.0 0.0
0.5 0.5
ˡw̲eights ← { M S → ⁿⁿs̲oftmax S − 1e9 × 1.0 − M }

Masks

ˡc̲ausal : Int -> Float

function · line 43

c̲ausal n: each token sees itself and the tokens before it.

      "at:" u̲se< "Attention"
      ᵃᵗc̲ausal 3
1.0 0.0 0.0
1.0 1.0 0.0
1.0 1.0 1.0
ˡc̲ausal ← { n → f̲loat (r̲ange n) '≥ t̲able r̲ange n }

ˡa̲ll : Int -> Float

function · line 46

a̲ll n: every token sees every token.

ˡa̲ll ← { n → (n c̲at n) r̲eshape 1.0 }

A head

ˡh̲ead : (Float, Float, Float) -> (Float, Float) -> Float

function · line 53

(Wq, Wk, Wv) h̲ead (X, M): one head of attention: the weights of the scores under the mask, times the values X Wv: n x the values' width.

ˡh̲ead ← { (Wq, Wk, Wv) (X, M) → (M ˡw̲eights (Wq, Wk) ˡs̲cores X) '+ '× i̲nner X '+ '× i̲nner Wv }