librarylibs/Attention/src/Attention.xtl
Attention: scaled dot-product scores, masked softmax weights, causal masks, a head
Import it with an alias of your choice: "at:" u_se< "Attention". Put libs/Attention/src on XETAL_PATH ("just path"); the reference is libs/Attention/docs. Names with l: are exported; h: names are private to this file.
Tokens X are a matrix, one token per row. A head's projections are matrices with a row per feature of X: Wq and Wk give queries and keys of d numbers each, Wv the values. A mask is n x n: 1 where a token may look at another, 0 where not. Attention is three array expressions: S = Q K^T / sqrt d, A = softmax of each row of S, Y = A V.
Scores and weights
ˡs̲cores : (Float, Float) -> Float -> Float
(Wq, Wk) s̲cores X: every token's query against every token's key,
over the square root of their length: n x n.
"at:" u̲se< "Attention"
X ← 2 2 r̲eshape 1.0 0.0 0.0 1.0
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᵗs̲cores X 1.414213562373095 0.0 0.0 1.414213562373095
ˡs̲cores ← { (Wq, Wk) X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ (f̲loat 2 s̲elect s̲hape Wq) ^ 0.5 }
ˡw̲eights : Float -> Float -> Float
M w̲eights S: each row of the scores S into weights summing to 1
(softmax), where the mask M allows; a masked place gets 0.
"at:" u̲se< "Attention"
(ᵃᵗc̲ausal 2) ᵃᵗw̲eights 2 2 r̲eshape 0.0 5.0 1.0 1.0 1.0 0.0 0.5 0.5
ˡw̲eights ← { M S → ⁿⁿs̲oftmax S − 1e9 × 1.0 − M }
Masks
ˡc̲ausal : Int -> Float
c̲ausal n: each token sees itself and the tokens before it.
"at:" u̲se< "Attention"
ᵃᵗc̲ausal 3 1.0 0.0 0.0 1.0 1.0 0.0 1.0 1.0 1.0
ˡc̲ausal ← { n → f̲loat (r̲ange n) '≥ t̲able r̲ange n }