sourcelibs/Attention/src/Attention.xtl

1⍝│ Attention: scaled dot-product scores, masked softmax weights, causal masks, a head 2⍝│ 3⍝│ Import it with an alias of your choice: "at:" u_se< "Attention". 4⍝│ Put libs/Attention/src on XETAL_PATH ("just path"); the reference is libs/Attention/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ Tokens X are a matrix, one token per row. A head's projections are 8⍝│ matrices with a row per feature of X: Wq and Wk give queries and keys 9⍝│ of d numbers each, Wv the values. A mask is n x n: 1 where a token 10⍝│ may look at another, 0 where not. Attention is three array 11⍝│ expressions: S = Q K^T / sqrt d, A = softmax of each row of S, 12⍝│ Y = A V. 13 14"nn:" u̲se< "NN" 15 16⍝━━ Scores and weights 17 18⍝│ (Wq, Wk) s̲cores X: every token's query against every token's key, 19⍝│ over the square root of their length: n x n. 20⍝│ >> "at:" u_se< "Attention" 21⍝│ >> X := 2 2 r_eshape 1.0 0.0 0.0 1.0 22⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) at:s_cores X 23⍝│ 1.414213562373095 0.0 24⍝│ 0.0 1.414213562373095 25ˡs̲cores ← { (Wq, Wk) X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ (f̲loat 2 s̲elect s̲hape Wq) ^ 0.5 } 26 27⍝│ M w̲eights S: each row of the scores S into weights summing to 1 28⍝│ (softmax), where the mask M allows; a masked place gets 0. 29⍝│ >> "at:" u_se< "Attention" 30⍝│ >> (at:c_ausal 2) at:w_eights 2 2 r_eshape 0.0 5.0 1.0 1.0 31⍝│ 1.0 0.0 32⍝│ 0.5 0.5 33ˡw̲eights ← { M S → ⁿⁿs̲oftmax S − 1e9 × 1.0 − M } 34 35⍝━━ Masks 36 37⍝│ c̲ausal n: each token sees itself and the tokens before it. 38⍝│ >> "at:" u_se< "Attention" 39⍝│ >> at:c_ausal 3 40⍝│ 1.0 0.0 0.0 41⍝│ 1.0 1.0 0.0 42⍝│ 1.0 1.0 1.0 43ˡc̲ausal ← { n → f̲loat (r̲ange n) '≥ t̲able r̲ange n } 44 45⍝│ a̲ll n: every token sees every token. 46ˡa̲ll ← { n → (n c̲at n) r̲eshape 1.0 } 47 48⍝━━ A head 49 50⍝│ (Wq, Wk, Wv) h̲ead (X, M): one head of attention: the weights of 51⍝│ the scores under the mask, times the values X Wv: n x the values' 52⍝│ width. 53ˡh̲ead ← { (Wq, Wk, Wv) (X, M) → (M ˡw̲eights (Wq, Wk) ˡs̲cores X) '+ '× i̲nner X '+ '× i̲nner Wv }