sourcelibs/Attention/src/Attention.xtl
1⍝│ Attention: scaled dot-product scores, masked softmax weights, causal masks, a head
2⍝│
3⍝│ Import it with an alias of your choice: "at:" u_se< "Attention".
4⍝│ Put libs/Attention/src on XETAL_PATH ("just path"); the reference is libs/Attention/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ Tokens X are a matrix, one token per row. A head's projections are
8⍝│ matrices with a row per feature of X: Wq and Wk give queries and keys
9⍝│ of d numbers each, Wv the values. A mask is n x n: 1 where a token
10⍝│ may look at another, 0 where not. Attention is three array
11⍝│ expressions: S = Q K^T / sqrt d, A = softmax of each row of S,
12⍝│ Y = A V.
13
14"nn:" u̲se< "NN"
15
16⍝━━ Scores and weights
17
18⍝│ (Wq, Wk) s̲cores X: every token's query against every token's key,
19⍝│ over the square root of their length: n x n.
20⍝│ >> "at:" u_se< "Attention"
21⍝│ >> X := 2 2 r_eshape 1.0 0.0 0.0 1.0
22⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) at:s_cores X
23⍝│ 1.414213562373095 0.0
24⍝│ 0.0 1.414213562373095
25ˡs̲cores ← { (Wq, Wk) X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ (f̲loat 2 s̲elect s̲hape Wq) ^ 0.5 }
26
27⍝│ M w̲eights S: each row of the scores S into weights summing to 1
28⍝│ (softmax), where the mask M allows; a masked place gets 0.
29⍝│ >> "at:" u_se< "Attention"
30⍝│ >> (at:c_ausal 2) at:w_eights 2 2 r_eshape 0.0 5.0 1.0 1.0
31⍝│ 1.0 0.0
32⍝│ 0.5 0.5
33ˡw̲eights ← { M S → ⁿⁿs̲oftmax S − 1e9 × 1.0 − M }
34
35⍝━━ Masks
36
37⍝│ c̲ausal n: each token sees itself and the tokens before it.
38⍝│ >> "at:" u_se< "Attention"
39⍝│ >> at:c_ausal 3
40⍝│ 1.0 0.0 0.0
41⍝│ 1.0 1.0 0.0
42⍝│ 1.0 1.0 1.0
43ˡc̲ausal ← { n → f̲loat (r̲ange n) '≥ t̲able r̲ange n }
44
45⍝│ a̲ll n: every token sees every token.
46ˡa̲ll ← { n → (n c̲at n) r̲eshape 1.0 }
47
48⍝━━ A head
49
50⍝│ (Wq, Wk, Wv) h̲ead (X, M): one head of attention: the weights of
51⍝│ the scores under the mask, times the values X Wv: n x the values'
52⍝│ width.
53ˡh̲ead ← { (Wq, Wk, Wv) (X, M) → (M ˡw̲eights (Wq, Wk) ˡs̲cores X) '+ '× i̲nner X '+ '× i̲nner Wv }