programdemos/attention/attention.xtl
Attention as array algebra: every token's query meets every token's key in one matrix product, S = Q K^T / sqrt d; softmax turns each row into weights; the output is the weighted sum of the values, Y = A V. One head, hand-set so that a word looks for what it can describe.
The vocabulary and the head, read from data/
The core: attention for tokens X, where mask M allows
ᵘs̲cores : Float -> Float
ᵘs̲cores ← { X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ 2.0 ^ 0.5 }
Used in: ᵘa̲ttend
Drawing: rows labeled with their words
ᵘr̲ows : Int -> Char -> Char
ᵘr̲ows ← { ids C → (ids s̲elect names) c̲at₂ (((t̲ally ids) c̲at 2) r̲eshape " ") c̲at₂ C }
ᵘw̲ide : a -> a
ᵘw̲ide ← { C → ((-1 t̲ake s̲hape C) r̲eshape 3) r̲eplicate₂ C }
ᵘh̲eader : Int -> Char
ᵘh̲eader ← { ids → " " c̲at r̲avel (1 t̲ake₂ ids s̲elect names) c̲at₂ ((t̲ally ids) c̲at 2) r̲eshape " " }
ᵘt̲op : Int -> Float -> Char
The word each row looks at most, or - when no word gets half its weight.
ᵘt̲op ← { ids A → sure ← 0.5 ≤ 'm̲ax r̲/₂ A ids ᵘr̲ows ((sure × (ⁿⁿa̲rgmax A) s̲elect ids) + 22 × n̲ot sure) s̲elect names c̲at 1 8 r̲eshape "- " }
ids : Int
"the animal did not cross the street because it was tired"
ids ← 1 3 12 15 13 1 7 20 11 14 16
causal : Float
A causal mask (a word sees only itself and the words before it, as when generating text): "it" still finds the animal; "the" sees itself.
causal ← f̲loat (r̲ange n) '≥ t̲able r̲ange n
Used in: demos/attention/attention.xtl:52
w : Int
"... because it was wide": wide finds the street. "it" still finds the animal: one head reads each word alone, so "it" cannot know yet which word comes after it; resolving it takes another layer.
w ← 1 3 12 15 13 1 7 20 11 14 18
Used in: B, demos/attention/attention.xtl:59