programdemos/attention/attention.xtl
Attention as array algebra: every token's query meets every token's key in one matrix product, S = Q K^T / sqrt d; softmax turns each row into weights; the output is the weighted sum of the values, Y = A V. One head, hand-set so that a word looks for what it can describe.
The vocabulary and the head, read from data/
The core: attention for tokens X, where mask M allows
ᵘs̲cores : Float -> Float
Every query against every key: Attention's s_cores, (X Wq) times (X Wk) transposed, over the square root of 2 (the keys' length).
ᵘs̲cores ← { X → (Wq, Wk) ᵃᵗs̲cores X }
ᵘa̲ttend : Float -> Float -> Float
Each row into weights (softmax), where the mask allows.
ᵘa̲ttend ← { X M → M ᵃᵗw̲eights ᵘs̲cores X }
Drawing: rows labeled with their words
ᵘr̲ows : Int -> Char -> Char
ᵘr̲ows ← { ids C → (ids s̲elect names) c̲at₂ (((t̲ally ids) c̲at 2) r̲eshape " ") c̲at₂ C }
ᵘw̲ide : a -> a
ᵘw̲ide ← { C → ((-1 t̲ake s̲hape C) r̲eshape 3) r̲eplicate₂ C }
ᵘh̲eader : Int -> Char
ᵘh̲eader ← { ids → " " c̲at r̲avel (1 t̲ake₂ ids s̲elect names) c̲at₂ ((t̲ally ids) c̲at 2) r̲eshape " " }
ᵘt̲op : Int -> Float -> Char
The word each row looks at most, or - when no word gets half its weight.
ᵘt̲op ← { ids A → sure ← 0.5 ≤ 'm̲ax r̲/₂ A ids ᵘr̲ows ((sure × (ⁿⁿa̲rgmax A) s̲elect ids) + 22 × n̲ot sure) s̲elect names c̲at 1 8 r̲eshape "- " }
ids : Int
"the animal did not cross the street because it was tired"
ids ← 1 3 12 15 13 1 7 20 11 14 16
causal : Float
A causal mask (a word sees only itself and the words before it, as when generating text): "it" still finds the animal; "the" sees itself.
causal ← ᵃᵗc̲ausal n
w : Int
"... because it was wide": wide finds the street. "it" still finds the animal: one head reads each word alone, so "it" cannot know yet which word comes after it; resolving it takes another layer.
w ← 1 3 12 15 13 1 7 20 11 14 18