sourcelibs/Attention/demos/previous.xtl

1⍝!/usr/bin/env xetal 2⍝│ A head that looks one token back: each token's key says where it is, 3⍝│ each query asks for the place before its own, so under the causal 4⍝│ mask every token takes on its predecessor's value. Run with "just 5⍝│ demo-lib Attention previous". 6 7"at:" u̲se< "Attention" 8 9⍝│ Five tokens, each with a one-hot place (5 numbers) and a value. 10n ← 5 11P ← (r̲ange n) '= t̲able r̲ange n 12X ← f̲loat P 13⍝│ Queries ask for the place before (the place matrix shifted), keys 14⍝│ are the places; both sharpened by 10 so the softmax picks one. 15Wq ← 10.0 × f̲loat (r̲ange n) '= t̲able 1 + r̲ange n 16Wk ← 10.0 × X 17Wv ← (n c̲at 1) r̲eshape 10.0 20.0 30.0 40.0 50.0 18⍝│ Each token's output: its predecessor's value (the first has none and keeps its own). 19(f̲loat f̲loor 0.5 + 10.0 × r̲avel (Wq, Wk, Wv) ᵃᵗh̲ead (X, ᵃᵗc̲ausal n)) ÷ 10.0