programlibs/Attention/demos/previous.xtl

A head that looks one token back: each token's key says where it is, each query asks for the place before its own, so under the causal mask every token takes on its predecessor's value. Run with "just demo-lib Attention previous".

source · imports at: libs/Attention/src/Attention.xtl

n : Int

value · line 10

Five tokens, each with a one-hot place (5 numbers) and a value.

n ← 5

P : Bool

value · line 11
P ← (r̲ange n) '= t̲able r̲ange n
Used in: X

X : Float

value · line 12
X ← f̲loat P

Wq : Float

value · line 15

Queries ask for the place before (the place matrix shifted), keys are the places; both sharpened by 10 so the softmax picks one.

Wq ← 10.0 × f̲loat (r̲ange n) '= t̲able 1 + r̲ange n

Wk : Float

value · line 16
Wk ← 10.0 × X

Wv : Float

value · line 17
Wv ← (n c̲at 1) r̲eshape 10.0 20.0 30.0 40.0 50.0