programlibs/Attention/demos/previous.xtl
A head that looks one token back: each token's key says where it is, each query asks for the place before its own, so under the causal mask every token takes on its predecessor's value. Run with "just demo-lib Attention previous".
Wq : Float
Queries ask for the place before (the place matrix shifted), keys are the places; both sharpened by 10 so the softmax picks one.
Wq ← 10.0 × f̲loat (r̲ange n) '= t̲able 1 + r̲ange n
Used in: libs/Attention/demos/previous.xtl:19
Wv : Float
Wv ← (n c̲at 1) r̲eshape 10.0 20.0 30.0 40.0 50.0
Used in: libs/Attention/demos/previous.xtl:19