programlibs/MoE/demos/routing.xtl

A mixture-of-experts layer on 32 tokens: 8 experts, each an 8 x 8 projection adapted at rank 2, the top 2 chosen per token. How the tokens spread over the experts, the sparse layer against the dense reference, and what running only the chosen experts saves. Run with "just demo-lib MoE routing".

source · imports moe: libs/MoE/src/MoE.xtl; lo: libs/LoRA/src/LoRA.xtl libs/LoRA/src/LoRA.xtlm

ᵘr̲ipple : Float -> Int -> Float

function · line 11
ᵘr̲ipple ← { c n → s̲in c × f̲loat (r̲ange n) × r̲ange n }
Used in: R, ex, x

R : Float

value · line 12
R ← 8 8 r̲eshape 0.9 ᵘr̲ipple 64
Used in: P

ex : (Float, Float, Float, Float)

value · line 13
ex ← (8 8 8 r̲eshape 0.7 ᵘr̲ipple 512, 8 2 8 r̲eshape 0.4 ᵘr̲ipple 128, 8 8 2 r̲eshape 0.3 ᵘr̲ipple 128, 4.0 ˡᵒs̲caling 2)

x : Float

value · line 14
x ← 8 32 r̲eshape 1.3 ᵘr̲ipple 256

P : Float

value · line 15
P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores x

G : Float

value · line 16
G ← 2 ᵐᵒᵉt̲opk P