programlibs/MoE/demos/routing.xtl
A mixture-of-experts layer on 32 tokens: 8 experts, each an 8 x 8 projection adapted at rank 2, the top 2 chosen per token. How the tokens spread over the experts, the sparse layer against the dense reference, and what running only the chosen experts saves. Run with "just demo-lib MoE routing".
ex : (Float, Float, Float, Float)
ex ← (8 8 8 r̲eshape 0.7 ᵘr̲ipple 512, 8 2 8 r̲eshape 0.4 ᵘr̲ipple 128, 8 8 2 r̲eshape 0.3 ᵘr̲ipple 128, 4.0 ˡᵒs̲caling 2)
Used in: libs/MoE/demos/routing.xtl:24
P : Float
P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores x