sourcelibs/MoE/demos/routing.xtl
1⍝!/usr/bin/env xetal
2⍝│ A mixture-of-experts layer on 32 tokens: 8 experts, each an 8 x 8
3⍝│ projection adapted at rank 2, the top 2 chosen per token. How the
4⍝│ tokens spread over the experts, the sparse layer against the dense
5⍝│ reference, and what running only the chosen experts saves.
6⍝│ Run with "just demo-lib MoE routing".
7
8"moe:" u̲se< "MoE"
9"lo:" u̲se< "LoRA"
10
11ᵘr̲ipple ← { c n → s̲in c × f̲loat (r̲ange n) × r̲ange n }
12R ← 8 8 r̲eshape 0.9 ᵘr̲ipple 64
13ex ← (8 8 8 r̲eshape 0.7 ᵘr̲ipple 512, 8 2 8 r̲eshape 0.4 ᵘr̲ipple 128, 8 8 2 r̲eshape 0.3 ᵘr̲ipple 128, 4.0 ˡᵒs̲caling 2)
14x ← 8 32 r̲eshape 1.3 ᵘr̲ipple 256
15P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores x
16G ← 2 ᵐᵒᵉt̲opk P
17⍝│ Each expert's share of the tokens (they sum to 2, the top 2), the
18⍝│ mean entropy of the routing (log 8 = 2.08 would be no preference),
19⍝│ and the load-balancing loss (1 when even).
20ᵐᵒᵉf̲requency G
21ᵐᵒᵉe̲ntropy P
22(P, G) ᵐᵒᵉb̲alance 2
23⍝│ The sparse layer equals the dense one that runs every expert (1).
240.000000000001 > 'm̲ax r̲/ r̲avel a̲bs ((ex, G) ᵐᵒᵉc̲ombine x) − (ex, G) ᵐᵒᵉd̲ense x
25⍝│ Expert passes: the sparse layer runs 2 of 8 per token.
26(f̲loat '+ r̲/ r̲avel G > 0.0) c̲at f̲loat 8 × 32