sourcelibs/MoE/demos/routing.xtl

1⍝!/usr/bin/env xetal 2⍝│ A mixture-of-experts layer on 32 tokens: 8 experts, each an 8 x 8 3⍝│ projection adapted at rank 2, the top 2 chosen per token. How the 4⍝│ tokens spread over the experts, the sparse layer against the dense 5⍝│ reference, and what running only the chosen experts saves. 6⍝│ Run with "just demo-lib MoE routing". 7 8"moe:" u̲se< "MoE" 9"lo:" u̲se< "LoRA" 10 11ᵘr̲ipple ← { c n → s̲in c × f̲loat (r̲ange n) × r̲ange n } 12R ← 8 8 r̲eshape 0.9 ᵘr̲ipple 64 13ex ← (8 8 8 r̲eshape 0.7 ᵘr̲ipple 512, 8 2 8 r̲eshape 0.4 ᵘr̲ipple 128, 8 8 2 r̲eshape 0.3 ᵘr̲ipple 128, 4.0 ˡᵒs̲caling 2) 14x ← 8 32 r̲eshape 1.3 ᵘr̲ipple 256 15P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores x 16G ← 2 ᵐᵒᵉt̲opk P 17⍝│ Each expert's share of the tokens (they sum to 2, the top 2), the 18⍝│ mean entropy of the routing (log 8 = 2.08 would be no preference), 19⍝│ and the load-balancing loss (1 when even). 20ᵐᵒᵉf̲requency G 21ᵐᵒᵉe̲ntropy P 22(P, G) ᵐᵒᵉb̲alance 2 23⍝│ The sparse layer equals the dense one that runs every expert (1). 240.000000000001 > 'm̲ax r̲/ r̲avel a̲bs ((ex, G) ᵐᵒᵉc̲ombine x) − (ex, G) ᵐᵒᵉd̲ense x 25⍝│ Expert passes: the sparse layer runs 2 of 8 per token. 26(f̲loat '+ r̲/ r̲avel G > 0.0) c̲at f̲loat 8 × 32