sourcedemos/moe-finetune/base.xtl

1⍝!/usr/bin/env xetal 2⍝│ The base model every fine-tuning method starts from: a tiny 3⍝│ mixture-of-experts trained in X_eTaL (train/base.xtl, fixed seeds) 4⍝│ on tokens of 8 features in 4 kinds, each token's class (1 to 3) a 5⍝│ rule of its kind's own. A shared layer z = tanh(S x), a router R on 6⍝│ z, 4 experts at top 2, softmax over the classes. Its baseline: the 7⍝│ accuracy on the training and the held-out tokens, kind by kind, and 8⍝│ its routing with the router frozen: which experts each kind's tokens 9⍝│ go to, how often each expert is chosen, the routing's entropy and 10⍝│ balance. A method that changes the experts but not the router keeps 11⍝│ this routing; one that trains the router can move it. 12 13"moe:" u̲se< "MoE" 14 15⍝━━ The checkpoint and the tokens 16 17⍝│ r̲ead file: (x, the classes one-hot, the kinds) of a data file: 18⍝│ the inputs (9 x N, the 9th a constant 1), the kinds, the classes. 19ᵘr̲ead ← { file → 20 v ← n̲umbers ⎕N̲GET file 21 N ← (t̲ally v) d̲iv 11 22 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v) 23} 24((x, Y), kinds) ← ᵘr̲ead "data/train.txt" 25((xt, Yt), kindst) ← ᵘr̲ead "data/test.txt" 26⍝│ The checkpoint: S (8 x 9), R (4 x 8), the experts' Ws (4 x 3 x 8); 27⍝│ the experts are LoRA adapters with no update yet (A, B 0, rank 2). 28w ← n̲umbers ⎕N̲GET "data/base.txt" 29S ← 8 9 r̲eshape 72 t̲ake w 30R ← 4 8 r̲eshape 32 t̲ake 72 d̲rop w 31Ws ← 4 3 8 r̲eshape 104 d̲rop w 32ex ← (Ws, 4 2 8 r̲eshape 0.0, 4 3 2 r̲eshape 0.0, 1.0) 33 34⍝━━ The model 35 36⍝│ r̲oute x: each token's gates over the 4 experts (top 2). 37ᵘr̲oute ← { x → 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner x } 38⍝│ p̲redict x: each token's probabilities over the 3 classes. 39ᵘp̲redict ← { x → ᵐᵒᵉs̲oftmax (ex, ᵘr̲oute x) ᵐᵒᵉc̲ombine t̲anh S '+ '× i̲nner x } 40⍝│ (P, Y) a̲ccuracy kinds: the share of tokens whose likeliest class 41⍝│ is theirs, kind by kind, then over all. 42ᵘa̲ccuracy ← { (P, Y) kinds → 43 right ← '+ r̲/₁ Y × 1 ᵐᵒᵉm̲ask P 44 K ← f̲loat (r̲ange 4) '= t̲able kinds 45 ((K '+ '× i̲nner right) ÷ '+ r̲/₂ K) c̲at ('+ r̲/ right) ÷ f̲loat t̲ally right 46} 47ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 } 48⍝ -- end of the base model ------------------------------ 49 50⍝│ The trainable numbers: S, R and the experts (72 + 32 + 96). 51t̲ally w 52⍝│ The accuracy on the 800 training tokens and the 400 held out: kinds 53⍝│ 1 to 4, then all. 54ᵘr̲ound (ᵘp̲redict x, Y) ᵘa̲ccuracy kinds 55ᵘr̲ound (ᵘp̲redict xt, Yt) ᵘa̲ccuracy kindst 56 57⍝━━ The routing, the router frozen 58 59⍝│ On the held-out tokens: the share of each kind's tokens (a column) 60⍝│ each expert (a row) gets; each column sums to 2, the top 2. 61G ← ᵘr̲oute xt 62Kt ← f̲loat (r̲ange 4) '= t̲able kindst 63ᵘr̲ound ((f̲loat G > 0.0) '+ '× i̲nner o̲\ Kt) ÷ 4 4 r̲eshape '+ r̲/₂ Kt 64⍝│ How often each expert is chosen (summing to 2), the routing's 65⍝│ entropy in nats (log 4 = 1.386 for no preference), and its balance 66⍝│ (1 when even, 4 when one expert takes all). 67ᵘr̲ound ᵐᵒᵉf̲requency G 68P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner xt 69ᵘr̲ound (ᵐᵒᵉe̲ntropy P) c̲at (P, G) ᵐᵒᵉb̲alance 2