sourcedemos/moe-finetune/base.xtl
1⍝!/usr/bin/env xetal
2⍝│ The base model every fine-tuning method starts from: a tiny
3⍝│ mixture-of-experts trained in X_eTaL (train/base.xtl, fixed seeds)
4⍝│ on tokens of 8 features in 4 kinds, each token's class (1 to 3) a
5⍝│ rule of its kind's own. A shared layer z = tanh(S x), a router R on
6⍝│ z, 4 experts at top 2, softmax over the classes. Its baseline: the
7⍝│ accuracy on the training and the held-out tokens, kind by kind, and
8⍝│ its routing with the router frozen: which experts each kind's tokens
9⍝│ go to, how often each expert is chosen, the routing's entropy and
10⍝│ balance. A method that changes the experts but not the router keeps
11⍝│ this routing; one that trains the router can move it.
12
13"moe:" u̲se< "MoE"
14
15⍝━━ The checkpoint and the tokens
16
17⍝│ r̲ead file: (x, the classes one-hot, the kinds) of a data file:
18⍝│ the inputs (9 x N, the 9th a constant 1), the kinds, the classes.
19ᵘr̲ead ← { file →
20 v ← n̲umbers ⎕N̲GET file
21 N ← (t̲ally v) d̲iv 11
22 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v)
23}
24((x, Y), kinds) ← ᵘr̲ead "data/train.txt"
25((xt, Yt), kindst) ← ᵘr̲ead "data/test.txt"
26⍝│ The checkpoint: S (8 x 9), R (4 x 8), the experts' Ws (4 x 3 x 8);
27⍝│ the experts are LoRA adapters with no update yet (A, B 0, rank 2).
28w ← n̲umbers ⎕N̲GET "data/base.txt"
29S ← 8 9 r̲eshape 72 t̲ake w
30R ← 4 8 r̲eshape 32 t̲ake 72 d̲rop w
31Ws ← 4 3 8 r̲eshape 104 d̲rop w
32ex ← (Ws, 4 2 8 r̲eshape 0.0, 4 3 2 r̲eshape 0.0, 1.0)
33
34⍝━━ The model
35
36⍝│ r̲oute x: each token's gates over the 4 experts (top 2).
37ᵘr̲oute ← { x → 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner x }
38⍝│ p̲redict x: each token's probabilities over the 3 classes.
39ᵘp̲redict ← { x → ᵐᵒᵉs̲oftmax (ex, ᵘr̲oute x) ᵐᵒᵉc̲ombine t̲anh S '+ '× i̲nner x }
40⍝│ (P, Y) a̲ccuracy kinds: the share of tokens whose likeliest class
41⍝│ is theirs, kind by kind, then over all.
42ᵘa̲ccuracy ← { (P, Y) kinds →
43 right ← '+ r̲/₁ Y × 1 ᵐᵒᵉm̲ask P
44 K ← f̲loat (r̲ange 4) '= t̲able kinds
45 ((K '+ '× i̲nner right) ÷ '+ r̲/₂ K) c̲at ('+ r̲/ right) ÷ f̲loat t̲ally right
46}
47ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 }
48⍝ -- end of the base model ------------------------------
49
50⍝│ The trainable numbers: S, R and the experts (72 + 32 + 96).
51t̲ally w
52⍝│ The accuracy on the 800 training tokens and the 400 held out: kinds
53⍝│ 1 to 4, then all.
54ᵘr̲ound (ᵘp̲redict x, Y) ᵘa̲ccuracy kinds
55ᵘr̲ound (ᵘp̲redict xt, Yt) ᵘa̲ccuracy kindst
56
57⍝━━ The routing, the router frozen
58
59⍝│ On the held-out tokens: the share of each kind's tokens (a column)
60⍝│ each expert (a row) gets; each column sums to 2, the top 2.
61G ← ᵘr̲oute xt
62Kt ← f̲loat (r̲ange 4) '= t̲able kindst
63ᵘr̲ound ((f̲loat G > 0.0) '+ '× i̲nner o̲\ Kt) ÷ 4 4 r̲eshape '+ r̲/₂ Kt
64⍝│ How often each expert is chosen (summing to 2), the routing's
65⍝│ entropy in nats (log 4 = 1.386 for no preference), and its balance
66⍝│ (1 when even, 4 when one expert takes all).
67ᵘr̲ound ᵐᵒᵉf̲requency G
68P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner xt
69ᵘr̲ound (ᵐᵒᵉe̲ntropy P) c̲at (P, G) ᵐᵒᵉb̲alance 2