programdemos/moe-finetune/base.xtl
The base model every fine-tuning method starts from: a tiny mixture-of-experts trained in X_eTaL (train/base.xtl, fixed seeds) on tokens of 8 features in 4 kinds, each token's class (1 to 3) a rule of its kind's own. A shared layer z = tanh(S x), a router R on z, 4 experts at top 2, softmax over the classes. Its baseline: the accuracy on the training and the held-out tokens, kind by kind, and its routing with the router frozen: which experts each kind's tokens go to, how often each expert is chosen, the routing's entropy and balance. A method that changes the experts but not the router keeps this routing; one that trains the router can move it.
The checkpoint and the tokens
ᵘr̲ead : Char -> ((Float, Float), Int)
r̲ead file: (x, the classes one-hot, the kinds) of a data file:
the inputs (9 x N, the 9th a constant 1), the kinds, the classes.
ᵘr̲ead ← { file → v ← n̲umbers ⎕N̲GET file N ← (t̲ally v) d̲iv 11 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v) }
w : Float
The checkpoint: S (8 x 9), R (4 x 8), the experts' Ws (4 x 3 x 8); the experts are LoRA adapters with no update yet (A, B 0, rank 2).
w ← n̲umbers ⎕N̲GET "data/base.txt"
ex : (Float, Float, Float, Float)
ex ← (Ws, 4 2 8 r̲eshape 0.0, 4 3 2 r̲eshape 0.0, 1.0)
The model
ᵘr̲oute : Float -> Float
r̲oute x: each token's gates over the 4 experts (top 2).
ᵘr̲oute ← { x → 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner x }
ᵘp̲redict : Float -> Float
p̲redict x: each token's probabilities over the 3 classes.
ᵘp̲redict ← { x → ᵐᵒᵉs̲oftmax (ex, ᵘr̲oute x) ᵐᵒᵉc̲ombine t̲anh S '+ '× i̲nner x }
ᵘa̲ccuracy : (Float, Float) -> Int -> Float
(P, Y) a̲ccuracy kinds: the share of tokens whose likeliest class
is theirs, kind by kind, then over all.
ᵘa̲ccuracy ← { (P, Y) kinds → right ← '+ r̲/₁ Y × 1 ᵐᵒᵉm̲ask P K ← f̲loat (r̲ange 4) '= t̲able kinds ((K '+ '× i̲nner right) ÷ '+ r̲/₂ K) c̲at ('+ r̲/ right) ÷ f̲loat t̲ally right }
The routing, the router frozen
G : Float
On the held-out tokens: the share of each kind's tokens (a column) each expert (a row) gets; each column sums to 2, the top 2.
G ← ᵘr̲oute xt
P : Float
P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner xt