programdemos/moe-finetune/base.xtl

The base model every fine-tuning method starts from: a tiny mixture-of-experts trained in X_eTaL (train/base.xtl, fixed seeds) on tokens of 8 features in 4 kinds, each token's class (1 to 3) a rule of its kind's own. A shared layer z = tanh(S x), a router R on z, 4 experts at top 2, softmax over the classes. Its baseline: the accuracy on the training and the held-out tokens, kind by kind, and its routing with the router frozen: which experts each kind's tokens go to, how often each expert is chosen, the routing's entropy and balance. A method that changes the experts but not the router keeps this routing; one that trains the router can move it.

source · imports moe: libs/MoE/src/MoE.xtl

The checkpoint and the tokens

ᵘr̲ead : Char -> ((Float, Float), Int)

function · line 19

r̲ead file: (x, the classes one-hot, the kinds) of a data file: the inputs (9 x N, the 9th a constant 1), the kinds, the classes.

ᵘr̲ead ← { file →
  v ← n̲umbers ⎕N̲GET file
  N ← (t̲ally v) d̲iv 11
  (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v)
}
Used in: x, xt

x : Float

value · line 24
((x, Y), kinds) ← ᵘr̲ead "data/train.txt"

Y : Float

value · line 24
((x, Y), kinds) ← ᵘr̲ead "data/train.txt"

kinds : Int

value · line 24
((x, Y), kinds) ← ᵘr̲ead "data/train.txt"

xt : Float

value · line 25
((xt, Yt), kindst) ← ᵘr̲ead "data/test.txt"

Yt : Float

value · line 25
((xt, Yt), kindst) ← ᵘr̲ead "data/test.txt"

kindst : Int

value · line 25
((xt, Yt), kindst) ← ᵘr̲ead "data/test.txt"

w : Float

value · line 28

The checkpoint: S (8 x 9), R (4 x 8), the experts' Ws (4 x 3 x 8); the experts are LoRA adapters with no update yet (A, B 0, rank 2).

w ← n̲umbers ⎕N̲GET "data/base.txt"

S : Float

value · line 29
S ← 8 9 r̲eshape 72 t̲ake w

R : Float

value · line 30
R ← 4 8 r̲eshape 32 t̲ake 72 d̲rop w
Used in: ᵘr̲oute, P

Ws : Float

value · line 31
Ws ← 4 3 8 r̲eshape 104 d̲rop w
Used in: ex

ex : (Float, Float, Float, Float)

value · line 32
ex ← (Ws, 4 2 8 r̲eshape 0.0, 4 3 2 r̲eshape 0.0, 1.0)
Used in: ᵘp̲redict

The model

ᵘr̲oute : Float -> Float

function · line 37

r̲oute x: each token's gates over the 4 experts (top 2).

ᵘr̲oute ← { x → 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner x }
Used in: ᵘp̲redict, G

ᵘp̲redict : Float -> Float

function · line 39

p̲redict x: each token's probabilities over the 3 classes.

ᵘp̲redict ← { x → ᵐᵒᵉs̲oftmax (ex, ᵘr̲oute x) ᵐᵒᵉc̲ombine t̲anh S '+ '× i̲nner x }

ᵘa̲ccuracy : (Float, Float) -> Int -> Float

function · line 42

(P, Y) a̲ccuracy kinds: the share of tokens whose likeliest class is theirs, kind by kind, then over all.

ᵘa̲ccuracy ← { (P, Y) kinds →
  right ← '+ r̲/₁ Y × 1 ᵐᵒᵉm̲ask P
  K ← f̲loat (r̲ange 4) '= t̲able kinds
  ((K '+ '× i̲nner right) ÷ '+ r̲/₂ K) c̲at ('+ r̲/ right) ÷ f̲loat t̲ally right
}

ᵘr̲ound : Float -> Float

function · line 47
ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 }

The routing, the router frozen

G : Float

value · line 61

On the held-out tokens: the share of each kind's tokens (a column) each expert (a row) gets; each column sums to 2, the top 2.

G ← ᵘr̲oute xt

Kt : Float

value · line 62
Kt ← f̲loat (r̲ange 4) '= t̲able kindst

P : Float

value · line 68
P ← ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores t̲anh S '+ '× i̲nner xt