programdemos/moe-finetune/moe-finetune.xtl
Fine-tuning a tiny mixture-of-experts, each method in turn. The base (base.xtl, trained by train/base.xtl) classifies tokens of 4 kinds, each by its kind's own rule; the new task changes kind 3's rule. A method fine-tunes the base on 200 tokens of the new task (alone, or mixed with 200 of the old task's: replay), the base's own weights frozen, with LoRA adapters (rank 2) on some of its parts: the shared layer only; every expert, the router frozen; both; the two experts the new tokens use most; every expert with the router trained too; or a new expert, added with its own router row, trained alone. Each is measured by the new task's accuracy, the old task's (the kinds whose rule did not change: what fine-tuning must not break, 3 points the bar), the numbers it trains and their bytes, where the tokens are routed before and after, and how large each expert's update is. Then four adapters compared (shared LoRA, expert LoRA, LoRA-FA, shared VeRA), and the adapters deployed: kept apart against merged (LoRA exactly, QLoRA not), two tasks' adapters switched on one base, and an expert cache measuring what each way moves. Last, preference tuning: SFT on either of two valid answers, then DPO on pairs of them, the better preferred.
The tokens
ᵘr̲ead : Char -> ((Float, Float), Int)
r̲ead file: (x, the classes one-hot, the kinds) of a data file:
the inputs (9 x N, the 9th a constant 1), the kinds, the classes.
ᵘr̲ead ← { file → v ← n̲umbers ⎕N̲GET file N ← (t̲ally v) d̲iv 11 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v) }
ᵘc̲olumns : Int -> Float -> Float
idx c̲olumns M: the columns (tokens) idx of M.
ᵘc̲olumns ← { idx M → M '+ '× i̲nner f̲loat (r̲ange 2 s̲elect s̲hape M) '= t̲able idx }
x : Float
The new task: 200 tokens of kind 3 by its new rule to fine-tune on, 200 held out.
((x, Y), _) ← ᵘr̲ead "data/shift-train.txt"
Y : Float
The new task: 200 tokens of kind 3 by its new rule to fine-tune on, 200 held out.
((x, Y), _) ← ᵘr̲ead "data/shift-train.txt"
xa : Float
The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).
((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
Ya : Float
The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).
((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
ka : Int
The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).
((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
xb : Float
Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.
((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
Yb : Float
Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.
((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
kb : Int
Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.
((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
The base, with room for a new expert
w : Float
The base: the shared layer S (8 x 9), the router (4 x 8), the experts (4 x 3 x 8). The expert layer's input here is z with a constant 1 under it (9 x N), so a router row and an expert can have a bias; the base's are 0. A fifth expert waits, 0, its router row a bias of -30 (never in a token's top 2: the gates of the chosen two are as the base's).
w ← n̲umbers ⎕N̲GET "data/base.txt"
R0 : Float
R0 ← (o̲\ (o̲\ 4 8 r̲eshape 32 t̲ake 72 d̲rop w) c̲at 1 4 r̲eshape 0.0) c̲at 1 9 r̲eshape (8 r̲eshape 0.0) c̲at -30.0
W0 : Float
W0 ← 5 3 9 r̲eshape r̲avel (o̲\ (o̲\ 12 8 r̲eshape 104 d̲rop w) c̲at 1 12 r̲eshape 0.0) c̲at 3 9 r̲eshape 0.0
s : Float
The adapters: (aS, bS) on the shared layer (2 x 9, 8 x 2), (As, Bs) on the experts (5 x 2 x 9, 5 x 3 x 2); s = alpha / r, alpha 4 at rank 2.
s ← 4.0 ˡᵒs̲caling 2
ᵘo̲ne : Float -> Float
ᵘo̲ne ← { M → M c̲at (1 c̲at 2 s̲elect s̲hape M) r̲eshape 1.0 }
ᵘl̲ayers : (Float, Float, Float, Float, Float, Float) -> Float -> (Float, Float, Float)
(aS, bS, R, Ws, As, Bs) l̲ayers x: (z, G, P): the shared layer's
output, the gates over the 5 experts, each token's probabilities
over the classes.
ᵘl̲ayers ← { (aS, bS, R, Ws, As, Bs) x → z ← ᵘo̲ne t̲anh (S, aS, bS, s) ˡᵒf̲orward x G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z (z, G, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉc̲ombine z) }
ᵘa̲ccuracy : (Float, Float, Float, Float, Float, Float) -> (Float, Float) -> Float
ᵘa̲ccuracy ← { m (x, Y) → (_, _, P) ← m ᵘl̲ayers x ('+ r̲/ r̲avel Y × 1 ᵐᵒᵉm̲ask P) ÷ f̲loat 2 s̲elect s̲hape Y }
ᵘs̲hare : (Float, Float, Float, Float, Float, Float) -> Float -> Float
m s̲hare x: the share of the tokens x each expert gets (summing to 2).
ᵘs̲hare ← { m x → (_, G, _) ← m ᵘl̲ayers x ᵐᵒᵉf̲requency G }
ᵘp̲er : Num a => a -> a -> a
f p̲er A: each expert's block of A (stacked) times its flag in f.
ᵘp̲er ← { f A → A × (s̲hape A) r̲eshape r̲avel f 'l̲eft t̲able o̲ffsets (t̲ally r̲avel A) d̲iv t̲ally f }
base : (Float, Float, Float, Float, Float, Float)
base ← (2 9 r̲eshape 0.0, 8 2 r̲eshape 0.0, R0, W0, 5 2 9 r̲eshape 0.0, 5 3 2 r̲eshape 0.0)
The new expert's router row
ᵘm̲ean : Float -> Float
It starts pointing at the new tokens and away from the old: the direction from the old training tokens' mean z to the new ones', 6 long, its bias putting 0 halfway between the two means.
ᵘm̲ean ← { M → ('+ r̲/₂ M) ÷ f̲loat 2 s̲elect s̲hape M }
R5 : Float
R5 ← 5 9 r̲eshape (36 t̲ake r̲avel R0) c̲at r5 c̲at n̲eg '+ r̲/ r5 × 0.5 × 8 t̲ake (ᵘm̲ean zn) + ᵘm̲ean zo
A method
ᵘf̲orward6 : (Float, Float, Float, Float, Float, Float) -> Float -> (Float, Float, Float)
A method is which parts train, (shared, experts, router, new): 1 or
0 for the shared layer's adapter, the base router's rows and the new
expert (its weights and router row), a flag per base expert's
adapter. The gradients of the cross-entropy go back through the
experts and the router (MoE's g_rads and g_radsW, each token's two
experts held fixed) and the shared layer (LoRA's g_rads, after tanh);
every expert runs on every token here (d_ense, the same sum as the
sparse c_ombine, faster at this size). A frozen part's gradient is 0,
so Adam leaves it where it is.
m f̲orward6 xs: the model on the tokens xs, (z0, z, P): the shared
layer's output, with its constant 1, each token's probabilities.
ᵘf̲orward6 ← { (aS, bS, R, Ws, As, Bs) xs → z0 ← t̲anh (S, aS, bS, s) ˡᵒf̲orward xs z ← ᵘo̲ne z0 G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z (z0, z, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉd̲ense z) }
ᵘb̲ack6 : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float)
m b̲ack6 (xs, z0, z, g): the gradients for every part, (daS, dbS,
dR, dW, dAs, dBs), from g, the loss's gradient at the softmax's input.
ᵘb̲ack6 ← { (aS, bS, R, Ws, As, Bs) (xs, z0, z, g) → (dR, dAs, dBs, dz) ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲rads (z, g) dW ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲radsW (z, g) (daS, dbS, _) ← (S, aS, bS, s) ˡᵒg̲rads (xs, (8 t̲ake dz) × 1.0 − z0 × z0) (daS, dbS, dR, dW, dAs, dBs) }
ᵘg̲rads6 : (Float, Float, Float, Float, Float, Float) -> (Float, Float) -> (Float, Float, Float, Float, Float, Float, Float)
m g̲rads6 (xs, Ys): the model's loss on the tokens (xs, Ys) and its
gradients for every part, (daS, dbS, dR, dW, dAs, dBs, loss).
ᵘg̲rads6 ← { m (xs, Ys) → (z0, z, P) ← m ᵘf̲orward6 xs (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (P − Ys) ÷ f̲loat 2 s̲elect s̲hape xs) (daS, dbS, dR, dW, dAs, dBs, n̲eg ('+ r̲/ r̲avel Ys × l̲og P) ÷ f̲loat 2 s̲elect s̲hape xs) }
ᵘa̲dam6 : (Float, Float, Float, Float) -> (((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, a), (Float, Float, Float, Float, Float, Float, a)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, a)
method a̲dam6 (state, grads): Adam's step on the parts the method
trains. The state: the model, Adam's averages, the step, the loss
after each step; the gradients and the loss, as g_rads6 gives them.
ᵘa̲dam6 ← { (fS, fE, fR, fN) (((aS, bS, R, Ws, As, Bs), (m1, v1, m2, v2, m3, v3, m4, v4, m5, v5, m6, v6), t, losses), (daS, dbS, dR, dW, dAs, dBs, loss)) → (aS2, m12, v12) ← (0.02, t) ᵒᵖa̲dam (aS, m1, v1, fS × daS) (bS2, m22, v22) ← (0.02, t) ᵒᵖa̲dam (bS, m2, v2, fS × dbS) (R2, m32, v32) ← (0.02, t) ᵒᵖa̲dam (R, m3, v3, ((4 r̲eshape fR) c̲at fN) ᵘp̲er dR) (W2, m42, v42) ← (0.02, t) ᵒᵖa̲dam (Ws, m4, v4, (0.0 0.0 0.0 0.0 c̲at fN) ᵘp̲er dW) (As2, m52, v52) ← (0.02, t) ᵒᵖa̲dam (As, m5, v5, (fE c̲at 0.0) ᵘp̲er dAs) (Bs2, m62, v62) ← (0.02, t) ᵒᵖa̲dam (Bs, m6, v6, (fE c̲at 0.0) ᵘp̲er dBs) ((aS2, bS2, R2, W2, As2, Bs2), (m12, v12, m22, v22, m32, v32, m42, v42, m52, v52, m62, v62), t + 1, losses c̲at loss) }
ᵘs̲tep : ((Float, Float, Float, Float), (Float, Float)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float)
(method, (xs, Ys)) s̲tep state: a step on the cross-entropy of the
tokens (xs, Ys).
ᵘs̲tep ← { (method, tokens) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲rads6 tokens) }
ᵘt̲uneOn : Num a => (Float, (Float, Float, Float, Float), (Float, Float), Int) -> a -> ((Float, Float, Float, Float, Float, Float), Float)
(Wb, method, tokens, steps) t̲uneOn seed: the base, its experts
Wb, fine-tuned by a method on the tokens: each A from the seed
(small), each B 0, so it starts as the base; with a new expert, its
router row as above.
ᵘt̲uneOn ← { (Wb, (fS, fE, fR, fN), tokens, steps) seed → aS ← 0.3 × seed ˡᵒs̲eeded 2 9 As ← 5 2 9 r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded 10 9 R ← R0 + fN × R5 − R0 zero ← (0.0 × aS, 8 2 r̲eshape 0.0, 0.0 × R0, 0.0 × Wb, 0.0 × As, 5 3 2 r̲eshape 0.0) (zS, zB, zR, zW, zA, zE) ← zero (m, _, _, losses) ← steps '{ st → ((fS, fE, fR, fN), tokens) ᵘs̲tep st } p̲ower ((aS, zB, R, Wb, As, zE), (zS, zS, zB, zB, zR, zR, zW, zW, zA, zA, zE, zE), 1, 0 r̲eshape 0.0) (m, losses) }
ᵘt̲une : Num a => ((Float, Float, Float, Float), (Float, Float), Int) -> a -> ((Float, Float, Float, Float, Float, Float), Float)
(method, tokens, steps) t̲une seed: the base itself fine-tuned.
ᵘt̲une ← { (method, tokens, steps) seed → (W0, method, tokens, steps) ᵘt̲uneOn seed }
ᵘs̲ize : Num a => (a, a) -> Float
s̲ize (A, B): an update's size, |s B A| (Frobenius).
ᵘs̲ize ← { (A, B) → s × s̲qrt '+ r̲/ r̲avel (B '+ '× i̲nner A) ^ 2 }
ᵘr̲eport : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float, Float)
m r̲eport method: a method's measures: the new task's accuracy and
the old task's; the numbers trained and their bytes in FP32; the
new and the old tokens' share of each of the 5 experts; the update's
size on each base expert, the shared layer and the new expert (its
weights, from 0).
ᵘr̲eport ← { m (fS, fE, fR, fN) → (aS, bS, R, Ws, As, Bs) ← m n ← (fS × 34.0) + (24.0 × '+ r̲/ fE) + (fR × 36.0) + fN × 36.0 sizes ← '{ e → ᵘs̲ize (e s̲elect As, e s̲elect Bs) } e̲ach r̲ange 4 new ← s̲qrt '+ r̲/ r̲avel (5 s̲elect Ws) ^ 2 (m ᵘa̲ccuracy (xn, Yn), m ᵘa̲ccuracy (xo, Yo), n, 4.0 × n, m ᵘs̲hare xn, m ᵘs̲hare xo, sizes c̲at (ᵘs̲ize (aS, bS)) c̲at new) }
picked : Float
The methods, in the research's order, then a new expert; "selected" takes the two experts the base sends most of the new tokens to.
picked ← 4 t̲ake r̲avel 2 ᵐᵒᵉm̲ask 5 1 r̲eshape base ᵘs̲hare x
Four adapters compared
ᵘp̲ad : Float -> Float
The research's comparison (L4307-4430), the router frozen: shared
LoRA (kind 1, on the shared layer: the research's attention LoRA),
expert LoRA (2), LoRA-FA (3: each expert's A frozen) and shared VeRA
(4: one pair of frozen projections for every expert, A0 (r x 9) and
B0 (3 x r) from the seed, each expert its own d (r numbers) and b
(3)), at any rank r, alpha 4. Each is the model above with its parts
filled in: VeRA's expert e is LoRA's factors diag(d_e) A0 and
diag(b_e) B0 (scale 1), which the MoE library runs as any adapter,
and its gradients follow from theirs: dd_e is dA_e times A0 summed
along each row, db_e is dB_e times B0 likewise.
p̲ad M: the 4 experts' stack M with a fifth of zeros (the waiting
expert).
ᵘp̲ad ← { M → M c̲at (1 c̲at 1 d̲rop s̲hape M) r̲eshape 0.0 }
ᵘe̲ffective : Num a => (a, Int, Float, Float) -> (Float, Float) -> (Float, Float, Float, Float, Float, Float)
(kind, r, A0, B0) e̲ffective (P, Q): the model of the adapter of a
kind at rank r, P and Q its trained arrays (A and B, or d and b).
ᵘe̲ffective ← { (kind, r, A0, B0) (P, Q) → k ← (4.0 ˡᵒs̲caling r) ÷ s kind = 1 ? (P, k × Q, R0, W0, 5 1 9 r̲eshape 0.0, 5 3 1 r̲eshape 0.0) kind < 4 ? (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad P, ᵘp̲ad k × Q) dA ← (4 c̲at r c̲at 9) r̲eshape (r̲avel P) '× t̲able 9 r̲eshape 1.0 dB ← (4 3 c̲at r) r̲eshape (r̲avel Q) '× t̲able r r̲eshape 1.0 (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad dA × (4 c̲at r c̲at 9) r̲eshape A0, ᵘp̲ad (dB × (4 3 c̲at r) r̲eshape B0) ÷ s) }
ᵘc̲hain : Num a => (a, Int, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float)
(kind, r, A0, B0) c̲hain (daS, dbS, dAs, dBs): the model's
gradients back to the trained arrays', (dP, dQ); LoRA-FA's A gets 0.
ᵘc̲hain ← { (kind, r, A0, B0) (daS, dbS, dAs, dBs) → k ← (4.0 ˡᵒs̲caling r) ÷ s kind = 1 ? (daS, k × dbS) dA ← (4 c̲at r c̲at 9) r̲eshape r̲avel dAs dB ← (4 3 c̲at r) r̲eshape r̲avel dBs kind = 2 ? (dA, k × dB) kind = 3 ? (0.0 × dA, k × dB) ('+ r̲/₃ dA × (s̲hape dA) r̲eshape A0, ('+ r̲/₃ dB × (s̲hape dB) r̲eshape B0) ÷ s) }
ᵘs̲tartC : (Num a, Num b) => (a, Int) -> b -> (Float, Float)
(kind, r) s̲tartC seed: where training starts: each A (or A0's
stack) from the seed, as the methods above start; B at 0; VeRA's d
at 0.1 and b at 0.
ᵘs̲tartC ← { (kind, r) seed → kind = 1 ? (0.3 × seed ˡᵒs̲eeded r c̲at 9, (8 c̲at r) r̲eshape 0.0) kind < 4 ? ((4 c̲at r c̲at 9) r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded (4 × r) c̲at 9, (4 3 c̲at r) r̲eshape 0.0) ((4 c̲at r) r̲eshape 0.1, 4 3 r̲eshape 0.0) }
ᵘs̲tepC : Num a => (((a, Int, Float, Float), Float), (Float, Float)) -> ((Float, Float), (Float, Float, Float, Float), Float, Float) -> ((Float, Float), (Float, Float, Float, Float), Float, Float)
((ctx, lr), (xs, Ys)) s̲tepC state: Adam's step on P and Q at the
rate lr.
ᵘs̲tepC ← { ((ctx, lr), (xs, Ys)) ((P, Q), (mP, vP, mQ, vQ), t, losses) → (daS, dbS, _, _, dAs, dBs, loss) ← (ctx ᵘe̲ffective (P, Q)) ᵘg̲rads6 (xs, Ys) (dP, dQ) ← ctx ᵘc̲hain (daS, dbS, dAs, dBs) (P2, mP2, vP2) ← (lr, t) ᵒᵖa̲dam (P, mP, vP, dP) (Q2, mQ2, vQ2) ← (lr, t) ᵒᵖa̲dam (Q, mQ, vQ, dQ) ((P2, Q2), (mP2, vP2, mQ2, vQ2), t + 1, losses c̲at loss) }
ᵘc̲ompare : Num a => (Int, Int, (Float, Float), Int, Float) -> a -> ((Float, Float, Float, Float, Float, Float), Float, Int)
(kind, r, tokens, steps, lr) c̲ompare seed: an adapter of the
kind at rank r fine-tuned on the tokens at the rate lr, VeRA's
projections from the seed: the model, the loss after each step, the
numbers it trains.
ᵘc̲ompare ← { (kind, r, tokens, steps, lr) seed → (A0, B0) ← (seed + 1) ˡᵒp̲rojections 3 9 c̲at r (P, Q) ← (kind, r) ᵘs̲tartC seed ((P2, Q2), _, _, losses) ← steps '{ st → (((kind, r, A0, B0), lr), tokens) ᵘs̲tepC st } p̲ower ((P, Q), (0.0 × P, 0.0 × P, 0.0 × Q, 0.0 × Q), 1, 0 r̲eshape 0.0) ((kind, r, A0, B0) ᵘe̲ffective (P2, Q2), losses, kind s̲elect (17 × r) c̲at (48 × r) c̲at (12 × r) c̲at 4 × 3 + r) }
equalRanks : Int
Equal ranks, 2 each; then equal budgets, about 48 numbers each: shared LoRA at rank 3 (51), expert LoRA at 1 (48), LoRA-FA at 4 (48), VeRA at 9 (48). Each at the methods' rate, 0.02, and at 0.1: an adapter whose projections are frozen moves slowly at LoRA's rate, and VeRA's paper trains at a rate several times LoRA's.
equalRanks ← 2 2 2 2
equalBudgets : Int
equalBudgets ← 3 1 4 9
Deploying
x2 : Float
The research's deployment (L381-460): an adapter kept apart from the base (switched per task) or merged into it (one task, as fast as the base). A second task, for switching: kind 1 by a new rule (200 tokens to fine-tune on, 200 held out); the kinds both tasks leave alone are 2 and 4 (held out, 200).
((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt"
Y2 : Float
The research's deployment (L381-460): an adapter kept apart from the base (switched per task) or merged into it (one task, as fast as the base). A second task, for switching: kind 1 by a new rule (200 tokens to fine-tune on, 200 held out); the kinds both tasks leave alone are 2 and 4 (held out, 200).
((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt"
ᵘd̲eltas : (Float, Float) -> Float
d̲eltas (As, Bs): each expert's update s B A (LoRA's d_elta),
stacked as the experts are (5 x 3 x 9).
ᵘd̲eltas ← { (As, Bs) → (D, _) ← 5 '{ (D, e) → (D c̲at r̲avel ˡᵒd̲elta (e s̲elect As, e s̲elect Bs, s), e + 1) } p̲ower (0 r̲eshape 0.0, 1) 5 3 9 r̲eshape D }
ᵘm̲erged : (Any a, Any b, Any c, Any d, Any e) => a -> (b, c, d, e, Float, Float) -> (b, c, d, a, Float, Float)
Wm m̲erged m: the model m with its experts' adapters folded into
the weights Wm (the adapters then 0): for LoRA, Ws + s B A; for
QLoRA, that quantized again.
ᵘm̲erged ← { Wm (aS, bS, R, Ws, As, Bs) → (aS, bS, R, Wm, 0.0 × As, 0.0 × Bs) }
Wq : Float
QLoRA's base: the experts in NF4, blocks of 9 (a row each), and what it computes with, W~, dequantized.
Wq ← ˡᵒd̲equant 9 ˡᵒq̲base W0
xh : Float
m a̲gainst m2: two models on every held-out token (both tasks' and
the old task's, 800): the largest difference of their probabilities,
and the tokens whose likeliest class differs.
xh ← o̲\ (o̲\ xn) c̲at (o̲\ xn2) c̲at o̲\ xa
ᵘa̲gainst : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float) -> Float
ᵘa̲gainst ← { m m2 → (_, _, P) ← m ᵘl̲ayers xh (_, _, P2) ← m2 ᵘl̲ayers xh ('m̲ax r̲/ r̲avel a̲bs P − P2) c̲at 0.5 × '+ r̲/ r̲avel a̲bs (1 ᵐᵒᵉm̲ask P) − 1 ᵐᵒᵉm̲ask P2 }
ᵘt̲asks : (Float, Float, Float, Float, Float, Float) -> Float
m t̲asks: a model on the first task (kind 3's new rule), the
second (kind 1's) and the kinds neither changed.
ᵘt̲asks ← { m → (m ᵘa̲ccuracy (xn, Yn)) c̲at (m ᵘa̲ccuracy (xn2, Yn2)) c̲at m ᵘa̲ccuracy (xe, Ye) }
G1 : Float
The cache: the held-out tokens of the two tasks in batches of 8, one task's batch then the other's (50 batches), each batch's experts as the base's router sends them (fine-tuning the experts leaves the routing as it was). Kept apart, the base experts are one cache's keys (4) and the adapters another's, one per task and expert (8); merged, each task has its own copy of every expert (8 keys).
(_, G1, _) ← base ᵘl̲ayers xn
Ua : Float
Ua ← o̲\ 50 8 r̲eshape r̲avel o̲\ U1 c̲at (0.0 × U1) c̲at (0.0 × U2) c̲at U2
ᵘt̲raffic : (Float, Float, Float, Float) -> Int -> Float
sizes t̲raffic c: with c slots in each cache, the loads (base
experts, adapters or merged copies) and what moved: merged copies,
then the base experts and LoRA, LoRA-FA and VeRA adapters kept apart,
in bytes, sizes the bytes of (an expert, a LoRA adapter, LoRA-FA's B,
VeRA's d and b), the frozen A and projections made from their seeds.
ᵘt̲raffic ← { (ne, nl, nf, nv) c → lb ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ub la ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ua (f̲loat c) c̲at lb c̲at la c̲at (la × ne) c̲at (lb × ne) + la × nl c̲at nf c̲at nv }
Preference: supervised fine-tuning, then DPO
ᵘo̲nehot : Num a => a -> Float
The research's order (L1475-1490, L2023-2046): supervised
fine-tuning (SFT) first, then preference pairs (DPO), both answers
of a pair valid, so the model learns a preference and not right from
wrong; reinforcement learning waits until those show a gain. Here a
new-task token's two best classes by its rule both count as right
(like two solutions that both pass the tests), and the better of the
two, its class, is preferred (the cleaner one). SFT on demonstrations
of either can learn what is right but not which is better.
c o̲nehot: the one-hot columns of the classes c.
ᵘo̲nehot ← { c → f̲loat (r̲ange 3) '= t̲able f̲loor c }
ᵘv̲alid : Num a => (Float, a) -> (Float, Float, Float)
v̲alid (Y, c2): (valid, preferred, rejected), one-hot: the two best
classes, the best (Y itself), the second best (c2, from its file).
ᵘv̲alid ← { (Yc, c2) → (Yc + ᵘo̲nehot c2, Yc, ᵘo̲nehot c2) }
ᵘp̲refers : (Float, Float, Float, Float, Float, Float) -> Float
m p̲refers x: how often a model's answer is one of the two valid
classes, and how often it is the preferred one, on the held-out new
tokens; then the old task's accuracy.
ᵘp̲refers ← { m → (_, _, P) ← m ᵘl̲ayers xn A ← 1 ᵐᵒᵉm̲ask P N ← f̲loat 2 s̲elect s̲hape xn (('+ r̲/ r̲avel A × Vn) ÷ N) c̲at (('+ r̲/ r̲avel A × Ywn) ÷ N) c̲at m ᵘa̲ccuracy (xo, Yo) }
ᵘg̲radsD : Num a => (Float, Float, Float, Float, Float, Float) -> ((Float, Float, Float), a, Float) -> (Float, Float, Float, Float, Float, Float, Float)
m g̲radsD ((xs, Yw, Yl), Pr, b): DPO's loss (Rafailov et al., 2023)
and gradients on the pairs: Yw the chosen answer of each token, Yl
the rejected, Pr the reference model's probabilities, b its beta. With
h = b ((log P - log Pr) . (Yw - Yl)) for a token, the loss is -log
sigma(h), and its gradient at the softmax's input is -b sigma(-h) (Yw
- Yl): the softmax's own part cancels between the two answers.
ᵘg̲radsD ← { m ((xs, Wy, Ly), Pr, b) → (z0, z, P) ← m ᵘf̲orward6 xs d ← Wy − Ly h ← b × '+ r̲/₁ d × (l̲og P) − l̲og Pr N ← f̲loat 2 s̲elect s̲hape xs q ← (s̲hape d) r̲eshape 1.0 ÷ 1.0 + e̲xp h (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (n̲eg b × q × d) ÷ N) (daS, dbS, dR, dW, dAs, dBs, ('+ r̲/ l̲og 1.0 + e̲xp n̲eg h) ÷ N) }
ᵘs̲tepD : Num a => ((Float, Float, Float, Float), ((Float, Float, Float), a, Float)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float)
(method, pairs) s̲tepD state: a step of DPO on the pairs.
ᵘs̲tepD ← { (method, pairs) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲radsD pairs) }
ᵘd̲po : ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float), (Float, Float, Float), Float) -> Int -> ((Float, Float, Float, Float, Float, Float), Float)
(m0, method, (xs, Yw, Yl), b) d̲po steps: DPO from the model m0,
which is also the reference, Adam starting afresh: the model and the
loss after each step.
ᵘd̲po ← { (m0, method, (xs, Yw2, Yl2), b) steps → (_, _, Pr) ← m0 ᵘf̲orward6 xs (aS, bS, R, Ws, As, Bs) ← m0 (m, _, _, losses) ← steps '{ st → (method, ((xs, Yw2, Yl2), Pr, b)) ᵘs̲tepD st } p̲ower (m0, (0.0 × aS, 0.0 × aS, 0.0 × bS, 0.0 × bS, 0.0 × R, 0.0 × R, 0.0 × Ws, 0.0 × Ws, 0.0 × As, 0.0 × As, 0.0 × Bs, 0.0 × Bs), 1, 0 r̲eshape 0.0) (m, losses) }
styled : (Float, Float, Float)
The pairs: the house style's, both valid (preferred, rejected); and, to show what the research warns of, each valid answer against the invalid one (two pairs a token), which teaches right from wrong only.
styled ← (x, Yw, Yl)
checked : (Float, Float, Float)
checked ← (o̲\ (o̲\ x) c̲at o̲\ x, o̲\ (o̲\ Yw) c̲at o̲\ Yl, o̲\ (o̲\ wrong) c̲at o̲\ wrong)
Each method, seed 1, 120 steps
ᵘr̲ow : ((Float, Float, Float, Float), (Float, Float)) -> Float
A line per method: the new task's accuracy, the old task's, the numbers trained, their bytes, the new tokens' share of each of the 5 experts, the old tokens', then the update sizes (the 4 experts, the shared layer, the new expert).
ᵘr̲ow ← { (method, tokens) → (m, _) ← (method, tokens, 120) ᵘt̲une 1 (a, o, n, b, sn, so, sz) ← m ᵘr̲eport method ᵘr̲ound a c̲at o c̲at n c̲at b c̲at sn c̲at so c̲at sz }
ᵘa̲ll : (Float, Float) -> Float
ᵘa̲ll ← { tokens → 7 20 r̲eshape (ᵘr̲ow (none, tokens)) c̲at (ᵘr̲ow (shared, tokens)) c̲at (ᵘr̲ow (experts, tokens)) c̲at (ᵘr̲ow (both, tokens)) c̲at (ᵘr̲ow (selected, tokens)) c̲at (ᵘr̲ow (router, tokens)) c̲at ᵘr̲ow (added, tokens) }
The four adapters, seed 1, 120 steps, on the new tokens alone
ᵘl̲ine : (Int, Int, Float) -> Float
A line per adapter (shared LoRA, expert LoRA, LoRA-FA, shared VeRA): the new task's accuracy, the old task's, the numbers trained; at equal ranks, then at equal budgets; at the rate 0.02, then 0.1.
ᵘl̲ine ← { (kind, r, lr) → (m, _, n) ← (kind, r, (x, Y), 120, lr) ᵘc̲ompare 1 (ᵘr̲ound m ᵘa̲ccuracy (xn, Yn)) c̲at (ᵘr̲ound m ᵘa̲ccuracy (xo, Yo)) c̲at f̲loat n }
ᵘf̲our : (Int, Float) -> Float
ᵘf̲our ← { (ranks, lr) → 4 3 r̲eshape (ᵘl̲ine (1, 1 s̲elect ranks, lr)) c̲at (ᵘl̲ine (2, 2 s̲elect ranks, lr)) c̲at (ᵘl̲ine (3, 3 s̲elect ranks, lr)) c̲at ᵘl̲ine (4, 4 s̲elect ranks, lr) }
Deploying, seed 1, 120 steps
m1 : (Float, Float, Float, Float, Float, Float)
Kept apart against merged, the new task's adapters on every expert (the router frozen): LoRA on the base, then QLoRA on its 4-bit base (trained on W~, merged by quantizing W~ + s B A again). A line each: the largest difference of the probabilities, the tokens whose answer changed (of 800), the new task's accuracy kept apart, then merged.
(m1, _) ← (experts, (x, Y), 120) ᵘt̲une 1
mm : (Float, Float, Float, Float, Float, Float)
mm ← (ˡᵒd̲equant 9 ˡᵒq̲base Wq + ᵘd̲eltas (Aq, Bq)) ᵘm̲erged mq
m2 : (Float, Float, Float, Float, Float, Float)
Two tasks' adapters on one base: the base, then with the first task's adapters, then with the second's (switched: the same base, the other adapters), each on the first task, the second and the kinds neither changed.
(m2, _) ← (experts, (x2, Y2), 120) ᵘt̲une 1
be : Float
At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.
(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)
bl : Float
At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.
(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)
bf : Float
At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.
(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)
bv : Float
At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.
(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)
Preference, seed 1, 120 steps each
mp : (Float, Float, Float, Float, Float, Float)
(mp, _) ← (experts, (x, Y), 120) ᵘt̲une 1
md1 : (Float, Float, Float, Float, Float, Float)
(md1, _) ← (ms, experts, styled, 1.0) ᵘd̲po 120