sourcedemos/moe-finetune/moe-finetune.xtl

1⍝!/usr/bin/env xetal 2⍝│ Fine-tuning a tiny mixture-of-experts, each method in turn. The base 3⍝│ (base.xtl, trained by train/base.xtl) classifies tokens of 4 kinds, 4⍝│ each by its kind's own rule; the new task changes kind 3's rule. A 5⍝│ method fine-tunes the base on 200 tokens of the new task (alone, or 6⍝│ mixed with 200 of the old task's: replay), the base's own weights 7⍝│ frozen, with LoRA adapters (rank 2) on some of its parts: the shared 8⍝│ layer only; every expert, the router frozen; both; the two experts 9⍝│ the new tokens use most; every expert with the router trained too; 10⍝│ or a new expert, added with its own router row, trained alone. Each 11⍝│ is measured by the new task's accuracy, the old task's (the kinds 12⍝│ whose rule did not change: what fine-tuning must not break, 3 points 13⍝│ the bar), the numbers it trains and their bytes, where the tokens are 14⍝│ routed before and after, and how large each expert's update is. 15⍝│ Then four adapters compared (shared LoRA, expert LoRA, LoRA-FA, 16⍝│ shared VeRA), and the adapters deployed: kept apart against merged 17⍝│ (LoRA exactly, QLoRA not), two tasks' adapters switched on one base, 18⍝│ and an expert cache measuring what each way moves. Last, preference 19⍝│ tuning: SFT on either of two valid answers, then DPO on pairs of 20⍝│ them, the better preferred. 21 22"moe:" u̲se< "MoE" 23"lo:" u̲se< "LoRA" 24"op:" u̲se< "Optim" 25 26⍝━━ The tokens 27 28⍝│ r̲ead file: (x, the classes one-hot, the kinds) of a data file: 29⍝│ the inputs (9 x N, the 9th a constant 1), the kinds, the classes. 30ᵘr̲ead ← { file → 31 v ← n̲umbers ⎕N̲GET file 32 N ← (t̲ally v) d̲iv 11 33 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v) 34} 35⍝│ idx c̲olumns M: the columns (tokens) idx of M. 36ᵘc̲olumns ← { idx M → M '+ '× i̲nner f̲loat (r̲ange 2 s̲elect s̲hape M) '= t̲able idx } 37⍝│ The new task: 200 tokens of kind 3 by its new rule to fine-tune on, 38⍝│ 200 held out. 39((x, Y), _) ← ᵘr̲ead "data/shift-train.txt" 40((xn, Yn), _) ← ᵘr̲ead "data/shift-test.txt" 41⍝│ The old task: the held-out tokens of kinds 1, 2 and 4, whose rule 42⍝│ the new task keeps (300). 43((xa, Ya), ka) ← ᵘr̲ead "data/test.txt" 44(xo, Yo) ← ((w̲here ka ≠ 3) ᵘc̲olumns xa, (w̲here ka ≠ 3) ᵘc̲olumns Ya) 45⍝│ Replay: the new tokens mixed with 200 of the old task's training 46⍝│ tokens (kinds 1, 2 and 4), the usual guard against forgetting. 47((xb, Yb), kb) ← ᵘr̲ead "data/train.txt" 48old ← 200 t̲ake w̲here kb ≠ 3 49(xr, Yr) ← (old ᵘc̲olumns xb, old ᵘc̲olumns Yb) 50mixed ← (o̲\ (o̲\ x) c̲at o̲\ xr, o̲\ (o̲\ Y) c̲at o̲\ Yr) 51 52⍝━━ The base, with room for a new expert 53 54⍝│ The base: the shared layer S (8 x 9), the router (4 x 8), the 55⍝│ experts (4 x 3 x 8). The expert layer's input here is z with a 56⍝│ constant 1 under it (9 x N), so a router row and an expert can have 57⍝│ a bias; the base's are 0. A fifth expert waits, 0, its router row a 58⍝│ bias of -30 (never in a token's top 2: the gates of the chosen two 59⍝│ are as the base's). 60w ← n̲umbers ⎕N̲GET "data/base.txt" 61S ← 8 9 r̲eshape 72 t̲ake w 62R0 ← (o̲\ (o̲\ 4 8 r̲eshape 32 t̲ake 72 d̲rop w) c̲at 1 4 r̲eshape 0.0) c̲at 1 9 r̲eshape (8 r̲eshape 0.0) c̲at -30.0 63W0 ← 5 3 9 r̲eshape r̲avel (o̲\ (o̲\ 12 8 r̲eshape 104 d̲rop w) c̲at 1 12 r̲eshape 0.0) c̲at 3 9 r̲eshape 0.0 64⍝│ The adapters: (aS, bS) on the shared layer (2 x 9, 8 x 2), (As, Bs) 65⍝│ on the experts (5 x 2 x 9, 5 x 3 x 2); s = alpha / r, alpha 4 at 66⍝│ rank 2. 67s ← 4.0 ˡᵒs̲caling 2 68ᵘo̲ne ← { M → M c̲at (1 c̲at 2 s̲elect s̲hape M) r̲eshape 1.0 } 69⍝│ (aS, bS, R, Ws, As, Bs) l̲ayers x: (z, G, P): the shared layer's 70⍝│ output, the gates over the 5 experts, each token's probabilities 71⍝│ over the classes. 72ᵘl̲ayers ← { (aS, bS, R, Ws, As, Bs) x → 73 z ← ᵘo̲ne t̲anh (S, aS, bS, s) ˡᵒf̲orward x 74 G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z 75 (z, G, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉc̲ombine z) 76} 77⍝│ m a̲ccuracy (x, Y): the share of tokens whose likeliest class is 78⍝│ theirs. 79ᵘa̲ccuracy ← { m (x, Y) → 80 (_, _, P) ← m ᵘl̲ayers x 81 ('+ r̲/ r̲avel Y × 1 ᵐᵒᵉm̲ask P) ÷ f̲loat 2 s̲elect s̲hape Y 82} 83⍝│ m s̲hare x: the share of the tokens x each expert gets (summing to 2). 84ᵘs̲hare ← { m x → 85 (_, G, _) ← m ᵘl̲ayers x 86 ᵐᵒᵉf̲requency G 87} 88⍝│ f p̲er A: each expert's block of A (stacked) times its flag in f. 89ᵘp̲er ← { f A → A × (s̲hape A) r̲eshape r̲avel f 'l̲eft t̲able o̲ffsets (t̲ally r̲avel A) d̲iv t̲ally f } 90base ← (2 9 r̲eshape 0.0, 8 2 r̲eshape 0.0, R0, W0, 5 2 9 r̲eshape 0.0, 5 3 2 r̲eshape 0.0) 91 92⍝━━ The new expert's router row 93 94⍝│ It starts pointing at the new tokens and away from the old: the 95⍝│ direction from the old training tokens' mean z to the new ones', 6 96⍝│ long, its bias putting 0 halfway between the two means. 97ᵘm̲ean ← { M → ('+ r̲/₂ M) ÷ f̲loat 2 s̲elect s̲hape M } 98(zn, _, _) ← base ᵘl̲ayers x 99(zo, _, _) ← base ᵘl̲ayers xr 100d ← (8 t̲ake ᵘm̲ean zn) − 8 t̲ake ᵘm̲ean zo 101r5 ← 6.0 × d ÷ s̲qrt '+ r̲/ d × d 102R5 ← 5 9 r̲eshape (36 t̲ake r̲avel R0) c̲at r5 c̲at n̲eg '+ r̲/ r5 × 0.5 × 8 t̲ake (ᵘm̲ean zn) + ᵘm̲ean zo 103 104⍝━━ A method 105 106⍝│ A method is which parts train, (shared, experts, router, new): 1 or 107⍝│ 0 for the shared layer's adapter, the base router's rows and the new 108⍝│ expert (its weights and router row), a flag per base expert's 109⍝│ adapter. The gradients of the cross-entropy go back through the 110⍝│ experts and the router (MoE's g_rads and g_radsW, each token's two 111⍝│ experts held fixed) and the shared layer (LoRA's g_rads, after tanh); 112⍝│ every expert runs on every token here (d_ense, the same sum as the 113⍝│ sparse c_ombine, faster at this size). A frozen part's gradient is 0, 114⍝│ so Adam leaves it where it is. 115⍝│ m f̲orward6 xs: the model on the tokens xs, (z0, z, P): the shared 116⍝│ layer's output, with its constant 1, each token's probabilities. 117ᵘf̲orward6 ← { (aS, bS, R, Ws, As, Bs) xs → 118 z0 ← t̲anh (S, aS, bS, s) ˡᵒf̲orward xs 119 z ← ᵘo̲ne z0 120 G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z 121 (z0, z, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉd̲ense z) 122} 123⍝│ m b̲ack6 (xs, z0, z, g): the gradients for every part, (daS, dbS, 124⍝│ dR, dW, dAs, dBs), from g, the loss's gradient at the softmax's input. 125ᵘb̲ack6 ← { (aS, bS, R, Ws, As, Bs) (xs, z0, z, g) → 126 (dR, dAs, dBs, dz) ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲rads (z, g) 127 dW ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲radsW (z, g) 128 (daS, dbS, _) ← (S, aS, bS, s) ˡᵒg̲rads (xs, (8 t̲ake dz) × 1.0 − z0 × z0) 129 (daS, dbS, dR, dW, dAs, dBs) 130} 131⍝│ m g̲rads6 (xs, Ys): the model's loss on the tokens (xs, Ys) and its 132⍝│ gradients for every part, (daS, dbS, dR, dW, dAs, dBs, loss). 133ᵘg̲rads6 ← { m (xs, Ys) → 134 (z0, z, P) ← m ᵘf̲orward6 xs 135 (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (P − Ys) ÷ f̲loat 2 s̲elect s̲hape xs) 136 (daS, dbS, dR, dW, dAs, dBs, n̲eg ('+ r̲/ r̲avel Ys × l̲og P) ÷ f̲loat 2 s̲elect s̲hape xs) 137} 138⍝│ method a̲dam6 (state, grads): Adam's step on the parts the method 139⍝│ trains. The state: the model, Adam's averages, the step, the loss 140⍝│ after each step; the gradients and the loss, as g_rads6 gives them. 141ᵘa̲dam6 ← { (fS, fE, fR, fN) (((aS, bS, R, Ws, As, Bs), (m1, v1, m2, v2, m3, v3, m4, v4, m5, v5, m6, v6), t, losses), (daS, dbS, dR, dW, dAs, dBs, loss)) → 142 (aS2, m12, v12) ← (0.02, t) ᵒᵖa̲dam (aS, m1, v1, fS × daS) 143 (bS2, m22, v22) ← (0.02, t) ᵒᵖa̲dam (bS, m2, v2, fS × dbS) 144 (R2, m32, v32) ← (0.02, t) ᵒᵖa̲dam (R, m3, v3, ((4 r̲eshape fR) c̲at fN) ᵘp̲er dR) 145 (W2, m42, v42) ← (0.02, t) ᵒᵖa̲dam (Ws, m4, v4, (0.0 0.0 0.0 0.0 c̲at fN) ᵘp̲er dW) 146 (As2, m52, v52) ← (0.02, t) ᵒᵖa̲dam (As, m5, v5, (fE c̲at 0.0) ᵘp̲er dAs) 147 (Bs2, m62, v62) ← (0.02, t) ᵒᵖa̲dam (Bs, m6, v6, (fE c̲at 0.0) ᵘp̲er dBs) 148 ((aS2, bS2, R2, W2, As2, Bs2), (m12, v12, m22, v22, m32, v32, m42, v42, m52, v52, m62, v62), t + 1, losses c̲at loss) 149} 150⍝│ (method, (xs, Ys)) s̲tep state: a step on the cross-entropy of the 151⍝│ tokens (xs, Ys). 152ᵘs̲tep ← { (method, tokens) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲rads6 tokens) } 153⍝│ (Wb, method, tokens, steps) t̲uneOn seed: the base, its experts 154⍝│ Wb, fine-tuned by a method on the tokens: each A from the seed 155⍝│ (small), each B 0, so it starts as the base; with a new expert, its 156⍝│ router row as above. 157ᵘt̲uneOn ← { (Wb, (fS, fE, fR, fN), tokens, steps) seed → 158 aS ← 0.3 × seed ˡᵒs̲eeded 2 9 159 As ← 5 2 9 r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded 10 9 160 R ← R0 + fN × R5 − R0 161 zero ← (0.0 × aS, 8 2 r̲eshape 0.0, 0.0 × R0, 0.0 × Wb, 0.0 × As, 5 3 2 r̲eshape 0.0) 162 (zS, zB, zR, zW, zA, zE) ← zero 163 (m, _, _, losses) ← steps '{ st → ((fS, fE, fR, fN), tokens) ᵘs̲tep st } p̲ower ((aS, zB, R, Wb, As, zE), (zS, zS, zB, zB, zR, zR, zW, zW, zA, zA, zE, zE), 1, 0 r̲eshape 0.0) 164 (m, losses) 165} 166⍝│ (method, tokens, steps) t̲une seed: the base itself fine-tuned. 167ᵘt̲une ← { (method, tokens, steps) seed → (W0, method, tokens, steps) ᵘt̲uneOn seed } 168⍝│ s̲ize (A, B): an update's size, |s B A| (Frobenius). 169ᵘs̲ize ← { (A, B) → s × s̲qrt '+ r̲/ r̲avel (B '+ '× i̲nner A) ^ 2 } 170⍝│ m r̲eport method: a method's measures: the new task's accuracy and 171⍝│ the old task's; the numbers trained and their bytes in FP32; the 172⍝│ new and the old tokens' share of each of the 5 experts; the update's 173⍝│ size on each base expert, the shared layer and the new expert (its 174⍝│ weights, from 0). 175ᵘr̲eport ← { m (fS, fE, fR, fN) → 176 (aS, bS, R, Ws, As, Bs) ← m 177 n ← (fS × 34.0) + (24.0 × '+ r̲/ fE) + (fR × 36.0) + fN × 36.0 178 sizes ← '{ e → ᵘs̲ize (e s̲elect As, e s̲elect Bs) } e̲ach r̲ange 4 179 new ← s̲qrt '+ r̲/ r̲avel (5 s̲elect Ws) ^ 2 180 (m ᵘa̲ccuracy (xn, Yn), m ᵘa̲ccuracy (xo, Yo), n, 4.0 × n, m ᵘs̲hare xn, m ᵘs̲hare xo, sizes c̲at (ᵘs̲ize (aS, bS)) c̲at new) 181} 182⍝│ The methods, in the research's order, then a new expert; "selected" 183⍝│ takes the two experts the base sends most of the new tokens to. 184picked ← 4 t̲ake r̲avel 2 ᵐᵒᵉm̲ask 5 1 r̲eshape base ᵘs̲hare x 185none ← (0.0, 4 r̲eshape 0.0, 0.0, 0.0) 186shared ← (1.0, 4 r̲eshape 0.0, 0.0, 0.0) 187experts ← (0.0, 4 r̲eshape 1.0, 0.0, 0.0) 188both ← (1.0, 4 r̲eshape 1.0, 0.0, 0.0) 189selected ← (0.0, picked, 0.0, 0.0) 190router ← (0.0, 4 r̲eshape 1.0, 1.0, 0.0) 191added ← (0.0, 4 r̲eshape 0.0, 0.0, 1.0) 192ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 } 193 194⍝━━ Four adapters compared 195 196⍝│ The research's comparison (L4307-4430), the router frozen: shared 197⍝│ LoRA (kind 1, on the shared layer: the research's attention LoRA), 198⍝│ expert LoRA (2), LoRA-FA (3: each expert's A frozen) and shared VeRA 199⍝│ (4: one pair of frozen projections for every expert, A0 (r x 9) and 200⍝│ B0 (3 x r) from the seed, each expert its own d (r numbers) and b 201⍝│ (3)), at any rank r, alpha 4. Each is the model above with its parts 202⍝│ filled in: VeRA's expert e is LoRA's factors diag(d_e) A0 and 203⍝│ diag(b_e) B0 (scale 1), which the MoE library runs as any adapter, 204⍝│ and its gradients follow from theirs: dd_e is dA_e times A0 summed 205⍝│ along each row, db_e is dB_e times B0 likewise. 206⍝│ p̲ad M: the 4 experts' stack M with a fifth of zeros (the waiting 207⍝│ expert). 208ᵘp̲ad ← { M → M c̲at (1 c̲at 1 d̲rop s̲hape M) r̲eshape 0.0 } 209⍝│ (kind, r, A0, B0) e̲ffective (P, Q): the model of the adapter of a 210⍝│ kind at rank r, P and Q its trained arrays (A and B, or d and b). 211ᵘe̲ffective ← { (kind, r, A0, B0) (P, Q) → 212 k ← (4.0 ˡᵒs̲caling r) ÷ s 213 kind = 1 ? (P, k × Q, R0, W0, 5 1 9 r̲eshape 0.0, 5 3 1 r̲eshape 0.0) 214 kind < 4 ? (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad P, ᵘp̲ad k × Q) 215 dA ← (4 c̲at r c̲at 9) r̲eshape (r̲avel P) '× t̲able 9 r̲eshape 1.0 216 dB ← (4 3 c̲at r) r̲eshape (r̲avel Q) '× t̲able r r̲eshape 1.0 217 (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad dA × (4 c̲at r c̲at 9) r̲eshape A0, ᵘp̲ad (dB × (4 3 c̲at r) r̲eshape B0) ÷ s) 218} 219⍝│ (kind, r, A0, B0) c̲hain (daS, dbS, dAs, dBs): the model's 220⍝│ gradients back to the trained arrays', (dP, dQ); LoRA-FA's A gets 0. 221ᵘc̲hain ← { (kind, r, A0, B0) (daS, dbS, dAs, dBs) → 222 k ← (4.0 ˡᵒs̲caling r) ÷ s 223 kind = 1 ? (daS, k × dbS) 224 dA ← (4 c̲at r c̲at 9) r̲eshape r̲avel dAs 225 dB ← (4 3 c̲at r) r̲eshape r̲avel dBs 226 kind = 2 ? (dA, k × dB) 227 kind = 3 ? (0.0 × dA, k × dB) 228 ('+ r̲/₃ dA × (s̲hape dA) r̲eshape A0, ('+ r̲/₃ dB × (s̲hape dB) r̲eshape B0) ÷ s) 229} 230⍝│ (kind, r) s̲tartC seed: where training starts: each A (or A0's 231⍝│ stack) from the seed, as the methods above start; B at 0; VeRA's d 232⍝│ at 0.1 and b at 0. 233ᵘs̲tartC ← { (kind, r) seed → 234 kind = 1 ? (0.3 × seed ˡᵒs̲eeded r c̲at 9, (8 c̲at r) r̲eshape 0.0) 235 kind < 4 ? ((4 c̲at r c̲at 9) r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded (4 × r) c̲at 9, (4 3 c̲at r) r̲eshape 0.0) 236 ((4 c̲at r) r̲eshape 0.1, 4 3 r̲eshape 0.0) 237} 238⍝│ ((ctx, lr), (xs, Ys)) s̲tepC state: Adam's step on P and Q at the 239⍝│ rate lr. 240ᵘs̲tepC ← { ((ctx, lr), (xs, Ys)) ((P, Q), (mP, vP, mQ, vQ), t, losses) → 241 (daS, dbS, _, _, dAs, dBs, loss) ← (ctx ᵘe̲ffective (P, Q)) ᵘg̲rads6 (xs, Ys) 242 (dP, dQ) ← ctx ᵘc̲hain (daS, dbS, dAs, dBs) 243 (P2, mP2, vP2) ← (lr, t) ᵒᵖa̲dam (P, mP, vP, dP) 244 (Q2, mQ2, vQ2) ← (lr, t) ᵒᵖa̲dam (Q, mQ, vQ, dQ) 245 ((P2, Q2), (mP2, vP2, mQ2, vQ2), t + 1, losses c̲at loss) 246} 247⍝│ (kind, r, tokens, steps, lr) c̲ompare seed: an adapter of the 248⍝│ kind at rank r fine-tuned on the tokens at the rate lr, VeRA's 249⍝│ projections from the seed: the model, the loss after each step, the 250⍝│ numbers it trains. 251ᵘc̲ompare ← { (kind, r, tokens, steps, lr) seed → 252 (A0, B0) ← (seed + 1) ˡᵒp̲rojections 3 9 c̲at r 253 (P, Q) ← (kind, r) ᵘs̲tartC seed 254 ((P2, Q2), _, _, losses) ← steps '{ st → (((kind, r, A0, B0), lr), tokens) ᵘs̲tepC st } p̲ower ((P, Q), (0.0 × P, 0.0 × P, 0.0 × Q, 0.0 × Q), 1, 0 r̲eshape 0.0) 255 ((kind, r, A0, B0) ᵘe̲ffective (P2, Q2), losses, kind s̲elect (17 × r) c̲at (48 × r) c̲at (12 × r) c̲at 4 × 3 + r) 256} 257⍝│ Equal ranks, 2 each; then equal budgets, about 48 numbers each: 258⍝│ shared LoRA at rank 3 (51), expert LoRA at 1 (48), LoRA-FA at 4 259⍝│ (48), VeRA at 9 (48). Each at the methods' rate, 0.02, and at 0.1: 260⍝│ an adapter whose projections are frozen moves slowly at LoRA's rate, 261⍝│ and VeRA's paper trains at a rate several times LoRA's. 262equalRanks ← 2 2 2 2 263equalBudgets ← 3 1 4 9 264 265⍝━━ Deploying 266 267⍝│ The research's deployment (L381-460): an adapter kept apart from the 268⍝│ base (switched per task) or merged into it (one task, as fast as the 269⍝│ base). A second task, for switching: kind 1 by a new rule (200 270⍝│ tokens to fine-tune on, 200 held out); the kinds both tasks leave 271⍝│ alone are 2 and 4 (held out, 200). 272((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt" 273((xn2, Yn2), _) ← ᵘr̲ead "data/shift2-test.txt" 274(xe, Ye) ← ((w̲here 0 = ka m̲od 2) ᵘc̲olumns xa, (w̲here 0 = ka m̲od 2) ᵘc̲olumns Ya) 275⍝│ d̲eltas (As, Bs): each expert's update s B A (LoRA's d_elta), 276⍝│ stacked as the experts are (5 x 3 x 9). 277ᵘd̲eltas ← { (As, Bs) → 278 (D, _) ← 5 '{ (D, e) → (D c̲at r̲avel ˡᵒd̲elta (e s̲elect As, e s̲elect Bs, s), e + 1) } p̲ower (0 r̲eshape 0.0, 1) 279 5 3 9 r̲eshape D 280} 281⍝│ Wm m̲erged m: the model m with its experts' adapters folded into 282⍝│ the weights Wm (the adapters then 0): for LoRA, Ws + s B A; for 283⍝│ QLoRA, that quantized again. 284ᵘm̲erged ← { Wm (aS, bS, R, Ws, As, Bs) → (aS, bS, R, Wm, 0.0 × As, 0.0 × Bs) } 285⍝│ QLoRA's base: the experts in NF4, blocks of 9 (a row each), and what 286⍝│ it computes with, W~, dequantized. 287Wq ← ˡᵒd̲equant 9 ˡᵒq̲base W0 288⍝│ m a̲gainst m2: two models on every held-out token (both tasks' and 289⍝│ the old task's, 800): the largest difference of their probabilities, 290⍝│ and the tokens whose likeliest class differs. 291xh ← o̲\ (o̲\ xn) c̲at (o̲\ xn2) c̲at o̲\ xa 292ᵘa̲gainst ← { m m2 → 293 (_, _, P) ← m ᵘl̲ayers xh 294 (_, _, P2) ← m2 ᵘl̲ayers xh 295 ('m̲ax r̲/ r̲avel a̲bs P − P2) c̲at 0.5 × '+ r̲/ r̲avel a̲bs (1 ᵐᵒᵉm̲ask P) − 1 ᵐᵒᵉm̲ask P2 296} 297⍝│ m t̲asks: a model on the first task (kind 3's new rule), the 298⍝│ second (kind 1's) and the kinds neither changed. 299ᵘt̲asks ← { m → (m ᵘa̲ccuracy (xn, Yn)) c̲at (m ᵘa̲ccuracy (xn2, Yn2)) c̲at m ᵘa̲ccuracy (xe, Ye) } 300⍝│ The cache: the held-out tokens of the two tasks in batches of 8, 301⍝│ one task's batch then the other's (50 batches), each batch's 302⍝│ experts as the base's router sends them (fine-tuning the experts 303⍝│ leaves the routing as it was). Kept apart, the base experts are one 304⍝│ cache's keys (4) and the adapters another's, one per task and expert 305⍝│ (8); merged, each task has its own copy of every expert (8 keys). 306(_, G1, _) ← base ᵘl̲ayers xn 307(_, G2, _) ← base ᵘl̲ayers xn2 308U1 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G1 309U2 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G2 310Ub ← o̲\ 50 4 r̲eshape r̲avel o̲\ U1 c̲at U2 311Ua ← o̲\ 50 8 r̲eshape r̲avel o̲\ U1 c̲at (0.0 × U1) c̲at (0.0 × U2) c̲at U2 312⍝│ sizes t̲raffic c: with c slots in each cache, the loads (base 313⍝│ experts, adapters or merged copies) and what moved: merged copies, 314⍝│ then the base experts and LoRA, LoRA-FA and VeRA adapters kept apart, 315⍝│ in bytes, sizes the bytes of (an expert, a LoRA adapter, LoRA-FA's B, 316⍝│ VeRA's d and b), the frozen A and projections made from their seeds. 317ᵘt̲raffic ← { (ne, nl, nf, nv) c → 318 lb ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ub 319 la ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ua 320 (f̲loat c) c̲at lb c̲at la c̲at (la × ne) c̲at (lb × ne) + la × nl c̲at nf c̲at nv 321} 322⍝━━ Preference: supervised fine-tuning, then DPO 323 324⍝│ The research's order (L1475-1490, L2023-2046): supervised 325⍝│ fine-tuning (SFT) first, then preference pairs (DPO), both answers 326⍝│ of a pair valid, so the model learns a preference and not right from 327⍝│ wrong; reinforcement learning waits until those show a gain. Here a 328⍝│ new-task token's two best classes by its rule both count as right 329⍝│ (like two solutions that both pass the tests), and the better of the 330⍝│ two, its class, is preferred (the cleaner one). SFT on demonstrations 331⍝│ of either can learn what is right but not which is better. 332⍝│ c o̲nehot: the one-hot columns of the classes c. 333ᵘo̲nehot ← { c → f̲loat (r̲ange 3) '= t̲able f̲loor c } 334⍝│ v̲alid (Y, c2): (valid, preferred, rejected), one-hot: the two best 335⍝│ classes, the best (Y itself), the second best (c2, from its file). 336ᵘv̲alid ← { (Yc, c2) → (Yc + ᵘo̲nehot c2, Yc, ᵘo̲nehot c2) } 337(V, Yw, Yl) ← ᵘv̲alid (Y, n̲umbers ⎕N̲GET "data/shift-second-train.txt") 338(Vn, Ywn, _) ← ᵘv̲alid (Yn, n̲umbers ⎕N̲GET "data/shift-second-test.txt") 339⍝│ m p̲refers x: how often a model's answer is one of the two valid 340⍝│ classes, and how often it is the preferred one, on the held-out new 341⍝│ tokens; then the old task's accuracy. 342ᵘp̲refers ← { m → 343 (_, _, P) ← m ᵘl̲ayers xn 344 A ← 1 ᵐᵒᵉm̲ask P 345 N ← f̲loat 2 s̲elect s̲hape xn 346 (('+ r̲/ r̲avel A × Vn) ÷ N) c̲at (('+ r̲/ r̲avel A × Ywn) ÷ N) c̲at m ᵘa̲ccuracy (xo, Yo) 347} 348⍝│ m g̲radsD ((xs, Yw, Yl), Pr, b): DPO's loss (Rafailov et al., 2023) 349⍝│ and gradients on the pairs: Yw the chosen answer of each token, Yl 350⍝│ the rejected, Pr the reference model's probabilities, b its beta. With 351⍝│ h = b ((log P - log Pr) . (Yw - Yl)) for a token, the loss is -log 352⍝│ sigma(h), and its gradient at the softmax's input is -b sigma(-h) (Yw 353⍝│ - Yl): the softmax's own part cancels between the two answers. 354ᵘg̲radsD ← { m ((xs, Wy, Ly), Pr, b) → 355 (z0, z, P) ← m ᵘf̲orward6 xs 356 d ← Wy − Ly 357 h ← b × '+ r̲/₁ d × (l̲og P) − l̲og Pr 358 N ← f̲loat 2 s̲elect s̲hape xs 359 q ← (s̲hape d) r̲eshape 1.0 ÷ 1.0 + e̲xp h 360 (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (n̲eg b × q × d) ÷ N) 361 (daS, dbS, dR, dW, dAs, dBs, ('+ r̲/ l̲og 1.0 + e̲xp n̲eg h) ÷ N) 362} 363⍝│ (method, pairs) s̲tepD state: a step of DPO on the pairs. 364ᵘs̲tepD ← { (method, pairs) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲radsD pairs) } 365⍝│ (m0, method, (xs, Yw, Yl), b) d̲po steps: DPO from the model m0, 366⍝│ which is also the reference, Adam starting afresh: the model and the 367⍝│ loss after each step. 368ᵘd̲po ← { (m0, method, (xs, Yw2, Yl2), b) steps → 369 (_, _, Pr) ← m0 ᵘf̲orward6 xs 370 (aS, bS, R, Ws, As, Bs) ← m0 371 (m, _, _, losses) ← steps '{ st → (method, ((xs, Yw2, Yl2), Pr, b)) ᵘs̲tepD st } p̲ower (m0, (0.0 × aS, 0.0 × aS, 0.0 × bS, 0.0 × bS, 0.0 × R, 0.0 × R, 0.0 × Ws, 0.0 × Ws, 0.0 × As, 0.0 × As, 0.0 × Bs, 0.0 × Bs), 1, 0 r̲eshape 0.0) 372 (m, losses) 373} 374⍝│ The pairs: the house style's, both valid (preferred, rejected); and, 375⍝│ to show what the research warns of, each valid answer against the 376⍝│ invalid one (two pairs a token), which teaches right from wrong only. 377styled ← (x, Yw, Yl) 378wrong ← 1.0 − V 379checked ← (o̲\ (o̲\ x) c̲at o̲\ x, o̲\ (o̲\ Yw) c̲at o̲\ Yl, o̲\ (o̲\ wrong) c̲at o̲\ wrong) 380⍝ -- end of the core ------------------------------ 381 382⍝━━ Each method, seed 1, 120 steps 383 384⍝│ A line per method: the new task's accuracy, the old task's, the 385⍝│ numbers trained, their bytes, the new tokens' share of each of the 5 386⍝│ experts, the old tokens', then the update sizes (the 4 experts, the 387⍝│ shared layer, the new expert). 388ᵘr̲ow ← { (method, tokens) → 389 (m, _) ← (method, tokens, 120) ᵘt̲une 1 390 (a, o, n, b, sn, so, sz) ← m ᵘr̲eport method 391 ᵘr̲ound a c̲at o c̲at n c̲at b c̲at sn c̲at so c̲at sz 392} 393ᵘa̲ll ← { tokens → 7 20 r̲eshape (ᵘr̲ow (none, tokens)) c̲at (ᵘr̲ow (shared, tokens)) c̲at (ᵘr̲ow (experts, tokens)) c̲at (ᵘr̲ow (both, tokens)) c̲at (ᵘr̲ow (selected, tokens)) c̲at (ᵘr̲ow (router, tokens)) c̲at ᵘr̲ow (added, tokens) } 394⍝│ Fine-tuned on the new tokens alone, then mixed with old ones. 395ᵘa̲ll (x, Y) 396ᵘa̲ll mixed 397 398⍝━━ The four adapters, seed 1, 120 steps, on the new tokens alone 399 400⍝│ A line per adapter (shared LoRA, expert LoRA, LoRA-FA, shared VeRA): 401⍝│ the new task's accuracy, the old task's, the numbers trained; at 402⍝│ equal ranks, then at equal budgets; at the rate 0.02, then 0.1. 403ᵘl̲ine ← { (kind, r, lr) → 404 (m, _, n) ← (kind, r, (x, Y), 120, lr) ᵘc̲ompare 1 405 (ᵘr̲ound m ᵘa̲ccuracy (xn, Yn)) c̲at (ᵘr̲ound m ᵘa̲ccuracy (xo, Yo)) c̲at f̲loat n 406} 407ᵘf̲our ← { (ranks, lr) → 4 3 r̲eshape (ᵘl̲ine (1, 1 s̲elect ranks, lr)) c̲at (ᵘl̲ine (2, 2 s̲elect ranks, lr)) c̲at (ᵘl̲ine (3, 3 s̲elect ranks, lr)) c̲at ᵘl̲ine (4, 4 s̲elect ranks, lr) } 408ᵘf̲our (equalRanks, 0.02) 409ᵘf̲our (equalBudgets, 0.02) 410ᵘf̲our (equalRanks, 0.1) 411ᵘf̲our (equalBudgets, 0.1) 412 413⍝━━ Deploying, seed 1, 120 steps 414 415⍝│ Kept apart against merged, the new task's adapters on every expert 416⍝│ (the router frozen): LoRA on the base, then QLoRA on its 4-bit base 417⍝│ (trained on W~, merged by quantizing W~ + s B A again). A line each: 418⍝│ the largest difference of the probabilities, the tokens whose answer 419⍝│ changed (of 800), the new task's accuracy kept apart, then merged. 420(m1, _) ← (experts, (x, Y), 120) ᵘt̲une 1 421(aS1, bS1, R1, W1, A1, B1) ← m1 422⍝ -- end of the first task's adapters 423(mq, _) ← (Wq, experts, (x, Y), 120) ᵘt̲uneOn 1 424(_, _, _, _, Aq, Bq) ← mq 425ml ← (W1 + ᵘd̲eltas (A1, B1)) ᵘm̲erged m1 426mm ← (ˡᵒd̲equant 9 ˡᵒq̲base Wq + ᵘd̲eltas (Aq, Bq)) ᵘm̲erged mq 427(m1 ᵘa̲gainst ml) c̲at (m1 ᵘa̲ccuracy (xn, Yn)) c̲at ml ᵘa̲ccuracy (xn, Yn) 428(mq ᵘa̲gainst mm) c̲at (mq ᵘa̲ccuracy (xn, Yn)) c̲at mm ᵘa̲ccuracy (xn, Yn) 429⍝ -- end of the merge 430⍝│ Two tasks' adapters on one base: the base, then with the first 431⍝│ task's adapters, then with the second's (switched: the same base, 432⍝│ the other adapters), each on the first task, the second and the 433⍝│ kinds neither changed. 434(m2, _) ← (experts, (x2, Y2), 120) ᵘt̲une 1 435(_, _, _, _, A2, B2) ← m2 4363 3 r̲eshape (ᵘr̲ound ᵘt̲asks base) c̲at (ᵘr̲ound ᵘt̲asks m1) c̲at ᵘr̲ound ᵘt̲asks (aS1, bS1, R1, W1, A2, B2) 437⍝ The bytes each keeps, FP32: the base (200 numbers), a task's adapters 438⍝ (96), a merged copy (the base again); for the two tasks kept apart, 439⍝ then merged. 440800 384 800 c̲at (800 + 2 × 384) c̲at 2 × 800 441⍝ -- end of the switch 442⍝ The cache, 1 to 4 slots in each: slots, base-expert loads, adapter 443⍝ (or merged-copy) loads, then the bytes moved: merged copies; kept 444⍝ apart with LoRA, LoRA-FA and VeRA adapters (rank 2: an expert 27 445⍝ numbers, LoRA 24, LoRA-FA's B 6, VeRA 5). 4464 7 r̲eshape ((108, 96, 24, 20) ᵘt̲raffic 1) c̲at ((108, 96, 24, 20) ᵘt̲raffic 2) c̲at ((108, 96, 24, 20) ᵘt̲raffic 3) c̲at (108, 96, 24, 20) ᵘt̲raffic 4 447⍝│ At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 448⍝│ in BF16 (the LoRA library's counts): the expert, its LoRA adapters, 449⍝│ LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots. 450(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336) 451be c̲at bl c̲at bf c̲at bv 452(be, bl, bf, bv) ᵘt̲raffic 2 453⍝ -- end of the cache 454 455⍝━━ Preference, seed 1, 120 steps each 456 457⍝ A line each: how often the answer on the held-out new tokens is 458⍝ valid (one of the two best), how often it is the preferred (the 459⍝ best), and the old task's accuracy. The base; SFT on the preferred 460⍝ answers alone (the method "every expert"); SFT on demonstrations of 461⍝ either valid answer; that, then DPO on the pairs (the better against 462⍝ the other valid), beta 1 and 5; then DPO on each valid answer against 463⍝ the invalid one, beta 1. 464(mp, _) ← (experts, (x, Y), 120) ᵘt̲une 1 465(ms, _) ← (experts, (x, 0.5 × V), 120) ᵘt̲une 1 466(md1, _) ← (ms, experts, styled, 1.0) ᵘd̲po 120 467(md5, _) ← (ms, experts, styled, 5.0) ᵘd̲po 120 468(mc, _) ← (ms, experts, checked, 1.0) ᵘd̲po 120 4696 3 r̲eshape (ᵘr̲ound ᵘp̲refers base) c̲at (ᵘr̲ound ᵘp̲refers mp) c̲at (ᵘr̲ound ᵘp̲refers ms) c̲at (ᵘr̲ound ᵘp̲refers md1) c̲at (ᵘr̲ound ᵘp̲refers md5) c̲at ᵘr̲ound ᵘp̲refers mc