sourcedemos/moe-finetune/moe-finetune.xtl
1⍝!/usr/bin/env xetal
2⍝│ Fine-tuning a tiny mixture-of-experts, each method in turn. The base
3⍝│ (base.xtl, trained by train/base.xtl) classifies tokens of 4 kinds,
4⍝│ each by its kind's own rule; the new task changes kind 3's rule. A
5⍝│ method fine-tunes the base on 200 tokens of the new task (alone, or
6⍝│ mixed with 200 of the old task's: replay), the base's own weights
7⍝│ frozen, with LoRA adapters (rank 2) on some of its parts: the shared
8⍝│ layer only; every expert, the router frozen; both; the two experts
9⍝│ the new tokens use most; every expert with the router trained too;
10⍝│ or a new expert, added with its own router row, trained alone. Each
11⍝│ is measured by the new task's accuracy, the old task's (the kinds
12⍝│ whose rule did not change: what fine-tuning must not break, 3 points
13⍝│ the bar), the numbers it trains and their bytes, where the tokens are
14⍝│ routed before and after, and how large each expert's update is.
15⍝│ Then four adapters compared (shared LoRA, expert LoRA, LoRA-FA,
16⍝│ shared VeRA), and the adapters deployed: kept apart against merged
17⍝│ (LoRA exactly, QLoRA not), two tasks' adapters switched on one base,
18⍝│ and an expert cache measuring what each way moves. Last, preference
19⍝│ tuning: SFT on either of two valid answers, then DPO on pairs of
20⍝│ them, the better preferred.
21
22"moe:" u̲se< "MoE"
23"lo:" u̲se< "LoRA"
24"op:" u̲se< "Optim"
25
26⍝━━ The tokens
27
28⍝│ r̲ead file: (x, the classes one-hot, the kinds) of a data file:
29⍝│ the inputs (9 x N, the 9th a constant 1), the kinds, the classes.
30ᵘr̲ead ← { file →
31 v ← n̲umbers ⎕N̲GET file
32 N ← (t̲ally v) d̲iv 11
33 (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v)
34}
35⍝│ idx c̲olumns M: the columns (tokens) idx of M.
36ᵘc̲olumns ← { idx M → M '+ '× i̲nner f̲loat (r̲ange 2 s̲elect s̲hape M) '= t̲able idx }
37⍝│ The new task: 200 tokens of kind 3 by its new rule to fine-tune on,
38⍝│ 200 held out.
39((x, Y), _) ← ᵘr̲ead "data/shift-train.txt"
40((xn, Yn), _) ← ᵘr̲ead "data/shift-test.txt"
41⍝│ The old task: the held-out tokens of kinds 1, 2 and 4, whose rule
42⍝│ the new task keeps (300).
43((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
44(xo, Yo) ← ((w̲here ka ≠ 3) ᵘc̲olumns xa, (w̲here ka ≠ 3) ᵘc̲olumns Ya)
45⍝│ Replay: the new tokens mixed with 200 of the old task's training
46⍝│ tokens (kinds 1, 2 and 4), the usual guard against forgetting.
47((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
48old ← 200 t̲ake w̲here kb ≠ 3
49(xr, Yr) ← (old ᵘc̲olumns xb, old ᵘc̲olumns Yb)
50mixed ← (o̲\ (o̲\ x) c̲at o̲\ xr, o̲\ (o̲\ Y) c̲at o̲\ Yr)
51
52⍝━━ The base, with room for a new expert
53
54⍝│ The base: the shared layer S (8 x 9), the router (4 x 8), the
55⍝│ experts (4 x 3 x 8). The expert layer's input here is z with a
56⍝│ constant 1 under it (9 x N), so a router row and an expert can have
57⍝│ a bias; the base's are 0. A fifth expert waits, 0, its router row a
58⍝│ bias of -30 (never in a token's top 2: the gates of the chosen two
59⍝│ are as the base's).
60w ← n̲umbers ⎕N̲GET "data/base.txt"
61S ← 8 9 r̲eshape 72 t̲ake w
62R0 ← (o̲\ (o̲\ 4 8 r̲eshape 32 t̲ake 72 d̲rop w) c̲at 1 4 r̲eshape 0.0) c̲at 1 9 r̲eshape (8 r̲eshape 0.0) c̲at -30.0
63W0 ← 5 3 9 r̲eshape r̲avel (o̲\ (o̲\ 12 8 r̲eshape 104 d̲rop w) c̲at 1 12 r̲eshape 0.0) c̲at 3 9 r̲eshape 0.0
64⍝│ The adapters: (aS, bS) on the shared layer (2 x 9, 8 x 2), (As, Bs)
65⍝│ on the experts (5 x 2 x 9, 5 x 3 x 2); s = alpha / r, alpha 4 at
66⍝│ rank 2.
67s ← 4.0 ˡᵒs̲caling 2
68ᵘo̲ne ← { M → M c̲at (1 c̲at 2 s̲elect s̲hape M) r̲eshape 1.0 }
69⍝│ (aS, bS, R, Ws, As, Bs) l̲ayers x: (z, G, P): the shared layer's
70⍝│ output, the gates over the 5 experts, each token's probabilities
71⍝│ over the classes.
72ᵘl̲ayers ← { (aS, bS, R, Ws, As, Bs) x →
73 z ← ᵘo̲ne t̲anh (S, aS, bS, s) ˡᵒf̲orward x
74 G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z
75 (z, G, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉc̲ombine z)
76}
77⍝│ m a̲ccuracy (x, Y): the share of tokens whose likeliest class is
78⍝│ theirs.
79ᵘa̲ccuracy ← { m (x, Y) →
80 (_, _, P) ← m ᵘl̲ayers x
81 ('+ r̲/ r̲avel Y × 1 ᵐᵒᵉm̲ask P) ÷ f̲loat 2 s̲elect s̲hape Y
82}
83⍝│ m s̲hare x: the share of the tokens x each expert gets (summing to 2).
84ᵘs̲hare ← { m x →
85 (_, G, _) ← m ᵘl̲ayers x
86 ᵐᵒᵉf̲requency G
87}
88⍝│ f p̲er A: each expert's block of A (stacked) times its flag in f.
89ᵘp̲er ← { f A → A × (s̲hape A) r̲eshape r̲avel f 'l̲eft t̲able o̲ffsets (t̲ally r̲avel A) d̲iv t̲ally f }
90base ← (2 9 r̲eshape 0.0, 8 2 r̲eshape 0.0, R0, W0, 5 2 9 r̲eshape 0.0, 5 3 2 r̲eshape 0.0)
91
92⍝━━ The new expert's router row
93
94⍝│ It starts pointing at the new tokens and away from the old: the
95⍝│ direction from the old training tokens' mean z to the new ones', 6
96⍝│ long, its bias putting 0 halfway between the two means.
97ᵘm̲ean ← { M → ('+ r̲/₂ M) ÷ f̲loat 2 s̲elect s̲hape M }
98(zn, _, _) ← base ᵘl̲ayers x
99(zo, _, _) ← base ᵘl̲ayers xr
100d ← (8 t̲ake ᵘm̲ean zn) − 8 t̲ake ᵘm̲ean zo
101r5 ← 6.0 × d ÷ s̲qrt '+ r̲/ d × d
102R5 ← 5 9 r̲eshape (36 t̲ake r̲avel R0) c̲at r5 c̲at n̲eg '+ r̲/ r5 × 0.5 × 8 t̲ake (ᵘm̲ean zn) + ᵘm̲ean zo
103
104⍝━━ A method
105
106⍝│ A method is which parts train, (shared, experts, router, new): 1 or
107⍝│ 0 for the shared layer's adapter, the base router's rows and the new
108⍝│ expert (its weights and router row), a flag per base expert's
109⍝│ adapter. The gradients of the cross-entropy go back through the
110⍝│ experts and the router (MoE's g_rads and g_radsW, each token's two
111⍝│ experts held fixed) and the shared layer (LoRA's g_rads, after tanh);
112⍝│ every expert runs on every token here (d_ense, the same sum as the
113⍝│ sparse c_ombine, faster at this size). A frozen part's gradient is 0,
114⍝│ so Adam leaves it where it is.
115⍝│ m f̲orward6 xs: the model on the tokens xs, (z0, z, P): the shared
116⍝│ layer's output, with its constant 1, each token's probabilities.
117ᵘf̲orward6 ← { (aS, bS, R, Ws, As, Bs) xs →
118 z0 ← t̲anh (S, aS, bS, s) ˡᵒf̲orward xs
119 z ← ᵘo̲ne z0
120 G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z
121 (z0, z, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉd̲ense z)
122}
123⍝│ m b̲ack6 (xs, z0, z, g): the gradients for every part, (daS, dbS,
124⍝│ dR, dW, dAs, dBs), from g, the loss's gradient at the softmax's input.
125ᵘb̲ack6 ← { (aS, bS, R, Ws, As, Bs) (xs, z0, z, g) →
126 (dR, dAs, dBs, dz) ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲rads (z, g)
127 dW ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲radsW (z, g)
128 (daS, dbS, _) ← (S, aS, bS, s) ˡᵒg̲rads (xs, (8 t̲ake dz) × 1.0 − z0 × z0)
129 (daS, dbS, dR, dW, dAs, dBs)
130}
131⍝│ m g̲rads6 (xs, Ys): the model's loss on the tokens (xs, Ys) and its
132⍝│ gradients for every part, (daS, dbS, dR, dW, dAs, dBs, loss).
133ᵘg̲rads6 ← { m (xs, Ys) →
134 (z0, z, P) ← m ᵘf̲orward6 xs
135 (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (P − Ys) ÷ f̲loat 2 s̲elect s̲hape xs)
136 (daS, dbS, dR, dW, dAs, dBs, n̲eg ('+ r̲/ r̲avel Ys × l̲og P) ÷ f̲loat 2 s̲elect s̲hape xs)
137}
138⍝│ method a̲dam6 (state, grads): Adam's step on the parts the method
139⍝│ trains. The state: the model, Adam's averages, the step, the loss
140⍝│ after each step; the gradients and the loss, as g_rads6 gives them.
141ᵘa̲dam6 ← { (fS, fE, fR, fN) (((aS, bS, R, Ws, As, Bs), (m1, v1, m2, v2, m3, v3, m4, v4, m5, v5, m6, v6), t, losses), (daS, dbS, dR, dW, dAs, dBs, loss)) →
142 (aS2, m12, v12) ← (0.02, t) ᵒᵖa̲dam (aS, m1, v1, fS × daS)
143 (bS2, m22, v22) ← (0.02, t) ᵒᵖa̲dam (bS, m2, v2, fS × dbS)
144 (R2, m32, v32) ← (0.02, t) ᵒᵖa̲dam (R, m3, v3, ((4 r̲eshape fR) c̲at fN) ᵘp̲er dR)
145 (W2, m42, v42) ← (0.02, t) ᵒᵖa̲dam (Ws, m4, v4, (0.0 0.0 0.0 0.0 c̲at fN) ᵘp̲er dW)
146 (As2, m52, v52) ← (0.02, t) ᵒᵖa̲dam (As, m5, v5, (fE c̲at 0.0) ᵘp̲er dAs)
147 (Bs2, m62, v62) ← (0.02, t) ᵒᵖa̲dam (Bs, m6, v6, (fE c̲at 0.0) ᵘp̲er dBs)
148 ((aS2, bS2, R2, W2, As2, Bs2), (m12, v12, m22, v22, m32, v32, m42, v42, m52, v52, m62, v62), t + 1, losses c̲at loss)
149}
150⍝│ (method, (xs, Ys)) s̲tep state: a step on the cross-entropy of the
151⍝│ tokens (xs, Ys).
152ᵘs̲tep ← { (method, tokens) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲rads6 tokens) }
153⍝│ (Wb, method, tokens, steps) t̲uneOn seed: the base, its experts
154⍝│ Wb, fine-tuned by a method on the tokens: each A from the seed
155⍝│ (small), each B 0, so it starts as the base; with a new expert, its
156⍝│ router row as above.
157ᵘt̲uneOn ← { (Wb, (fS, fE, fR, fN), tokens, steps) seed →
158 aS ← 0.3 × seed ˡᵒs̲eeded 2 9
159 As ← 5 2 9 r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded 10 9
160 R ← R0 + fN × R5 − R0
161 zero ← (0.0 × aS, 8 2 r̲eshape 0.0, 0.0 × R0, 0.0 × Wb, 0.0 × As, 5 3 2 r̲eshape 0.0)
162 (zS, zB, zR, zW, zA, zE) ← zero
163 (m, _, _, losses) ← steps '{ st → ((fS, fE, fR, fN), tokens) ᵘs̲tep st } p̲ower ((aS, zB, R, Wb, As, zE), (zS, zS, zB, zB, zR, zR, zW, zW, zA, zA, zE, zE), 1, 0 r̲eshape 0.0)
164 (m, losses)
165}
166⍝│ (method, tokens, steps) t̲une seed: the base itself fine-tuned.
167ᵘt̲une ← { (method, tokens, steps) seed → (W0, method, tokens, steps) ᵘt̲uneOn seed }
168⍝│ s̲ize (A, B): an update's size, |s B A| (Frobenius).
169ᵘs̲ize ← { (A, B) → s × s̲qrt '+ r̲/ r̲avel (B '+ '× i̲nner A) ^ 2 }
170⍝│ m r̲eport method: a method's measures: the new task's accuracy and
171⍝│ the old task's; the numbers trained and their bytes in FP32; the
172⍝│ new and the old tokens' share of each of the 5 experts; the update's
173⍝│ size on each base expert, the shared layer and the new expert (its
174⍝│ weights, from 0).
175ᵘr̲eport ← { m (fS, fE, fR, fN) →
176 (aS, bS, R, Ws, As, Bs) ← m
177 n ← (fS × 34.0) + (24.0 × '+ r̲/ fE) + (fR × 36.0) + fN × 36.0
178 sizes ← '{ e → ᵘs̲ize (e s̲elect As, e s̲elect Bs) } e̲ach r̲ange 4
179 new ← s̲qrt '+ r̲/ r̲avel (5 s̲elect Ws) ^ 2
180 (m ᵘa̲ccuracy (xn, Yn), m ᵘa̲ccuracy (xo, Yo), n, 4.0 × n, m ᵘs̲hare xn, m ᵘs̲hare xo, sizes c̲at (ᵘs̲ize (aS, bS)) c̲at new)
181}
182⍝│ The methods, in the research's order, then a new expert; "selected"
183⍝│ takes the two experts the base sends most of the new tokens to.
184picked ← 4 t̲ake r̲avel 2 ᵐᵒᵉm̲ask 5 1 r̲eshape base ᵘs̲hare x
185none ← (0.0, 4 r̲eshape 0.0, 0.0, 0.0)
186shared ← (1.0, 4 r̲eshape 0.0, 0.0, 0.0)
187experts ← (0.0, 4 r̲eshape 1.0, 0.0, 0.0)
188both ← (1.0, 4 r̲eshape 1.0, 0.0, 0.0)
189selected ← (0.0, picked, 0.0, 0.0)
190router ← (0.0, 4 r̲eshape 1.0, 1.0, 0.0)
191added ← (0.0, 4 r̲eshape 0.0, 0.0, 1.0)
192ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 }
193
194⍝━━ Four adapters compared
195
196⍝│ The research's comparison (L4307-4430), the router frozen: shared
197⍝│ LoRA (kind 1, on the shared layer: the research's attention LoRA),
198⍝│ expert LoRA (2), LoRA-FA (3: each expert's A frozen) and shared VeRA
199⍝│ (4: one pair of frozen projections for every expert, A0 (r x 9) and
200⍝│ B0 (3 x r) from the seed, each expert its own d (r numbers) and b
201⍝│ (3)), at any rank r, alpha 4. Each is the model above with its parts
202⍝│ filled in: VeRA's expert e is LoRA's factors diag(d_e) A0 and
203⍝│ diag(b_e) B0 (scale 1), which the MoE library runs as any adapter,
204⍝│ and its gradients follow from theirs: dd_e is dA_e times A0 summed
205⍝│ along each row, db_e is dB_e times B0 likewise.
206⍝│ p̲ad M: the 4 experts' stack M with a fifth of zeros (the waiting
207⍝│ expert).
208ᵘp̲ad ← { M → M c̲at (1 c̲at 1 d̲rop s̲hape M) r̲eshape 0.0 }
209⍝│ (kind, r, A0, B0) e̲ffective (P, Q): the model of the adapter of a
210⍝│ kind at rank r, P and Q its trained arrays (A and B, or d and b).
211ᵘe̲ffective ← { (kind, r, A0, B0) (P, Q) →
212 k ← (4.0 ˡᵒs̲caling r) ÷ s
213 kind = 1 ? (P, k × Q, R0, W0, 5 1 9 r̲eshape 0.0, 5 3 1 r̲eshape 0.0)
214 kind < 4 ? (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad P, ᵘp̲ad k × Q)
215 dA ← (4 c̲at r c̲at 9) r̲eshape (r̲avel P) '× t̲able 9 r̲eshape 1.0
216 dB ← (4 3 c̲at r) r̲eshape (r̲avel Q) '× t̲able r r̲eshape 1.0
217 (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad dA × (4 c̲at r c̲at 9) r̲eshape A0, ᵘp̲ad (dB × (4 3 c̲at r) r̲eshape B0) ÷ s)
218}
219⍝│ (kind, r, A0, B0) c̲hain (daS, dbS, dAs, dBs): the model's
220⍝│ gradients back to the trained arrays', (dP, dQ); LoRA-FA's A gets 0.
221ᵘc̲hain ← { (kind, r, A0, B0) (daS, dbS, dAs, dBs) →
222 k ← (4.0 ˡᵒs̲caling r) ÷ s
223 kind = 1 ? (daS, k × dbS)
224 dA ← (4 c̲at r c̲at 9) r̲eshape r̲avel dAs
225 dB ← (4 3 c̲at r) r̲eshape r̲avel dBs
226 kind = 2 ? (dA, k × dB)
227 kind = 3 ? (0.0 × dA, k × dB)
228 ('+ r̲/₃ dA × (s̲hape dA) r̲eshape A0, ('+ r̲/₃ dB × (s̲hape dB) r̲eshape B0) ÷ s)
229}
230⍝│ (kind, r) s̲tartC seed: where training starts: each A (or A0's
231⍝│ stack) from the seed, as the methods above start; B at 0; VeRA's d
232⍝│ at 0.1 and b at 0.
233ᵘs̲tartC ← { (kind, r) seed →
234 kind = 1 ? (0.3 × seed ˡᵒs̲eeded r c̲at 9, (8 c̲at r) r̲eshape 0.0)
235 kind < 4 ? ((4 c̲at r c̲at 9) r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded (4 × r) c̲at 9, (4 3 c̲at r) r̲eshape 0.0)
236 ((4 c̲at r) r̲eshape 0.1, 4 3 r̲eshape 0.0)
237}
238⍝│ ((ctx, lr), (xs, Ys)) s̲tepC state: Adam's step on P and Q at the
239⍝│ rate lr.
240ᵘs̲tepC ← { ((ctx, lr), (xs, Ys)) ((P, Q), (mP, vP, mQ, vQ), t, losses) →
241 (daS, dbS, _, _, dAs, dBs, loss) ← (ctx ᵘe̲ffective (P, Q)) ᵘg̲rads6 (xs, Ys)
242 (dP, dQ) ← ctx ᵘc̲hain (daS, dbS, dAs, dBs)
243 (P2, mP2, vP2) ← (lr, t) ᵒᵖa̲dam (P, mP, vP, dP)
244 (Q2, mQ2, vQ2) ← (lr, t) ᵒᵖa̲dam (Q, mQ, vQ, dQ)
245 ((P2, Q2), (mP2, vP2, mQ2, vQ2), t + 1, losses c̲at loss)
246}
247⍝│ (kind, r, tokens, steps, lr) c̲ompare seed: an adapter of the
248⍝│ kind at rank r fine-tuned on the tokens at the rate lr, VeRA's
249⍝│ projections from the seed: the model, the loss after each step, the
250⍝│ numbers it trains.
251ᵘc̲ompare ← { (kind, r, tokens, steps, lr) seed →
252 (A0, B0) ← (seed + 1) ˡᵒp̲rojections 3 9 c̲at r
253 (P, Q) ← (kind, r) ᵘs̲tartC seed
254 ((P2, Q2), _, _, losses) ← steps '{ st → (((kind, r, A0, B0), lr), tokens) ᵘs̲tepC st } p̲ower ((P, Q), (0.0 × P, 0.0 × P, 0.0 × Q, 0.0 × Q), 1, 0 r̲eshape 0.0)
255 ((kind, r, A0, B0) ᵘe̲ffective (P2, Q2), losses, kind s̲elect (17 × r) c̲at (48 × r) c̲at (12 × r) c̲at 4 × 3 + r)
256}
257⍝│ Equal ranks, 2 each; then equal budgets, about 48 numbers each:
258⍝│ shared LoRA at rank 3 (51), expert LoRA at 1 (48), LoRA-FA at 4
259⍝│ (48), VeRA at 9 (48). Each at the methods' rate, 0.02, and at 0.1:
260⍝│ an adapter whose projections are frozen moves slowly at LoRA's rate,
261⍝│ and VeRA's paper trains at a rate several times LoRA's.
262equalRanks ← 2 2 2 2
263equalBudgets ← 3 1 4 9
264
265⍝━━ Deploying
266
267⍝│ The research's deployment (L381-460): an adapter kept apart from the
268⍝│ base (switched per task) or merged into it (one task, as fast as the
269⍝│ base). A second task, for switching: kind 1 by a new rule (200
270⍝│ tokens to fine-tune on, 200 held out); the kinds both tasks leave
271⍝│ alone are 2 and 4 (held out, 200).
272((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt"
273((xn2, Yn2), _) ← ᵘr̲ead "data/shift2-test.txt"
274(xe, Ye) ← ((w̲here 0 = ka m̲od 2) ᵘc̲olumns xa, (w̲here 0 = ka m̲od 2) ᵘc̲olumns Ya)
275⍝│ d̲eltas (As, Bs): each expert's update s B A (LoRA's d_elta),
276⍝│ stacked as the experts are (5 x 3 x 9).
277ᵘd̲eltas ← { (As, Bs) →
278 (D, _) ← 5 '{ (D, e) → (D c̲at r̲avel ˡᵒd̲elta (e s̲elect As, e s̲elect Bs, s), e + 1) } p̲ower (0 r̲eshape 0.0, 1)
279 5 3 9 r̲eshape D
280}
281⍝│ Wm m̲erged m: the model m with its experts' adapters folded into
282⍝│ the weights Wm (the adapters then 0): for LoRA, Ws + s B A; for
283⍝│ QLoRA, that quantized again.
284ᵘm̲erged ← { Wm (aS, bS, R, Ws, As, Bs) → (aS, bS, R, Wm, 0.0 × As, 0.0 × Bs) }
285⍝│ QLoRA's base: the experts in NF4, blocks of 9 (a row each), and what
286⍝│ it computes with, W~, dequantized.
287Wq ← ˡᵒd̲equant 9 ˡᵒq̲base W0
288⍝│ m a̲gainst m2: two models on every held-out token (both tasks' and
289⍝│ the old task's, 800): the largest difference of their probabilities,
290⍝│ and the tokens whose likeliest class differs.
291xh ← o̲\ (o̲\ xn) c̲at (o̲\ xn2) c̲at o̲\ xa
292ᵘa̲gainst ← { m m2 →
293 (_, _, P) ← m ᵘl̲ayers xh
294 (_, _, P2) ← m2 ᵘl̲ayers xh
295 ('m̲ax r̲/ r̲avel a̲bs P − P2) c̲at 0.5 × '+ r̲/ r̲avel a̲bs (1 ᵐᵒᵉm̲ask P) − 1 ᵐᵒᵉm̲ask P2
296}
297⍝│ m t̲asks: a model on the first task (kind 3's new rule), the
298⍝│ second (kind 1's) and the kinds neither changed.
299ᵘt̲asks ← { m → (m ᵘa̲ccuracy (xn, Yn)) c̲at (m ᵘa̲ccuracy (xn2, Yn2)) c̲at m ᵘa̲ccuracy (xe, Ye) }
300⍝│ The cache: the held-out tokens of the two tasks in batches of 8,
301⍝│ one task's batch then the other's (50 batches), each batch's
302⍝│ experts as the base's router sends them (fine-tuning the experts
303⍝│ leaves the routing as it was). Kept apart, the base experts are one
304⍝│ cache's keys (4) and the adapters another's, one per task and expert
305⍝│ (8); merged, each task has its own copy of every expert (8 keys).
306(_, G1, _) ← base ᵘl̲ayers xn
307(_, G2, _) ← base ᵘl̲ayers xn2
308U1 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G1
309U2 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G2
310Ub ← o̲\ 50 4 r̲eshape r̲avel o̲\ U1 c̲at U2
311Ua ← o̲\ 50 8 r̲eshape r̲avel o̲\ U1 c̲at (0.0 × U1) c̲at (0.0 × U2) c̲at U2
312⍝│ sizes t̲raffic c: with c slots in each cache, the loads (base
313⍝│ experts, adapters or merged copies) and what moved: merged copies,
314⍝│ then the base experts and LoRA, LoRA-FA and VeRA adapters kept apart,
315⍝│ in bytes, sizes the bytes of (an expert, a LoRA adapter, LoRA-FA's B,
316⍝│ VeRA's d and b), the frozen A and projections made from their seeds.
317ᵘt̲raffic ← { (ne, nl, nf, nv) c →
318 lb ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ub
319 la ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ua
320 (f̲loat c) c̲at lb c̲at la c̲at (la × ne) c̲at (lb × ne) + la × nl c̲at nf c̲at nv
321}
322⍝━━ Preference: supervised fine-tuning, then DPO
323
324⍝│ The research's order (L1475-1490, L2023-2046): supervised
325⍝│ fine-tuning (SFT) first, then preference pairs (DPO), both answers
326⍝│ of a pair valid, so the model learns a preference and not right from
327⍝│ wrong; reinforcement learning waits until those show a gain. Here a
328⍝│ new-task token's two best classes by its rule both count as right
329⍝│ (like two solutions that both pass the tests), and the better of the
330⍝│ two, its class, is preferred (the cleaner one). SFT on demonstrations
331⍝│ of either can learn what is right but not which is better.
332⍝│ c o̲nehot: the one-hot columns of the classes c.
333ᵘo̲nehot ← { c → f̲loat (r̲ange 3) '= t̲able f̲loor c }
334⍝│ v̲alid (Y, c2): (valid, preferred, rejected), one-hot: the two best
335⍝│ classes, the best (Y itself), the second best (c2, from its file).
336ᵘv̲alid ← { (Yc, c2) → (Yc + ᵘo̲nehot c2, Yc, ᵘo̲nehot c2) }
337(V, Yw, Yl) ← ᵘv̲alid (Y, n̲umbers ⎕N̲GET "data/shift-second-train.txt")
338(Vn, Ywn, _) ← ᵘv̲alid (Yn, n̲umbers ⎕N̲GET "data/shift-second-test.txt")
339⍝│ m p̲refers x: how often a model's answer is one of the two valid
340⍝│ classes, and how often it is the preferred one, on the held-out new
341⍝│ tokens; then the old task's accuracy.
342ᵘp̲refers ← { m →
343 (_, _, P) ← m ᵘl̲ayers xn
344 A ← 1 ᵐᵒᵉm̲ask P
345 N ← f̲loat 2 s̲elect s̲hape xn
346 (('+ r̲/ r̲avel A × Vn) ÷ N) c̲at (('+ r̲/ r̲avel A × Ywn) ÷ N) c̲at m ᵘa̲ccuracy (xo, Yo)
347}
348⍝│ m g̲radsD ((xs, Yw, Yl), Pr, b): DPO's loss (Rafailov et al., 2023)
349⍝│ and gradients on the pairs: Yw the chosen answer of each token, Yl
350⍝│ the rejected, Pr the reference model's probabilities, b its beta. With
351⍝│ h = b ((log P - log Pr) . (Yw - Yl)) for a token, the loss is -log
352⍝│ sigma(h), and its gradient at the softmax's input is -b sigma(-h) (Yw
353⍝│ - Yl): the softmax's own part cancels between the two answers.
354ᵘg̲radsD ← { m ((xs, Wy, Ly), Pr, b) →
355 (z0, z, P) ← m ᵘf̲orward6 xs
356 d ← Wy − Ly
357 h ← b × '+ r̲/₁ d × (l̲og P) − l̲og Pr
358 N ← f̲loat 2 s̲elect s̲hape xs
359 q ← (s̲hape d) r̲eshape 1.0 ÷ 1.0 + e̲xp h
360 (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (n̲eg b × q × d) ÷ N)
361 (daS, dbS, dR, dW, dAs, dBs, ('+ r̲/ l̲og 1.0 + e̲xp n̲eg h) ÷ N)
362}
363⍝│ (method, pairs) s̲tepD state: a step of DPO on the pairs.
364ᵘs̲tepD ← { (method, pairs) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲radsD pairs) }
365⍝│ (m0, method, (xs, Yw, Yl), b) d̲po steps: DPO from the model m0,
366⍝│ which is also the reference, Adam starting afresh: the model and the
367⍝│ loss after each step.
368ᵘd̲po ← { (m0, method, (xs, Yw2, Yl2), b) steps →
369 (_, _, Pr) ← m0 ᵘf̲orward6 xs
370 (aS, bS, R, Ws, As, Bs) ← m0
371 (m, _, _, losses) ← steps '{ st → (method, ((xs, Yw2, Yl2), Pr, b)) ᵘs̲tepD st } p̲ower (m0, (0.0 × aS, 0.0 × aS, 0.0 × bS, 0.0 × bS, 0.0 × R, 0.0 × R, 0.0 × Ws, 0.0 × Ws, 0.0 × As, 0.0 × As, 0.0 × Bs, 0.0 × Bs), 1, 0 r̲eshape 0.0)
372 (m, losses)
373}
374⍝│ The pairs: the house style's, both valid (preferred, rejected); and,
375⍝│ to show what the research warns of, each valid answer against the
376⍝│ invalid one (two pairs a token), which teaches right from wrong only.
377styled ← (x, Yw, Yl)
378wrong ← 1.0 − V
379checked ← (o̲\ (o̲\ x) c̲at o̲\ x, o̲\ (o̲\ Yw) c̲at o̲\ Yl, o̲\ (o̲\ wrong) c̲at o̲\ wrong)
380⍝ -- end of the core ------------------------------
381
382⍝━━ Each method, seed 1, 120 steps
383
384⍝│ A line per method: the new task's accuracy, the old task's, the
385⍝│ numbers trained, their bytes, the new tokens' share of each of the 5
386⍝│ experts, the old tokens', then the update sizes (the 4 experts, the
387⍝│ shared layer, the new expert).
388ᵘr̲ow ← { (method, tokens) →
389 (m, _) ← (method, tokens, 120) ᵘt̲une 1
390 (a, o, n, b, sn, so, sz) ← m ᵘr̲eport method
391 ᵘr̲ound a c̲at o c̲at n c̲at b c̲at sn c̲at so c̲at sz
392}
393ᵘa̲ll ← { tokens → 7 20 r̲eshape (ᵘr̲ow (none, tokens)) c̲at (ᵘr̲ow (shared, tokens)) c̲at (ᵘr̲ow (experts, tokens)) c̲at (ᵘr̲ow (both, tokens)) c̲at (ᵘr̲ow (selected, tokens)) c̲at (ᵘr̲ow (router, tokens)) c̲at ᵘr̲ow (added, tokens) }
394⍝│ Fine-tuned on the new tokens alone, then mixed with old ones.
395ᵘa̲ll (x, Y)
396ᵘa̲ll mixed
397
398⍝━━ The four adapters, seed 1, 120 steps, on the new tokens alone
399
400⍝│ A line per adapter (shared LoRA, expert LoRA, LoRA-FA, shared VeRA):
401⍝│ the new task's accuracy, the old task's, the numbers trained; at
402⍝│ equal ranks, then at equal budgets; at the rate 0.02, then 0.1.
403ᵘl̲ine ← { (kind, r, lr) →
404 (m, _, n) ← (kind, r, (x, Y), 120, lr) ᵘc̲ompare 1
405 (ᵘr̲ound m ᵘa̲ccuracy (xn, Yn)) c̲at (ᵘr̲ound m ᵘa̲ccuracy (xo, Yo)) c̲at f̲loat n
406}
407ᵘf̲our ← { (ranks, lr) → 4 3 r̲eshape (ᵘl̲ine (1, 1 s̲elect ranks, lr)) c̲at (ᵘl̲ine (2, 2 s̲elect ranks, lr)) c̲at (ᵘl̲ine (3, 3 s̲elect ranks, lr)) c̲at ᵘl̲ine (4, 4 s̲elect ranks, lr) }
408ᵘf̲our (equalRanks, 0.02)
409ᵘf̲our (equalBudgets, 0.02)
410ᵘf̲our (equalRanks, 0.1)
411ᵘf̲our (equalBudgets, 0.1)
412
413⍝━━ Deploying, seed 1, 120 steps
414
415⍝│ Kept apart against merged, the new task's adapters on every expert
416⍝│ (the router frozen): LoRA on the base, then QLoRA on its 4-bit base
417⍝│ (trained on W~, merged by quantizing W~ + s B A again). A line each:
418⍝│ the largest difference of the probabilities, the tokens whose answer
419⍝│ changed (of 800), the new task's accuracy kept apart, then merged.
420(m1, _) ← (experts, (x, Y), 120) ᵘt̲une 1
421(aS1, bS1, R1, W1, A1, B1) ← m1
422⍝ -- end of the first task's adapters
423(mq, _) ← (Wq, experts, (x, Y), 120) ᵘt̲uneOn 1
424(_, _, _, _, Aq, Bq) ← mq
425ml ← (W1 + ᵘd̲eltas (A1, B1)) ᵘm̲erged m1
426mm ← (ˡᵒd̲equant 9 ˡᵒq̲base Wq + ᵘd̲eltas (Aq, Bq)) ᵘm̲erged mq
427(m1 ᵘa̲gainst ml) c̲at (m1 ᵘa̲ccuracy (xn, Yn)) c̲at ml ᵘa̲ccuracy (xn, Yn)
428(mq ᵘa̲gainst mm) c̲at (mq ᵘa̲ccuracy (xn, Yn)) c̲at mm ᵘa̲ccuracy (xn, Yn)
429⍝ -- end of the merge
430⍝│ Two tasks' adapters on one base: the base, then with the first
431⍝│ task's adapters, then with the second's (switched: the same base,
432⍝│ the other adapters), each on the first task, the second and the
433⍝│ kinds neither changed.
434(m2, _) ← (experts, (x2, Y2), 120) ᵘt̲une 1
435(_, _, _, _, A2, B2) ← m2
4363 3 r̲eshape (ᵘr̲ound ᵘt̲asks base) c̲at (ᵘr̲ound ᵘt̲asks m1) c̲at ᵘr̲ound ᵘt̲asks (aS1, bS1, R1, W1, A2, B2)
437⍝ The bytes each keeps, FP32: the base (200 numbers), a task's adapters
438⍝ (96), a merged copy (the base again); for the two tasks kept apart,
439⍝ then merged.
440800 384 800 c̲at (800 + 2 × 384) c̲at 2 × 800
441⍝ -- end of the switch
442⍝ The cache, 1 to 4 slots in each: slots, base-expert loads, adapter
443⍝ (or merged-copy) loads, then the bytes moved: merged copies; kept
444⍝ apart with LoRA, LoRA-FA and VeRA adapters (rank 2: an expert 27
445⍝ numbers, LoRA 24, LoRA-FA's B 6, VeRA 5).
4464 7 r̲eshape ((108, 96, 24, 20) ᵘt̲raffic 1) c̲at ((108, 96, 24, 20) ᵘt̲raffic 2) c̲at ((108, 96, 24, 20) ᵘt̲raffic 3) c̲at (108, 96, 24, 20) ᵘt̲raffic 4
447⍝│ At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16
448⍝│ in BF16 (the LoRA library's counts): the expert, its LoRA adapters,
449⍝│ LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.
450(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)
451be c̲at bl c̲at bf c̲at bv
452(be, bl, bf, bv) ᵘt̲raffic 2
453⍝ -- end of the cache
454
455⍝━━ Preference, seed 1, 120 steps each
456
457⍝ A line each: how often the answer on the held-out new tokens is
458⍝ valid (one of the two best), how often it is the preferred (the
459⍝ best), and the old task's accuracy. The base; SFT on the preferred
460⍝ answers alone (the method "every expert"); SFT on demonstrations of
461⍝ either valid answer; that, then DPO on the pairs (the better against
462⍝ the other valid), beta 1 and 5; then DPO on each valid answer against
463⍝ the invalid one, beta 1.
464(mp, _) ← (experts, (x, Y), 120) ᵘt̲une 1
465(ms, _) ← (experts, (x, 0.5 × V), 120) ᵘt̲une 1
466(md1, _) ← (ms, experts, styled, 1.0) ᵘd̲po 120
467(md5, _) ← (ms, experts, styled, 5.0) ᵘd̲po 120
468(mc, _) ← (ms, experts, checked, 1.0) ᵘd̲po 120
4696 3 r̲eshape (ᵘr̲ound ᵘp̲refers base) c̲at (ᵘr̲ound ᵘp̲refers mp) c̲at (ᵘr̲ound ᵘp̲refers ms) c̲at (ᵘr̲ound ᵘp̲refers md1) c̲at (ᵘr̲ound ᵘp̲refers md5) c̲at ᵘr̲ound ᵘp̲refers mc