programdemos/moe-finetune/moe-finetune.xtl

Fine-tuning a tiny mixture-of-experts, each method in turn. The base (base.xtl, trained by train/base.xtl) classifies tokens of 4 kinds, each by its kind's own rule; the new task changes kind 3's rule. A method fine-tunes the base on 200 tokens of the new task (alone, or mixed with 200 of the old task's: replay), the base's own weights frozen, with LoRA adapters (rank 2) on some of its parts: the shared layer only; every expert, the router frozen; both; the two experts the new tokens use most; every expert with the router trained too; or a new expert, added with its own router row, trained alone. Each is measured by the new task's accuracy, the old task's (the kinds whose rule did not change: what fine-tuning must not break, 3 points the bar), the numbers it trains and their bytes, where the tokens are routed before and after, and how large each expert's update is. Then four adapters compared (shared LoRA, expert LoRA, LoRA-FA, shared VeRA), and the adapters deployed: kept apart against merged (LoRA exactly, QLoRA not), two tasks' adapters switched on one base, and an expert cache measuring what each way moves. Last, preference tuning: SFT on either of two valid answers, then DPO on pairs of them, the better preferred.

source · imports moe: libs/MoE/src/MoE.xtl; lo: libs/LoRA/src/LoRA.xtl libs/LoRA/src/LoRA.xtlm; op: libs/Optim/src/Optim.xtl

The tokens

ᵘr̲ead : Char -> ((Float, Float), Int)

function · line 30

r̲ead file: (x, the classes one-hot, the kinds) of a data file: the inputs (9 x N, the 9th a constant 1), the kinds, the classes.

ᵘr̲ead ← { file →
  v ← n̲umbers ⎕N̲GET file
  N ← (t̲ally v) d̲iv 11
  (((9 c̲at N) r̲eshape (9 × N) t̲ake v, f̲loat (r̲ange 3) '= t̲able f̲loor (n̲eg N) t̲ake v), f̲loor N t̲ake (9 × N) d̲rop v)
}
Used in: x, xn, xa, xb, x2, xn2

ᵘc̲olumns : Int -> Float -> Float

function · line 36

idx c̲olumns M: the columns (tokens) idx of M.

ᵘc̲olumns ← { idx M → M '+ '× i̲nner f̲loat (r̲ange 2 s̲elect s̲hape M) '= t̲able idx }
Used in: xo, xr, xe

x : Float

value · line 39

The new task: 200 tokens of kind 3 by its new rule to fine-tune on, 200 held out.

((x, Y), _) ← ᵘr̲ead "data/shift-train.txt"

Y : Float

value · line 39

The new task: 200 tokens of kind 3 by its new rule to fine-tune on, 200 held out.

((x, Y), _) ← ᵘr̲ead "data/shift-train.txt"

xn : Float

value · line 40
((xn, Yn), _) ← ᵘr̲ead "data/shift-test.txt"

Yn : Float

value · line 40
((xn, Yn), _) ← ᵘr̲ead "data/shift-test.txt"

xa : Float

value · line 43

The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).

((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
Used in: xa, xo, xe, xh

Ya : Float

value · line 43

The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).

((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
Used in: xa, xo, xe

ka : Int

value · line 43

The old task: the held-out tokens of kinds 1, 2 and 4, whose rule the new task keeps (300).

((xa, Ya), ka) ← ᵘr̲ead "data/test.txt"
Used in: xa, xo, xe

xo : Float

value · line 44
(xo, Yo) ← ((w̲here ka ≠ 3) ᵘc̲olumns xa, (w̲here ka ≠ 3) ᵘc̲olumns Ya)

Yo : Float

value · line 44
(xo, Yo) ← ((w̲here ka ≠ 3) ᵘc̲olumns xa, (w̲here ka ≠ 3) ᵘc̲olumns Ya)

xb : Float

value · line 47

Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.

((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
Used in: xb, xr

Yb : Float

value · line 47

Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.

((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
Used in: xb, xr

kb : Int

value · line 47

Replay: the new tokens mixed with 200 of the old task's training tokens (kinds 1, 2 and 4), the usual guard against forgetting.

((xb, Yb), kb) ← ᵘr̲ead "data/train.txt"
Used in: xb, old

old : Int

value · line 48
old ← 200 t̲ake w̲here kb ≠ 3
Used in: xr

xr : Float

value · line 49
(xr, Yr) ← (old ᵘc̲olumns xb, old ᵘc̲olumns Yb)
Used in: xr, mixed, zo

Yr : Float

value · line 49
(xr, Yr) ← (old ᵘc̲olumns xb, old ᵘc̲olumns Yb)
Used in: xr, mixed

mixed : (Float, Float)

value · line 50
mixed ← (o̲\ (o̲\ x) c̲at o̲\ xr, o̲\ (o̲\ Y) c̲at o̲\ Yr)

The base, with room for a new expert

w : Float

value · line 60

The base: the shared layer S (8 x 9), the router (4 x 8), the experts (4 x 3 x 8). The expert layer's input here is z with a constant 1 under it (9 x N), so a router row and an expert can have a bias; the base's are 0. A fifth expert waits, 0, its router row a bias of -30 (never in a token's top 2: the gates of the chosen two are as the base's).

w ← n̲umbers ⎕N̲GET "data/base.txt"
Used in: S, R0, W0

S : Float

value · line 61
S ← 8 9 r̲eshape 72 t̲ake w

R0 : Float

value · line 62
R0 ← (o̲\ (o̲\ 4 8 r̲eshape 32 t̲ake 72 d̲rop w) c̲at 1 4 r̲eshape 0.0) c̲at 1 9 r̲eshape (8 r̲eshape 0.0) c̲at -30.0

W0 : Float

value · line 63
W0 ← 5 3 9 r̲eshape r̲avel (o̲\ (o̲\ 12 8 r̲eshape 104 d̲rop w) c̲at 1 12 r̲eshape 0.0) c̲at 3 9 r̲eshape 0.0

s : Float

value · line 67

The adapters: (aS, bS) on the shared layer (2 x 9, 8 x 2), (As, Bs) on the experts (5 x 2 x 9, 5 x 3 x 2); s = alpha / r, alpha 4 at rank 2.

s ← 4.0 ˡᵒs̲caling 2

ᵘo̲ne : Float -> Float

function · line 68
ᵘo̲ne ← { M → M c̲at (1 c̲at 2 s̲elect s̲hape M) r̲eshape 1.0 }

ᵘl̲ayers : (Float, Float, Float, Float, Float, Float) -> Float -> (Float, Float, Float)

function · line 72

(aS, bS, R, Ws, As, Bs) l̲ayers x: (z, G, P): the shared layer's output, the gates over the 5 experts, each token's probabilities over the classes.

ᵘl̲ayers ← { (aS, bS, R, Ws, As, Bs) x →
  z ← ᵘo̲ne t̲anh (S, aS, bS, s) ˡᵒf̲orward x
  G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z
  (z, G, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉc̲ombine z)
}

ᵘa̲ccuracy : (Float, Float, Float, Float, Float, Float) -> (Float, Float) -> Float

function · line 79

m a̲ccuracy (x, Y): the share of tokens whose likeliest class is theirs.

ᵘa̲ccuracy ← { m (x, Y) →
  (_, _, P) ← m ᵘl̲ayers x
  ('+ r̲/ r̲avel Y × 1 ᵐᵒᵉm̲ask P) ÷ f̲loat 2 s̲elect s̲hape Y
}

ᵘs̲hare : (Float, Float, Float, Float, Float, Float) -> Float -> Float

function · line 84

m s̲hare x: the share of the tokens x each expert gets (summing to 2).

ᵘs̲hare ← { m x →
  (_, G, _) ← m ᵘl̲ayers x
  ᵐᵒᵉf̲requency G
}

ᵘp̲er : Num a => a -> a -> a

function · line 89

f p̲er A: each expert's block of A (stacked) times its flag in f.

ᵘp̲er ← { f A → A × (s̲hape A) r̲eshape r̲avel f 'l̲eft t̲able o̲ffsets (t̲ally r̲avel A) d̲iv t̲ally f }
Used in: ᵘa̲dam6

base : (Float, Float, Float, Float, Float, Float)

value · line 90
base ← (2 9 r̲eshape 0.0, 8 2 r̲eshape 0.0, R0, W0, 5 2 9 r̲eshape 0.0, 5 3 2 r̲eshape 0.0)

The new expert's router row

ᵘm̲ean : Float -> Float

function · line 97

It starts pointing at the new tokens and away from the old: the direction from the old training tokens' mean z to the new ones', 6 long, its bias putting 0 halfway between the two means.

ᵘm̲ean ← { M → ('+ r̲/₂ M) ÷ f̲loat 2 s̲elect s̲hape M }
Used in: d, R5

zn : Float

value · line 98
(zn, _, _) ← base ᵘl̲ayers x
Used in: zn, d, R5

zo : Float

value · line 99
(zo, _, _) ← base ᵘl̲ayers xr
Used in: zo, d, R5

d : Float

value · line 100
d ← (8 t̲ake ᵘm̲ean zn) − 8 t̲ake ᵘm̲ean zo
Used in: r5

r5 : Float

value · line 101
r5 ← 6.0 × d ÷ s̲qrt '+ r̲/ d × d
Used in: R5

R5 : Float

value · line 102
R5 ← 5 9 r̲eshape (36 t̲ake r̲avel R0) c̲at r5 c̲at n̲eg '+ r̲/ r5 × 0.5 × 8 t̲ake (ᵘm̲ean zn) + ᵘm̲ean zo
Used in: ᵘt̲uneOn

A method

ᵘf̲orward6 : (Float, Float, Float, Float, Float, Float) -> Float -> (Float, Float, Float)

function · line 117

A method is which parts train, (shared, experts, router, new): 1 or 0 for the shared layer's adapter, the base router's rows and the new expert (its weights and router row), a flag per base expert's adapter. The gradients of the cross-entropy go back through the experts and the router (MoE's g_rads and g_radsW, each token's two experts held fixed) and the shared layer (LoRA's g_rads, after tanh); every expert runs on every token here (d_ense, the same sum as the sparse c_ombine, faster at this size). A frozen part's gradient is 0, so Adam leaves it where it is. m f̲orward6 xs: the model on the tokens xs, (z0, z, P): the shared layer's output, with its constant 1, each token's probabilities.

ᵘf̲orward6 ← { (aS, bS, R, Ws, As, Bs) xs →
  z0 ← t̲anh (S, aS, bS, s) ˡᵒf̲orward xs
  z ← ᵘo̲ne z0
  G ← 2 ᵐᵒᵉt̲opk ᵐᵒᵉs̲oftmax R ᵐᵒᵉs̲cores z
  (z0, z, ᵐᵒᵉs̲oftmax ((Ws, As, Bs, s), G) ᵐᵒᵉd̲ense z)
}

ᵘb̲ack6 : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float)

function · line 125

m b̲ack6 (xs, z0, z, g): the gradients for every part, (daS, dbS, dR, dW, dAs, dBs), from g, the loss's gradient at the softmax's input.

ᵘb̲ack6 ← { (aS, bS, R, Ws, As, Bs) (xs, z0, z, g) →
  (dR, dAs, dBs, dz) ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲rads (z, g)
  dW ← (R, (Ws, As, Bs, s), 2) ᵐᵒᵉg̲radsW (z, g)
  (daS, dbS, _) ← (S, aS, bS, s) ˡᵒg̲rads (xs, (8 t̲ake dz) × 1.0 − z0 × z0)
  (daS, dbS, dR, dW, dAs, dBs)
}

ᵘg̲rads6 : (Float, Float, Float, Float, Float, Float) -> (Float, Float) -> (Float, Float, Float, Float, Float, Float, Float)

function · line 133

m g̲rads6 (xs, Ys): the model's loss on the tokens (xs, Ys) and its gradients for every part, (daS, dbS, dR, dW, dAs, dBs, loss).

ᵘg̲rads6 ← { m (xs, Ys) →
  (z0, z, P) ← m ᵘf̲orward6 xs
  (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (P − Ys) ÷ f̲loat 2 s̲elect s̲hape xs)
  (daS, dbS, dR, dW, dAs, dBs, n̲eg ('+ r̲/ r̲avel Ys × l̲og P) ÷ f̲loat 2 s̲elect s̲hape xs)
}

ᵘa̲dam6 : (Float, Float, Float, Float) -> (((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, a), (Float, Float, Float, Float, Float, Float, a)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, a)

function · line 141

method a̲dam6 (state, grads): Adam's step on the parts the method trains. The state: the model, Adam's averages, the step, the loss after each step; the gradients and the loss, as g_rads6 gives them.

ᵘa̲dam6 ← { (fS, fE, fR, fN) (((aS, bS, R, Ws, As, Bs), (m1, v1, m2, v2, m3, v3, m4, v4, m5, v5, m6, v6), t, losses), (daS, dbS, dR, dW, dAs, dBs, loss)) →
  (aS2, m12, v12) ← (0.02, t) ᵒᵖa̲dam (aS, m1, v1, fS × daS)
  (bS2, m22, v22) ← (0.02, t) ᵒᵖa̲dam (bS, m2, v2, fS × dbS)
  (R2, m32, v32) ← (0.02, t) ᵒᵖa̲dam (R, m3, v3, ((4 r̲eshape fR) c̲at fN) ᵘp̲er dR)
  (W2, m42, v42) ← (0.02, t) ᵒᵖa̲dam (Ws, m4, v4, (0.0 0.0 0.0 0.0 c̲at fN) ᵘp̲er dW)
  (As2, m52, v52) ← (0.02, t) ᵒᵖa̲dam (As, m5, v5, (fE c̲at 0.0) ᵘp̲er dAs)
  (Bs2, m62, v62) ← (0.02, t) ᵒᵖa̲dam (Bs, m6, v6, (fE c̲at 0.0) ᵘp̲er dBs)
  ((aS2, bS2, R2, W2, As2, Bs2), (m12, v12, m22, v22, m32, v32, m42, v42, m52, v52, m62, v62), t + 1, losses c̲at loss)
}

ᵘs̲tep : ((Float, Float, Float, Float), (Float, Float)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float)

function · line 152

(method, (xs, Ys)) s̲tep state: a step on the cross-entropy of the tokens (xs, Ys).

ᵘs̲tep ← { (method, tokens) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲rads6 tokens) }
Used in: ᵘt̲uneOn

ᵘt̲uneOn : Num a => (Float, (Float, Float, Float, Float), (Float, Float), Int) -> a -> ((Float, Float, Float, Float, Float, Float), Float)

function · line 157

(Wb, method, tokens, steps) t̲uneOn seed: the base, its experts Wb, fine-tuned by a method on the tokens: each A from the seed (small), each B 0, so it starts as the base; with a new expert, its router row as above.

ᵘt̲uneOn ← { (Wb, (fS, fE, fR, fN), tokens, steps) seed →
  aS ← 0.3 × seed ˡᵒs̲eeded 2 9
  As ← 5 2 9 r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded 10 9
  R ← R0 + fN × R5 − R0
  zero ← (0.0 × aS, 8 2 r̲eshape 0.0, 0.0 × R0, 0.0 × Wb, 0.0 × As, 5 3 2 r̲eshape 0.0)
  (zS, zB, zR, zW, zA, zE) ← zero
  (m, _, _, losses) ← steps '{ st → ((fS, fE, fR, fN), tokens) ᵘs̲tep st } p̲ower ((aS, zB, R, Wb, As, zE), (zS, zS, zB, zB, zR, zR, zW, zW, zA, zA, zE, zE), 1, 0 r̲eshape 0.0)
  (m, losses)
}
Used in: ᵘt̲une, mq

ᵘt̲une : Num a => ((Float, Float, Float, Float), (Float, Float), Int) -> a -> ((Float, Float, Float, Float, Float, Float), Float)

function · line 167

(method, tokens, steps) t̲une seed: the base itself fine-tuned.

ᵘt̲une ← { (method, tokens, steps) seed → (W0, method, tokens, steps) ᵘt̲uneOn seed }
Used in: ᵘr̲ow, m1, m2, mp, ms

ᵘs̲ize : Num a => (a, a) -> Float

function · line 169

s̲ize (A, B): an update's size, |s B A| (Frobenius).

ᵘs̲ize ← { (A, B) → s × s̲qrt '+ r̲/ r̲avel (B '+ '× i̲nner A) ^ 2 }
Used in: ᵘr̲eport

ᵘr̲eport : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float, Float)

function · line 175

m r̲eport method: a method's measures: the new task's accuracy and the old task's; the numbers trained and their bytes in FP32; the new and the old tokens' share of each of the 5 experts; the update's size on each base expert, the shared layer and the new expert (its weights, from 0).

ᵘr̲eport ← { m (fS, fE, fR, fN) →
  (aS, bS, R, Ws, As, Bs) ← m
  n ← (fS × 34.0) + (24.0 × '+ r̲/ fE) + (fR × 36.0) + fN × 36.0
  sizes ← '{ e → ᵘs̲ize (e s̲elect As, e s̲elect Bs) } e̲ach r̲ange 4
  new ← s̲qrt '+ r̲/ r̲avel (5 s̲elect Ws) ^ 2
  (m ᵘa̲ccuracy (xn, Yn), m ᵘa̲ccuracy (xo, Yo), n, 4.0 × n, m ᵘs̲hare xn, m ᵘs̲hare xo, sizes c̲at (ᵘs̲ize (aS, bS)) c̲at new)
}
Used in: ᵘr̲ow

picked : Float

value · line 184

The methods, in the research's order, then a new expert; "selected" takes the two experts the base sends most of the new tokens to.

picked ← 4 t̲ake r̲avel 2 ᵐᵒᵉm̲ask 5 1 r̲eshape base ᵘs̲hare x
Used in: selected

none : (Float, Float, Float, Float)

value · line 185
none ← (0.0, 4 r̲eshape 0.0, 0.0, 0.0)
Used in: ᵘa̲ll

shared : (Float, Float, Float, Float)

value · line 186
shared ← (1.0, 4 r̲eshape 0.0, 0.0, 0.0)
Used in: ᵘa̲ll

experts : (Float, Float, Float, Float)

value · line 187
experts ← (0.0, 4 r̲eshape 1.0, 0.0, 0.0)
Used in: ᵘa̲ll, m1, mq, m2, mp, ms, md1, md5, mc

both : (Float, Float, Float, Float)

value · line 188
both ← (1.0, 4 r̲eshape 1.0, 0.0, 0.0)
Used in: ᵘa̲ll

selected : (Float, Float, Float, Float)

value · line 189
selected ← (0.0, picked, 0.0, 0.0)
Used in: ᵘa̲ll

router : (Float, Float, Float, Float)

value · line 190
router ← (0.0, 4 r̲eshape 1.0, 1.0, 0.0)
Used in: ᵘa̲ll

added : (Float, Float, Float, Float)

value · line 191
added ← (0.0, 4 r̲eshape 0.0, 0.0, 1.0)
Used in: ᵘa̲ll

ᵘr̲ound : Float -> Float

function · line 192
ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + v × 1000.0) ÷ 1000.0 }

Four adapters compared

ᵘp̲ad : Float -> Float

function · line 208

The research's comparison (L4307-4430), the router frozen: shared LoRA (kind 1, on the shared layer: the research's attention LoRA), expert LoRA (2), LoRA-FA (3: each expert's A frozen) and shared VeRA (4: one pair of frozen projections for every expert, A0 (r x 9) and B0 (3 x r) from the seed, each expert its own d (r numbers) and b (3)), at any rank r, alpha 4. Each is the model above with its parts filled in: VeRA's expert e is LoRA's factors diag(d_e) A0 and diag(b_e) B0 (scale 1), which the MoE library runs as any adapter, and its gradients follow from theirs: dd_e is dA_e times A0 summed along each row, db_e is dB_e times B0 likewise. p̲ad M: the 4 experts' stack M with a fifth of zeros (the waiting expert).

ᵘp̲ad ← { M → M c̲at (1 c̲at 1 d̲rop s̲hape M) r̲eshape 0.0 }

ᵘe̲ffective : Num a => (a, Int, Float, Float) -> (Float, Float) -> (Float, Float, Float, Float, Float, Float)

function · line 211

(kind, r, A0, B0) e̲ffective (P, Q): the model of the adapter of a kind at rank r, P and Q its trained arrays (A and B, or d and b).

ᵘe̲ffective ← { (kind, r, A0, B0) (P, Q) →
  k ← (4.0 ˡᵒs̲caling r) ÷ s
  kind = 1 ? (P, k × Q, R0, W0, 5 1 9 r̲eshape 0.0, 5 3 1 r̲eshape 0.0)
  kind < 4 ? (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad P, ᵘp̲ad k × Q)
  dA ← (4 c̲at r c̲at 9) r̲eshape (r̲avel P) '× t̲able 9 r̲eshape 1.0
  dB ← (4 3 c̲at r) r̲eshape (r̲avel Q) '× t̲able r r̲eshape 1.0
  (1 9 r̲eshape 0.0, 8 1 r̲eshape 0.0, R0, W0, ᵘp̲ad dA × (4 c̲at r c̲at 9) r̲eshape A0, ᵘp̲ad (dB × (4 3 c̲at r) r̲eshape B0) ÷ s)
}

ᵘc̲hain : Num a => (a, Int, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float)

function · line 221

(kind, r, A0, B0) c̲hain (daS, dbS, dAs, dBs): the model's gradients back to the trained arrays', (dP, dQ); LoRA-FA's A gets 0.

ᵘc̲hain ← { (kind, r, A0, B0) (daS, dbS, dAs, dBs) →
  k ← (4.0 ˡᵒs̲caling r) ÷ s
  kind = 1 ? (daS, k × dbS)
  dA ← (4 c̲at r c̲at 9) r̲eshape r̲avel dAs
  dB ← (4 3 c̲at r) r̲eshape r̲avel dBs
  kind = 2 ? (dA, k × dB)
  kind = 3 ? (0.0 × dA, k × dB)
  ('+ r̲/₃ dA × (s̲hape dA) r̲eshape A0, ('+ r̲/₃ dB × (s̲hape dB) r̲eshape B0) ÷ s)
}
Used in: ᵘs̲tepC

ᵘs̲tartC : (Num a, Num b) => (a, Int) -> b -> (Float, Float)

function · line 233

(kind, r) s̲tartC seed: where training starts: each A (or A0's stack) from the seed, as the methods above start; B at 0; VeRA's d at 0.1 and b at 0.

ᵘs̲tartC ← { (kind, r) seed →
  kind = 1 ? (0.3 × seed ˡᵒs̲eeded r c̲at 9, (8 c̲at r) r̲eshape 0.0)
  kind < 4 ? ((4 c̲at r c̲at 9) r̲eshape 0.3 × (seed + 1) ˡᵒs̲eeded (4 × r) c̲at 9, (4 3 c̲at r) r̲eshape 0.0)
  ((4 c̲at r) r̲eshape 0.1, 4 3 r̲eshape 0.0)
}
Used in: ᵘc̲ompare

ᵘs̲tepC : Num a => (((a, Int, Float, Float), Float), (Float, Float)) -> ((Float, Float), (Float, Float, Float, Float), Float, Float) -> ((Float, Float), (Float, Float, Float, Float), Float, Float)

function · line 240

((ctx, lr), (xs, Ys)) s̲tepC state: Adam's step on P and Q at the rate lr.

ᵘs̲tepC ← { ((ctx, lr), (xs, Ys)) ((P, Q), (mP, vP, mQ, vQ), t, losses) →
  (daS, dbS, _, _, dAs, dBs, loss) ← (ctx ᵘe̲ffective (P, Q)) ᵘg̲rads6 (xs, Ys)
  (dP, dQ) ← ctx ᵘc̲hain (daS, dbS, dAs, dBs)
  (P2, mP2, vP2) ← (lr, t) ᵒᵖa̲dam (P, mP, vP, dP)
  (Q2, mQ2, vQ2) ← (lr, t) ᵒᵖa̲dam (Q, mQ, vQ, dQ)
  ((P2, Q2), (mP2, vP2, mQ2, vQ2), t + 1, losses c̲at loss)
}
Used in: ᵘc̲ompare

ᵘc̲ompare : Num a => (Int, Int, (Float, Float), Int, Float) -> a -> ((Float, Float, Float, Float, Float, Float), Float, Int)

function · line 251

(kind, r, tokens, steps, lr) c̲ompare seed: an adapter of the kind at rank r fine-tuned on the tokens at the rate lr, VeRA's projections from the seed: the model, the loss after each step, the numbers it trains.

ᵘc̲ompare ← { (kind, r, tokens, steps, lr) seed →
  (A0, B0) ← (seed + 1) ˡᵒp̲rojections 3 9 c̲at r
  (P, Q) ← (kind, r) ᵘs̲tartC seed
  ((P2, Q2), _, _, losses) ← steps '{ st → (((kind, r, A0, B0), lr), tokens) ᵘs̲tepC st } p̲ower ((P, Q), (0.0 × P, 0.0 × P, 0.0 × Q, 0.0 × Q), 1, 0 r̲eshape 0.0)
  ((kind, r, A0, B0) ᵘe̲ffective (P2, Q2), losses, kind s̲elect (17 × r) c̲at (48 × r) c̲at (12 × r) c̲at 4 × 3 + r)
}
Used in: ᵘl̲ine

equalRanks : Int

value · line 262

Equal ranks, 2 each; then equal budgets, about 48 numbers each: shared LoRA at rank 3 (51), expert LoRA at 1 (48), LoRA-FA at 4 (48), VeRA at 9 (48). Each at the methods' rate, 0.02, and at 0.1: an adapter whose projections are frozen moves slowly at LoRA's rate, and VeRA's paper trains at a rate several times LoRA's.

equalRanks ← 2 2 2 2

equalBudgets : Int

value · line 263
equalBudgets ← 3 1 4 9

Deploying

x2 : Float

value · line 272

The research's deployment (L381-460): an adapter kept apart from the base (switched per task) or merged into it (one task, as fast as the base). A second task, for switching: kind 1 by a new rule (200 tokens to fine-tune on, 200 held out); the kinds both tasks leave alone are 2 and 4 (held out, 200).

((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt"
Used in: x2, m2

Y2 : Float

value · line 272

The research's deployment (L381-460): an adapter kept apart from the base (switched per task) or merged into it (one task, as fast as the base). A second task, for switching: kind 1 by a new rule (200 tokens to fine-tune on, 200 held out); the kinds both tasks leave alone are 2 and 4 (held out, 200).

((x2, Y2), _) ← ᵘr̲ead "data/shift2-train.txt"
Used in: x2, m2

xn2 : Float

value · line 273
((xn2, Yn2), _) ← ᵘr̲ead "data/shift2-test.txt"
Used in: xn2, xh, ᵘt̲asks, G2

Yn2 : Float

value · line 273
((xn2, Yn2), _) ← ᵘr̲ead "data/shift2-test.txt"
Used in: xn2, ᵘt̲asks

xe : Float

value · line 274
(xe, Ye) ← ((w̲here 0 = ka m̲od 2) ᵘc̲olumns xa, (w̲here 0 = ka m̲od 2) ᵘc̲olumns Ya)
Used in: xe, ᵘt̲asks

Ye : Float

value · line 274
(xe, Ye) ← ((w̲here 0 = ka m̲od 2) ᵘc̲olumns xa, (w̲here 0 = ka m̲od 2) ᵘc̲olumns Ya)
Used in: xe, ᵘt̲asks

ᵘd̲eltas : (Float, Float) -> Float

function · line 277

d̲eltas (As, Bs): each expert's update s B A (LoRA's d_elta), stacked as the experts are (5 x 3 x 9).

ᵘd̲eltas ← { (As, Bs) →
  (D, _) ← 5 '{ (D, e) → (D c̲at r̲avel ˡᵒd̲elta (e s̲elect As, e s̲elect Bs, s), e + 1) } p̲ower (0 r̲eshape 0.0, 1)
  5 3 9 r̲eshape D
}
Used in: ml, mm

ᵘm̲erged : (Any a, Any b, Any c, Any d, Any e) => a -> (b, c, d, e, Float, Float) -> (b, c, d, a, Float, Float)

function · line 284

Wm m̲erged m: the model m with its experts' adapters folded into the weights Wm (the adapters then 0): for LoRA, Ws + s B A; for QLoRA, that quantized again.

ᵘm̲erged ← { Wm (aS, bS, R, Ws, As, Bs) → (aS, bS, R, Wm, 0.0 × As, 0.0 × Bs) }
Used in: ml, mm

Wq : Float

value · line 287

QLoRA's base: the experts in NF4, blocks of 9 (a row each), and what it computes with, W~, dequantized.

Wq ← ˡᵒd̲equant 9 ˡᵒq̲base W0
Used in: mq, mm

xh : Float

value · line 291

m a̲gainst m2: two models on every held-out token (both tasks' and the old task's, 800): the largest difference of their probabilities, and the tokens whose likeliest class differs.

xh ← o̲\ (o̲\ xn) c̲at (o̲\ xn2) c̲at o̲\ xa
Used in: ᵘa̲gainst

ᵘa̲gainst : (Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float) -> Float

function · line 292
ᵘa̲gainst ← { m m2 →
  (_, _, P) ← m ᵘl̲ayers xh
  (_, _, P2) ← m2 ᵘl̲ayers xh
  ('m̲ax r̲/ r̲avel a̲bs P − P2) c̲at 0.5 × '+ r̲/ r̲avel a̲bs (1 ᵐᵒᵉm̲ask P) − 1 ᵐᵒᵉm̲ask P2
}

ᵘt̲asks : (Float, Float, Float, Float, Float, Float) -> Float

function · line 299

m t̲asks: a model on the first task (kind 3's new rule), the second (kind 1's) and the kinds neither changed.

ᵘt̲asks ← { m → (m ᵘa̲ccuracy (xn, Yn)) c̲at (m ᵘa̲ccuracy (xn2, Yn2)) c̲at m ᵘa̲ccuracy (xe, Ye) }

G1 : Float

value · line 306

The cache: the held-out tokens of the two tasks in batches of 8, one task's batch then the other's (50 batches), each batch's experts as the base's router sends them (fine-tuning the experts leaves the routing as it was). Kept apart, the base experts are one cache's keys (4) and the adapters another's, one per task and expert (8); merged, each task has its own copy of every expert (8 keys).

(_, G1, _) ← base ᵘl̲ayers xn
Used in: G1, U1

G2 : Float

value · line 307
(_, G2, _) ← base ᵘl̲ayers xn2
Used in: G2, U2

U1 : Float

value · line 308
U1 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G1
Used in: Ub, Ua

U2 : Float

value · line 309
U2 ← 8 ᵐᵒᵉu̲sed 4 200 r̲eshape 800 t̲ake r̲avel G2
Used in: Ub, Ua

Ub : Float

value · line 310
Ub ← o̲\ 50 4 r̲eshape r̲avel o̲\ U1 c̲at U2
Used in: ᵘt̲raffic

Ua : Float

value · line 311
Ua ← o̲\ 50 8 r̲eshape r̲avel o̲\ U1 c̲at (0.0 × U1) c̲at (0.0 × U2) c̲at U2
Used in: ᵘt̲raffic

ᵘt̲raffic : (Float, Float, Float, Float) -> Int -> Float

function · line 317

sizes t̲raffic c: with c slots in each cache, the loads (base experts, adapters or merged copies) and what moved: merged copies, then the base experts and LoRA, LoRA-FA and VeRA adapters kept apart, in bytes, sizes the bytes of (an expert, a LoRA adapter, LoRA-FA's B, VeRA's d and b), the frozen A and projections made from their seeds.

ᵘt̲raffic ← { (ne, nl, nf, nv) c →
  lb ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ub
  la ← '+ r̲/ r̲avel c ᵐᵒᵉl̲ru Ua
  (f̲loat c) c̲at lb c̲at la c̲at (la × ne) c̲at (lb × ne) + la × nl c̲at nf c̲at nv
}

Preference: supervised fine-tuning, then DPO

ᵘo̲nehot : Num a => a -> Float

function · line 333

The research's order (L1475-1490, L2023-2046): supervised fine-tuning (SFT) first, then preference pairs (DPO), both answers of a pair valid, so the model learns a preference and not right from wrong; reinforcement learning waits until those show a gain. Here a new-task token's two best classes by its rule both count as right (like two solutions that both pass the tests), and the better of the two, its class, is preferred (the cleaner one). SFT on demonstrations of either can learn what is right but not which is better. c o̲nehot: the one-hot columns of the classes c.

ᵘo̲nehot ← { c → f̲loat (r̲ange 3) '= t̲able f̲loor c }
Used in: ᵘv̲alid

ᵘv̲alid : Num a => (Float, a) -> (Float, Float, Float)

function · line 336

v̲alid (Y, c2): (valid, preferred, rejected), one-hot: the two best classes, the best (Y itself), the second best (c2, from its file).

ᵘv̲alid ← { (Yc, c2) → (Yc + ᵘo̲nehot c2, Yc, ᵘo̲nehot c2) }
Used in: V, Vn

V : Float

value · line 337
(V, Yw, Yl) ← ᵘv̲alid (Y, n̲umbers ⎕N̲GET "data/shift-second-train.txt")
Used in: V, wrong, ms

Yw : Float

value · line 337
(V, Yw, Yl) ← ᵘv̲alid (Y, n̲umbers ⎕N̲GET "data/shift-second-train.txt")
Used in: V, styled, checked

Yl : Float

value · line 337
(V, Yw, Yl) ← ᵘv̲alid (Y, n̲umbers ⎕N̲GET "data/shift-second-train.txt")
Used in: V, styled, checked

Vn : Float

value · line 338
(Vn, Ywn, _) ← ᵘv̲alid (Yn, n̲umbers ⎕N̲GET "data/shift-second-test.txt")
Used in: Vn, ᵘp̲refers

Ywn : Float

value · line 338
(Vn, Ywn, _) ← ᵘv̲alid (Yn, n̲umbers ⎕N̲GET "data/shift-second-test.txt")
Used in: Vn, ᵘp̲refers

ᵘp̲refers : (Float, Float, Float, Float, Float, Float) -> Float

function · line 342

m p̲refers x: how often a model's answer is one of the two valid classes, and how often it is the preferred one, on the held-out new tokens; then the old task's accuracy.

ᵘp̲refers ← { m →
  (_, _, P) ← m ᵘl̲ayers xn
  A ← 1 ᵐᵒᵉm̲ask P
  N ← f̲loat 2 s̲elect s̲hape xn
  (('+ r̲/ r̲avel A × Vn) ÷ N) c̲at (('+ r̲/ r̲avel A × Ywn) ÷ N) c̲at m ᵘa̲ccuracy (xo, Yo)
}

ᵘg̲radsD : Num a => (Float, Float, Float, Float, Float, Float) -> ((Float, Float, Float), a, Float) -> (Float, Float, Float, Float, Float, Float, Float)

function · line 354

m g̲radsD ((xs, Yw, Yl), Pr, b): DPO's loss (Rafailov et al., 2023) and gradients on the pairs: Yw the chosen answer of each token, Yl the rejected, Pr the reference model's probabilities, b its beta. With h = b ((log P - log Pr) . (Yw - Yl)) for a token, the loss is -log sigma(h), and its gradient at the softmax's input is -b sigma(-h) (Yw - Yl): the softmax's own part cancels between the two answers.

ᵘg̲radsD ← { m ((xs, Wy, Ly), Pr, b) →
  (z0, z, P) ← m ᵘf̲orward6 xs
  d ← Wy − Ly
  h ← b × '+ r̲/₁ d × (l̲og P) − l̲og Pr
  N ← f̲loat 2 s̲elect s̲hape xs
  q ← (s̲hape d) r̲eshape 1.0 ÷ 1.0 + e̲xp h
  (daS, dbS, dR, dW, dAs, dBs) ← m ᵘb̲ack6 (xs, z0, z, (n̲eg b × q × d) ÷ N)
  (daS, dbS, dR, dW, dAs, dBs, ('+ r̲/ l̲og 1.0 + e̲xp n̲eg h) ÷ N)
}
Used in: ᵘs̲tepD

ᵘs̲tepD : Num a => ((Float, Float, Float, Float), ((Float, Float, Float), a, Float)) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float) -> ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float), Float, Float)

function · line 364

(method, pairs) s̲tepD state: a step of DPO on the pairs.

ᵘs̲tepD ← { (method, pairs) (m, adam, t, losses) → method ᵘa̲dam6 ((m, adam, t, losses), m ᵘg̲radsD pairs) }
Used in: ᵘd̲po

ᵘd̲po : ((Float, Float, Float, Float, Float, Float), (Float, Float, Float, Float), (Float, Float, Float), Float) -> Int -> ((Float, Float, Float, Float, Float, Float), Float)

function · line 368

(m0, method, (xs, Yw, Yl), b) d̲po steps: DPO from the model m0, which is also the reference, Adam starting afresh: the model and the loss after each step.

ᵘd̲po ← { (m0, method, (xs, Yw2, Yl2), b) steps →
  (_, _, Pr) ← m0 ᵘf̲orward6 xs
  (aS, bS, R, Ws, As, Bs) ← m0
  (m, _, _, losses) ← steps '{ st → (method, ((xs, Yw2, Yl2), Pr, b)) ᵘs̲tepD st } p̲ower (m0, (0.0 × aS, 0.0 × aS, 0.0 × bS, 0.0 × bS, 0.0 × R, 0.0 × R, 0.0 × Ws, 0.0 × Ws, 0.0 × As, 0.0 × As, 0.0 × Bs, 0.0 × Bs), 1, 0 r̲eshape 0.0)
  (m, losses)
}
Used in: md1, md5, mc

styled : (Float, Float, Float)

value · line 377

The pairs: the house style's, both valid (preferred, rejected); and, to show what the research warns of, each valid answer against the invalid one (two pairs a token), which teaches right from wrong only.

styled ← (x, Yw, Yl)
Used in: md1, md5

wrong : Float

value · line 378
wrong ← 1.0 − V
Used in: checked

checked : (Float, Float, Float)

value · line 379
checked ← (o̲\ (o̲\ x) c̲at o̲\ x, o̲\ (o̲\ Yw) c̲at o̲\ Yl, o̲\ (o̲\ wrong) c̲at o̲\ wrong)
Used in: mc

Each method, seed 1, 120 steps

ᵘr̲ow : ((Float, Float, Float, Float), (Float, Float)) -> Float

function · line 388

A line per method: the new task's accuracy, the old task's, the numbers trained, their bytes, the new tokens' share of each of the 5 experts, the old tokens', then the update sizes (the 4 experts, the shared layer, the new expert).

ᵘr̲ow ← { (method, tokens) →
  (m, _) ← (method, tokens, 120) ᵘt̲une 1
  (a, o, n, b, sn, so, sz) ← m ᵘr̲eport method
  ᵘr̲ound a c̲at o c̲at n c̲at b c̲at sn c̲at so c̲at sz
}
Used in: ᵘa̲ll

ᵘa̲ll : (Float, Float) -> Float

function · line 393
ᵘa̲ll ← { tokens → 7 20 r̲eshape (ᵘr̲ow (none, tokens)) c̲at (ᵘr̲ow (shared, tokens)) c̲at (ᵘr̲ow (experts, tokens)) c̲at (ᵘr̲ow (both, tokens)) c̲at (ᵘr̲ow (selected, tokens)) c̲at (ᵘr̲ow (router, tokens)) c̲at ᵘr̲ow (added, tokens) }

The four adapters, seed 1, 120 steps, on the new tokens alone

ᵘl̲ine : (Int, Int, Float) -> Float

function · line 403

A line per adapter (shared LoRA, expert LoRA, LoRA-FA, shared VeRA): the new task's accuracy, the old task's, the numbers trained; at equal ranks, then at equal budgets; at the rate 0.02, then 0.1.

ᵘl̲ine ← { (kind, r, lr) →
  (m, _, n) ← (kind, r, (x, Y), 120, lr) ᵘc̲ompare 1
  (ᵘr̲ound m ᵘa̲ccuracy (xn, Yn)) c̲at (ᵘr̲ound m ᵘa̲ccuracy (xo, Yo)) c̲at f̲loat n
}
Used in: ᵘf̲our

ᵘf̲our : (Int, Float) -> Float

function · line 407
ᵘf̲our ← { (ranks, lr) → 4 3 r̲eshape (ᵘl̲ine (1, 1 s̲elect ranks, lr)) c̲at (ᵘl̲ine (2, 2 s̲elect ranks, lr)) c̲at (ᵘl̲ine (3, 3 s̲elect ranks, lr)) c̲at ᵘl̲ine (4, 4 s̲elect ranks, lr) }

Deploying, seed 1, 120 steps

m1 : (Float, Float, Float, Float, Float, Float)

value · line 420

Kept apart against merged, the new task's adapters on every expert (the router frozen): LoRA on the base, then QLoRA on its 4-bit base (trained on W~, merged by quantizing W~ + s B A again). A line each: the largest difference of the probabilities, the tokens whose answer changed (of 800), the new task's accuracy kept apart, then merged.

(m1, _) ← (experts, (x, Y), 120) ᵘt̲une 1

aS1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1

bS1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1

R1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1

W1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1

A1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1
Used in: aS1, ml

B1 : Float

value · line 421
(aS1, bS1, R1, W1, A1, B1) ← m1
Used in: aS1, ml

mq : (Float, Float, Float, Float, Float, Float)

value · line 423
(mq, _) ← (Wq, experts, (x, Y), 120) ᵘt̲uneOn 1

Aq : Float

value · line 424
(_, _, _, _, Aq, Bq) ← mq
Used in: Aq, mm

Bq : Float

value · line 424
(_, _, _, _, Aq, Bq) ← mq
Used in: Aq, mm

ml : (Float, Float, Float, Float, Float, Float)

value · line 425
ml ← (W1 + ᵘd̲eltas (A1, B1)) ᵘm̲erged m1

mm : (Float, Float, Float, Float, Float, Float)

value · line 426
mm ← (ˡᵒd̲equant 9 ˡᵒq̲base Wq + ᵘd̲eltas (Aq, Bq)) ᵘm̲erged mq

m2 : (Float, Float, Float, Float, Float, Float)

value · line 434

Two tasks' adapters on one base: the base, then with the first task's adapters, then with the second's (switched: the same base, the other adapters), each on the first task, the second and the kinds neither changed.

(m2, _) ← (experts, (x2, Y2), 120) ᵘt̲une 1
Used in: ᵘa̲dam6, m2, A2

A2 : Float

value · line 435
(_, _, _, _, A2, B2) ← m2

B2 : Float

value · line 435
(_, _, _, _, A2, B2) ← m2

be : Float

value · line 450

At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.

(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)

bl : Float

value · line 450

At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.

(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)

bf : Float

value · line 450

At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.

(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)

bv : Float

value · line 450

At the research's size, one SwiGLU expert of 4096 x 14336 at rank 16 in BF16 (the LoRA library's counts): the expert, its LoRA adapters, LoRA-FA's B's, VeRA's vectors, in bytes; and the cache at 2 slots.

(be, bl, bf, bv) ← (16 ˡᵒb̲ytes 3 × 4096 × 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpert 4096 14336, 16 ˡᵒb̲ytes f̲irst 16 ˡᵒe̲xpertFA 4096 14336, 16 ˡᵒb̲ytes 16 ˡᵒe̲xpertV 4096 14336)

Preference, seed 1, 120 steps each

mp : (Float, Float, Float, Float, Float, Float)

value · line 464
(mp, _) ← (experts, (x, Y), 120) ᵘt̲une 1

ms : (Float, Float, Float, Float, Float, Float)

value · line 465
(ms, _) ← (experts, (x, 0.5 × V), 120) ᵘt̲une 1

md1 : (Float, Float, Float, Float, Float, Float)

value · line 466
(md1, _) ← (ms, experts, styled, 1.0) ᵘd̲po 120

md5 : (Float, Float, Float, Float, Float, Float)

value · line 467
(md5, _) ← (ms, experts, styled, 5.0) ᵘd̲po 120

mc : (Float, Float, Float, Float, Float, Float)

value · line 468
(mc, _) ← (ms, experts, checked, 1.0) ᵘd̲po 120