librarylibs/LoRA/src/LoRA.xtl
LoRA: low-rank adapters: the adapted forward pass in factored form, the merge, the counts, the gradients and steps, LoRA-FA, QLoRA, VeRA
Import it with an alias of your choice: "lo:" u_se< "LoRA". Put libs/LoRA/src on XETAL_PATH ("just path"); the reference is libs/LoRA/docs. Names with l: are exported; h: names are private to this file.
A frozen weight matrix W (m x n: m outputs, n inputs) is adapted by
a low-rank update s B A: A is r x n, B is m x r, the rank r much
smaller than m and n, s the scale alpha / r. An adapter is the tuple
(W, A, B, s). The inputs are columns: x is one input of n numbers or
an n x N matrix of N inputs, and W x is W '+ '× i̲nner x. The
forward pass keeps the factors apart, B (A x), as training does:
the merged matrix W + s B A is made only to deploy it.
Training moves A and B by the gradients of the squared error (g_rads, s_gd, a_dam through the Optim library); LoRA-FA freezes A (regenerated from a seed, s_eeded) and trains B alone; QLoRA keeps W in 4 bits (NF4, through the Quant library) and computes with it dequantized; VeRA freezes A and B too (random, from a seed, shared by every matrix of the shape) and trains two vectors, d and b.
The forward pass
ˡs̲caling : Num a => Float -> a -> Float
alpha s̲caling r: the adapter's scale s, alpha / r (so that a
change of rank keeps the update's size).
"lo:" u̲se< "LoRA"
16.0 ˡᵒs̲caling 8 2.0
ˡs̲caling ← { alpha r → alpha ÷ f̲loat r }
ˡs̲calingRs : Num a => Float -> a -> Float
alpha s̲calingRs r: rsLoRA's rank-stabilized scale, alpha / sqrt r
(Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning
with LoRA", 2023): with alpha / r the update's learning slows as r
grows; over sqrt r it does not. An adapter takes either, (W, A, B, s).
"lo:" u̲se< "LoRA"
16.0 ˡᵒs̲calingRs 4 8.0
ˡs̲calingRs ← { alpha r → alpha ÷ s̲qrt f̲loat r }
ˡf̲orward : Num a => (a, a, a, a) -> a -> a
(W, A, B, s) f̲orward x: the adapted layer, W x + s B (A x), the
update in its factored form: A x (r numbers per input), then B of
that, never the m x n matrix B A.
"lo:" u̲se< "LoRA"
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒf̲orward 2.0 4.0 5.0 1.0
ˡf̲orward ← { (W, A, B, s) x → (W '+ '× i̲nner x) + s × B '+ '× i̲nner A '+ '× i̲nner x }
The merge
ˡd̲elta : Num a => (a, a, a) -> a
d̲elta (A, B, s): the update itself, s B A, an m x n matrix of
rank r at most.
"lo:" u̲se< "LoRA"
ˡᵒd̲elta (1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) 0.5 0.5 -0.5 -0.5
ˡd̲elta ← { (A, B, s) → s × B '+ '× i̲nner A }
ˡm̲erge : Num a => (a, a, a, a) -> a
m̲erge (W, A, B, s): the adapter folded into the weights, W + s B
A: one matrix again, as fast as the base layer, for one task.
"lo:" u̲se< "LoRA"
ˡᵒm̲erge (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) 1.5 0.5 -0.5 0.5
ˡm̲erge ← { (W, A, B, s) → W + ˡd̲elta (A, B, s) }
Gradients
ˡl̲oss : (Float, Float, Float, Float) -> (Float, Float) -> Float
(W, A, B, s) l̲oss (x, y): the squared error of the adapted layer
on the inputs x (columns) against the wanted outputs y, half the sum
of the squared differences, so that its gradient at the output, g,
is the difference itself.
"lo:" u̲se< "LoRA"
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒl̲oss (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0) 1.0
ˡl̲oss ← { a (x, y) → e ← (a ˡf̲orward x) − y 0.5 × '+ r̲/ r̲avel e × e }
ˡg̲rads : Num a => (a, a, a, a) -> (a, a) -> (a, a, a)
(W, A, B, s) g̲rads (x, g): the gradients for the inputs x (an n x
N matrix, one input per column) and g, the loss's gradient at the
output (m x N): with z = A x, dL/dA = s (B' g) x', dL/dB = s g z' and
dL/dx = W' g + s A' (B' g), as the tuple (dA, dB, dx). B' g is made
once and used twice; no m x n matrix but W's own is made.
"lo:" u̲se< "LoRA"
(dA, dB, dx) ← (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒg̲rads (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 1.0 -1.0)
dB
3.0
-3.0
ˡg̲rads ← { (W, A, B, s) (x, g) → b ← (o̲\ B) '+ '× i̲nner g (s × b '+ '× i̲nner o̲\ x, s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner b) }
Training steps
ˡs̲gd : Num a => a -> ((a, a, a, a), (a, a)) -> (a, a, a, a)
lr s̲gd ((W, A, B, s), (x, y)): one step of gradient descent on
the squared error (l_oss) at the rate lr, A and B moved by Optim's
s_gd, W and s left as they are: the adapter after the step. p_ower
repeats it.
"lo:" u̲se< "LoRA"
(_, A, B, _) ← 0.1 ˡᵒs̲gd ((2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5), (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0))
B
0.7
-0.7
ˡs̲gd ← { lr ((W, A, B, s), (x, y)) → (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y) (W, lr ᵒᵖs̲gd (A, dA), lr ᵒᵖs̲gd (B, dB), s) }
ˡa̲damStart : (Any a, Any b, Num c) => (a, Float, Float, b) -> ((a, Float, Float, b), (Float, Float, Float, Float), c)
a̲damStart (W, A, B, s): what Adam starts from: the adapter, its
four running averages at 0 (A's two, then B's) and the step count 1.
ˡa̲damStart ← { (W, A, B, s) → ((W, A, B, s), (0.0 × A, 0.0 × A, 0.0 × B, 0.0 × B), 1) }
ˡa̲dam : Float -> (((Float, Float, Float, Float), (Float, Float, Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)
lr a̲dam (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)): Adam's step
t on the squared error, A and B moved by Optim's a_dam at the rate
lr; the next state, the step count one more.
"lo:" u̲se< "LoRA"
((_, _, B, _), _, _) ← 0.1 ˡᵒa̲dam ((ˡᵒa̲damStart (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5)), (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0))
B
0.9000000003333333
-0.9000000003333333
ˡa̲dam ← { lr (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)) → (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y) (A2, mA2, vA2) ← (lr, t) ᵒᵖa̲dam (A, mA, vA, dA) (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB) ((W, A2, B2, s), (mA2, vA2, mB2, vB2), t + 1) }
LoRA-FA: A frozen
ˡs̲eeded : Num a => a -> Int -> Float
seed s̲eeded r n: an r x n matrix regenerated from a whole-number
seed (LoRA-FA's frozen A, which a checkpoint then need not hold):
sin(seed k k) for k = 1 .. r n, over the square root of n (the
inputs' count, so A x keeps x's size). No chance: the same seed
gives the same matrix everywhere.
"lo:" u̲se< "LoRA"
7 ˡᵒs̲eeded 2 4 0.32849329935939453 0.13545289415393452 0.08367785015140346 -0.44499780218341667 -0.4005672975890204 0.3115061055018265 -0.26829917759428185 0.47413628607807934
ˡs̲eeded ← { seed rn → k ← f̲loat r̲ange '× r̲/ rn rn r̲eshape (s̲in (f̲loat seed) × k × k) ÷ s̲qrt f̲loat 2 s̲elect rn }
ˡg̲radsFA : Num a => (a, a, a, a) -> (a, a) -> (a, a)
(W, A, B, s) g̲radsFA (x, g): LoRA-FA's gradients, (dB, dx): A is
frozen, so its gradient is skipped, but x's still flows back through
A, dx = W' g + s A' (B' g).
ˡg̲radsFA ← { (W, A, B, s) (x, g) → (s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner (o̲\ B) '+ '× i̲nner g) }
ˡs̲gdFA : Num a => a -> ((a, a, a, a), (a, a)) -> (a, a, a, a)
lr s̲gdFA ((W, A, B, s), (x, y)): LoRA-FA's step of gradient
descent: B alone moves. Every update s B A it reaches has its rows
in A's row space.
ˡs̲gdFA ← { lr ((W, A, B, s), (x, y)) → (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y) (W, A, lr ᵒᵖs̲gd (B, dB), s) }
ˡa̲damStartFA : (Any a, Any b, Any c, Num d) => (a, b, Float, c) -> ((a, b, Float, c), (Float, Float), d)
a̲damStartFA (W, A, B, s): LoRA-FA's Adam state: the adapter, B's
two running averages at 0 (A has none: half of LoRA's optimizer
memory on a square layer), the step 1.
ˡa̲damStartFA ← { (W, A, B, s) → ((W, A, B, s), (0.0 × B, 0.0 × B), 1) }
ˡa̲damFA : Float -> (((Float, Float, Float, Float), (Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float), (Float, Float), Float)
lr a̲damFA (((W, A, B, s), (mB, vB), t), (x, y)): LoRA-FA's Adam
step: B alone moves; the next state.
ˡa̲damFA ← { lr (((W, A, B, s), (mB, vB), t), (x, y)) → (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y) (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB) ((W, A, B2, s), (mB2, vB2), t + 1) }
VeRA: shared random projections, two vectors trained
ˡp̲rojections : Num a => a -> Int -> (Float, Float)
seed p̲rojections m n r: VeRA's frozen projections for an m x n
matrix at rank r, (A0, B0): A0 (r x n) and B0 (m x r) regenerated
from the seed as s_eeded makes them (B0 from seed + 1), so a
checkpoint need not hold them and every layer or expert of the same
shape can share them.
"lo:" u̲se< "LoRA"
(A0, B0) ← 7 ˡᵒp̲rojections 3 4 2
s̲hape B0 3 2
ˡp̲rojections ← { seed mnr → (m, n, r) ← (f̲irst mnr, 2 s̲elect mnr, 3 s̲elect mnr) (seed ˡs̲eeded r c̲at n, (seed + 1) ˡs̲eeded m c̲at r) }
ˡv̲era : Num a => a -> (b, Int) -> (b, Float, Float, Float, Float)
seed v̲era (W, r): a VeRA adapter on the frozen W (m x n) at rank
r, as the paper starts it (Kopiczko, Blankevoort and Asano, "VeRA:
Vector-based Random Matrix Adaptation", 2024): (W, A0, B0, d, b),
the projections from the seed, d (r numbers) at 0.1 and b (m
numbers) at 0, so the adapted layer starts as the base. d and b are
all it trains: m + r numbers.
"lo:" u̲se< "LoRA"
(_, _, _, d, b) ← 7 ˡᵒv̲era (2 3 r̲eshape 1.0, 4)
d c̲at b 0.1 0.1 0.1 0.1 0.0 0.0
ˡv̲era ← { seed (W, r) → (A0, B0) ← seed ˡp̲rojections (s̲hape W) c̲at r (W, A0, B0, r r̲eshape 0.1, (f̲irst s̲hape W) r̲eshape 0.0) }
ˡd̲eltaV : Num a => (a, a, a, a) -> a
(A0, B0, d, b) d̲eltaV: VeRA's update, diag(b) B0 diag(d) A0, an
m x n matrix of rank r at most. The two diagonals need no matrix and
no broadcasting: diag(b) B0 diag(d) is B0 times the table of b by d,
item by item (an m x r matrix, B's size).
"lo:" u̲se< "LoRA"
ˡᵒd̲eltaV (1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) 1.0 1.0 -2.0 -2.0
ˡd̲eltaV ← { (A0, B0, d, b) → (B0 × b '× t̲able d) '+ '× i̲nner A0 }
ˡf̲orwardV : Num a => (a, a, a, a, a) -> a -> a
(W, A0, B0, d, b) f̲orwardV x: the VeRA layer, W x + b (B0 (d (A0
x))), the vectors scaling A0 x's rows and B0's output's: in factored
form, A0 x first, then the m x r matrix diag(b) B0 diag(d) of that.
"lo:" u̲se< "LoRA"
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) ˡᵒf̲orwardV 2.0 4.0 8.0 -8.0
ˡf̲orwardV ← { (W, A0, B0, d, b) x → (W '+ '× i̲nner x) + (B0 × b '× t̲able d) '+ '× i̲nner A0 '+ '× i̲nner x }
ˡm̲ergeV : Num a => (a, a, a, a, a) -> a
m̲ergeV (W, A0, B0, d, b): the VeRA adapter folded into the
weights, W + diag(b) B0 diag(d) A0.
"lo:" u̲se< "LoRA"
ˡᵒm̲ergeV (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) 2.0 1.0 -2.0 -1.0
ˡm̲ergeV ← { (W, A0, B0, d, b) → W + ˡd̲eltaV (A0, B0, d, b) }
ˡl̲ossV : (Float, Float, Float, Float, Float) -> (Float, Float) -> Float
(W, A0, B0, d, b) l̲ossV (x, y): the squared error of the VeRA
layer, as l_oss.
ˡl̲ossV ← { a (x, y) → e ← (a ˡf̲orwardV x) − y 0.5 × '+ r̲/ r̲avel e × e }
ˡg̲radsV : Num a => (a, a, a, a, a) -> (a, a) -> (a, a, a)
(W, A0, B0, d, b) g̲radsV (x, g): VeRA's gradients for the inputs
x (n x N) and the output's gradient g (m x N), as (dd, db, dx). With
z = A0 x and Q = B0 times g z' item by item (m x r): dd = b' Q (the
research's (B0' (g b)) z, summed over the inputs), db = Q d (its g
(B0 (d z))), dx = W' g + A0' (diag(b) B0 diag(d))' g. The projections
are in every gradient and get none.
"lo:" u̲se< "LoRA"
(dd, db, dx) ← (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) ˡᵒg̲radsV (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 1.0 -1.0)
dd c̲at db 36.0 3.0 3.0
ˡg̲radsV ← { (W, A0, B0, d, b) (x, g) → Q ← B0 × g '+ '× i̲nner o̲\ A0 '+ '× i̲nner x (b '+ '× i̲nner Q, Q '+ '× i̲nner d, ((o̲\ W) '+ '× i̲nner g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × b '× t̲able d) '+ '× i̲nner g) }
ˡs̲gdV : Num a => a -> ((a, a, a, a, a), (a, a)) -> (a, a, a, a, a)
lr s̲gdV ((W, A0, B0, d, b), (x, y)): VeRA's step of gradient
descent on the squared error: d and b move, the rest stays.
ˡs̲gdV ← { lr ((W, A0, B0, d, b), (x, y)) → (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y) (W, A0, B0, lr ᵒᵖs̲gd (d, dd), lr ᵒᵖs̲gd (b, db)) }
ˡa̲damStartV : (Any a, Any b, Any c, Num d) => (a, b, c, Float, Float) -> ((a, b, c, Float, Float), (Float, Float, Float, Float), d)
a̲damStartV (W, A0, B0, d, b): VeRA's Adam state: the adapter, the
running averages of d and b at 0, the step 1.
ˡa̲damStartV ← { (W, A0, B0, d, b) → ((W, A0, B0, d, b), (0.0 × d, 0.0 × d, 0.0 × b, 0.0 × b), 1) }
ˡa̲damV : Float -> (((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float)
lr a̲damV (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)):
VeRA's Adam step: d and b move; the next state.
ˡa̲damV ← { lr (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)) → (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y) (d2, md2, vd2) ← (lr, t) ᵒᵖa̲dam (d, md, vd, dd) (b2, mb2, vb2) ← (lr, t) ᵒᵖa̲dam (b, mb, vb, db) ((W, A0, B0, d2, b2), (md2, vd2, mb2, vb2), t + 1) }
QLoRA: the base in 4 bits
ˡq̲base : Int -> Float -> (Int, Float, Int)
k q̲base W: the base as QLoRA stores it: W in NF4, blocks of k
(QLoRA's 64), as (codes, scales, shape): 4-bit codes, a scale per
block, W's shape to come back to.
"lo:" u̲se< "LoRA"
(q, s, sh) ← 2 ˡᵒq̲base 2 2 r̲eshape 1.0 -4.0 2.0 0.5
s c̲at f̲loat sh 4.0 2.0 2.0 2.0
ˡq̲base ← { k W → (q, s) ← k qz:n̲f4Codes W (q, s, s̲hape W) }
ˡd̲equant : (Int, Float, Int) -> Float
d̲equant (q, s, shape): the stored base back to numbers, W~, the
matrix QLoRA computes with: each code's level times its block's
scale, in W's shape. It is not W: the difference is the 4-bit
error.
"lo:" u̲se< "LoRA"
ˡᵒd̲equant 2 ˡᵒq̲base 2 2 r̲eshape 1.0 -4.0 2.0 0.5 0.9844492077827454 -4.0 2.0 0.4922246038913727
ˡd̲equant ← { (q, s, sh) → sh r̲eshape qz:n̲f4Values (q, s) }
ˡf̲orwardQ : ((Int, Float, Int), Float, Float, Float) -> Float -> Float
(Q, A, B, s) f̲orwardQ x: QLoRA's adapted layer, W~ x + s B (A x):
the 4-bit base dequantized when the layer runs, the adapter in full
precision. Its gradients and steps are LoRA's with W~ for W:
lr s̲gd ((d̲equant Q, A, B, s), (x, y)).
ˡf̲orwardQ ← { (Q, A, B, s) x → (ˡd̲equant Q, A, B, s) ˡf̲orward x }
ˡm̲ergeQ : Int -> ((Int, Float, Int), Float, Float, Float) -> (Int, Float, Int)
k m̲ergeQ (Q, A, B, s): the adapter merged into a 4-bit base: W~ +
s B A quantized again (blocks of k), as a stored base. Not exact:
d_equant of it is not W~ + s B A, since the merged numbers fall
between NF4's levels.
ˡm̲ergeQ ← { k (Q, A, B, s) → k ˡq̲base (ˡd̲equant Q) + ˡd̲elta (A, B, s) }
ˡq̲bytes : (Num a, Num b) => (a, b) -> a -> Float
(r, k) q̲bytes m n: QLoRA's bytes for an m x n base at rank r,
blocks of k: the base in NF4 (4 bits and a 32-bit scale per block),
the adapter in BF16 (16 bits), Adam's two averages for it in FP32;
against FP16's 2 m n for the base alone.
"lo:" u̲se< "LoRA"
(16, 64) ˡᵒq̲bytes 4096 14336 33030144.0 589824.0 2359296.0
ˡq̲bytes ← { (r, k) mn → n ← r ˡp̲arams mn ((f̲loat '× r̲/ mn) × (4.0 + 32.0 ÷ f̲loat k) ÷ 8.0) c̲at (16 ˡb̲ytes n) c̲at 2.0 × 32 ˡb̲ytes n }
Rank
ʰs̲tep : Num a => Float -> (Float, Int, a) -> (Float, Int, a)
One step of Gram-Schmidt on the rows: the k-th row's remainder R[k], if larger than tol, becomes a direction and is taken out of every row, and the count c goes up.
ʰs̲tep ← { tol (R, k, c) → v ← k s̲elect R n ← s̲qrt '+ r̲/ v × v n ≤ tol ? (R, k + 1, c) (R − (R '+ '× i̲nner v ÷ n) '× t̲able v ÷ n, k + 1, c + 1) }
ˡr̲ank : Num a => Float -> a
r̲ank M: the number of independent rows of the matrix M (its
rank), by Gram-Schmidt: a row counts when what is left of it, after
the directions of the rows before it are taken out, is larger than
1e-9 times the largest row.
"lo:" u̲se< "LoRA"
ˡᵒr̲ank 3 3 r̲eshape 1.0 2.0 3.0 2.0 4.0 6.0 0.0 1.0 1.0 2
ˡr̲ank ← { M → tol ← 0.000000001 × s̲qrt 'm̲ax r̲/ '+ r̲/₂ M × M (_, _, c) ← (f̲irst s̲hape M) '{ R → tol ʰs̲tep R } p̲ower (M, 1, 0) c }
Counts
ˡp̲arams : Num a => a -> a -> a
r p̲arams m n: the numbers an adapter of rank r on an m x n matrix
trains, r (m + n) (B's m r and A's r n), against the m n of the
matrix itself.
"lo:" u̲se< "LoRA"
16 ˡᵒp̲arams 4096 14336 294912
ˡp̲arams ← { r mn → r × '+ r̲/ mn }
ˡe̲xpert : Num a => a -> a -> a
r e̲xpert d f: the numbers rank-r adapters train on one SwiGLU
expert of hidden size d and inner size f: its three matrices (gate
and up, f x d; down, d x f), 3 r (d + f).
"lo:" u̲se< "LoRA"
16 ˡᵒe̲xpert 4096 14336 884736
ˡe̲xpert ← { r df → 3 × r ˡp̲arams df }
ˡe̲xpertFA : Num a => a -> a -> a
r e̲xpertFA d f: the same expert under LoRA-FA: the numbers it
trains (the B's, r (2f + d)), then the frozen A's it regenerates
(r (2d + f)); together r e_xpert d f.
"lo:" u̲se< "LoRA"
16 ˡᵒe̲xpertFA 4096 14336 524288 360448
ˡe̲xpertFA ← { r df → (d, f) ← (f̲irst df, 2 s̲elect df) (r × f + f + d) c̲at r × d + d + f }
ˡb̲ytes : Num a => a -> a -> Float
b b̲ytes n: the bytes n numbers take at b bits each (16 for BF16
and FP16, 32 for FP32, 4 for NF4).
"lo:" u̲se< "LoRA"
16 ˡᵒb̲ytes 884736 1769472.0
ˡb̲ytes ← { b n → (f̲loat b × n) ÷ 8.0 }
ˡc̲osts : Num a => a -> a -> a
r c̲osts m n: the multiply-adds for one input through rank-r
factors on an m x n matrix, B (A x) as r (m + n), then what making
the matrix B A first costs, r m n.
"lo:" u̲se< "LoRA"
8 ˡᵒc̲osts 64 64 1024 32768
ˡc̲osts ← { r mn → (r ˡp̲arams mn) c̲at r × '× r̲/ mn }
ˡp̲aramsV : Num a => a -> a -> a
r p̲aramsV m n: the numbers VeRA trains on an m x n matrix at rank
r, m + r (b's m and d's r), against LoRA's r (m + n); its frozen
projections, r (m + n) numbers, are shared by every matrix of the
shape (or regenerated from their seed).
"lo:" u̲se< "LoRA"
16 ˡᵒp̲aramsV 14336 4096 14352
ˡp̲aramsV ← { r mn → r + f̲irst mn }
ˡe̲xpertV : Num a => a -> a -> a
r e̲xpertV d f: the numbers VeRA trains on one SwiGLU expert of
hidden size d and inner size f, its three matrices at rank r: m + r
each, 2 f + d + 3 r. The projections are frozen and shared by every
expert: two pairs (one for gate and up, f x d; one for down, d x f),
2 r (d + f) numbers, once.
"lo:" u̲se< "LoRA"
16 ˡᵒe̲xpertV 4096 14336 32816
ˡe̲xpertV ← { r df → (3 × r) + (2 × 2 s̲elect df) + f̲irst df }