librarylibs/LoRA/src/LoRA.xtl

LoRA: low-rank adapters: the adapted forward pass in factored form, the merge, the counts, the gradients and steps, LoRA-FA, QLoRA, VeRA

Import it with an alias of your choice: "lo:" u_se< "LoRA". Put libs/LoRA/src on XETAL_PATH ("just path"); the reference is libs/LoRA/docs. Names with l: are exported; h: names are private to this file.

A frozen weight matrix W (m x n: m outputs, n inputs) is adapted by a low-rank update s B A: A is r x n, B is m x r, the rank r much smaller than m and n, s the scale alpha / r. An adapter is the tuple (W, A, B, s). The inputs are columns: x is one input of n numbers or an n x N matrix of N inputs, and W x is W '+ '× i̲nner x. The forward pass keeps the factors apart, B (A x), as training does: the merged matrix W + s B A is made only to deploy it.

Training moves A and B by the gradients of the squared error (g_rads, s_gd, a_dam through the Optim library); LoRA-FA freezes A (regenerated from a seed, s_eeded) and trains B alone; QLoRA keeps W in 4 bits (NF4, through the Quant library) and computes with it dequantized; VeRA freezes A and B too (random, from a seed, shared by every matrix of the shape) and trains two vectors, d and b.

source · imports op: libs/Optim/src/Optim.xtl; qz: libs/Quant/src/Quant.xtl

The forward pass

ˡs̲caling : Num a => Float -> a -> Float

function · line 32

alpha s̲caling r: the adapter's scale s, alpha / r (so that a change of rank keeps the update's size).

      "lo:" u̲se< "LoRA"
      16.0 ˡᵒs̲caling 8
2.0
ˡs̲caling ← { alpha r → alpha ÷ f̲loat r }

ˡs̲calingRs : Num a => Float -> a -> Float

function · line 41

alpha s̲calingRs r: rsLoRA's rank-stabilized scale, alpha / sqrt r (Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA", 2023): with alpha / r the update's learning slows as r grows; over sqrt r it does not. An adapter takes either, (W, A, B, s).

      "lo:" u̲se< "LoRA"
      16.0 ˡᵒs̲calingRs 4
8.0
ˡs̲calingRs ← { alpha r → alpha ÷ s̲qrt f̲loat r }

ˡf̲orward : Num a => (a, a, a, a) -> a -> a

function · line 49

(W, A, B, s) f̲orward x: the adapted layer, W x + s B (A x), the update in its factored form: A x (r numbers per input), then B of that, never the m x n matrix B A.

      "lo:" u̲se< "LoRA"
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒf̲orward 2.0 4.0
5.0 1.0
ˡf̲orward ← { (W, A, B, s) x → (W '+ '× i̲nner x) + s × B '+ '× i̲nner A '+ '× i̲nner x }

The merge

ˡd̲elta : Num a => (a, a, a) -> a

function · line 59

d̲elta (A, B, s): the update itself, s B A, an m x n matrix of rank r at most.

      "lo:" u̲se< "LoRA"
      ˡᵒd̲elta (1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5)
 0.5  0.5
-0.5 -0.5
ˡd̲elta ← { (A, B, s) → s × B '+ '× i̲nner A }

ˡm̲erge : Num a => (a, a, a, a) -> a

function · line 67

m̲erge (W, A, B, s): the adapter folded into the weights, W + s B A: one matrix again, as fast as the base layer, for one task.

      "lo:" u̲se< "LoRA"
      ˡᵒm̲erge (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5)
 1.5 0.5
-0.5 0.5
ˡm̲erge ← { (W, A, B, s) → W + ˡd̲elta (A, B, s) }

Gradients

ˡl̲oss : (Float, Float, Float, Float) -> (Float, Float) -> Float

function · line 78

(W, A, B, s) l̲oss (x, y): the squared error of the adapted layer on the inputs x (columns) against the wanted outputs y, half the sum of the squared differences, so that its gradient at the output, g, is the difference itself.

      "lo:" u̲se< "LoRA"
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒl̲oss (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0)
1.0
ˡl̲oss ← { a (x, y) →
  e ← (a ˡf̲orward x) − y
  0.5 × '+ r̲/ r̲avel e × e
}
Used in: ᵘl̲oss

ˡg̲rads : Num a => (a, a, a, a) -> (a, a) -> (a, a, a)

function · line 93

(W, A, B, s) g̲rads (x, g): the gradients for the inputs x (an n x N matrix, one input per column) and g, the loss's gradient at the output (m x N): with z = A x, dL/dA = s (B' g) x', dL/dB = s g z' and dL/dx = W' g + s A' (B' g), as the tuple (dA, dB, dx). B' g is made once and used twice; no m x n matrix but W's own is made.

      "lo:" u̲se< "LoRA"
      (dA, dB, dx) ← (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5) ˡᵒg̲rads (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 1.0 -1.0)
      dB
 3.0
-3.0
ˡg̲rads ← { (W, A, B, s) (x, g) →
  b ← (o̲\ B) '+ '× i̲nner g
  (s × b '+ '× i̲nner o̲\ x, s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner b)
}

Training steps

ˡs̲gd : Num a => a -> ((a, a, a, a), (a, a)) -> (a, a, a, a)

function · line 109

lr s̲gd ((W, A, B, s), (x, y)): one step of gradient descent on the squared error (l_oss) at the rate lr, A and B moved by Optim's s_gd, W and s left as they are: the adapter after the step. p_ower repeats it.

      "lo:" u̲se< "LoRA"
      (_, A, B, _) ← 0.1 ˡᵒs̲gd ((2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5), (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0))
      B
 0.7
-0.7
ˡs̲gd ← { lr ((W, A, B, s), (x, y)) →
  (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y)
  (W, lr ᵒᵖs̲gd (A, dA), lr ᵒᵖs̲gd (B, dB), s)
}

ˡa̲damStart : (Any a, Any b, Num c) => (a, Float, Float, b) -> ((a, Float, Float, b), (Float, Float, Float, Float), c)

function · line 116

a̲damStart (W, A, B, s): what Adam starts from: the adapter, its four running averages at 0 (A's two, then B's) and the step count 1.

ˡa̲damStart ← { (W, A, B, s) → ((W, A, B, s), (0.0 × A, 0.0 × A, 0.0 × B, 0.0 × B), 1) }
Used in: q1, q3, ᵘl̲ora

ˡa̲dam : Float -> (((Float, Float, Float, Float), (Float, Float, Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)

function · line 126

lr a̲dam (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)): Adam's step t on the squared error, A and B moved by Optim's a_dam at the rate lr; the next state, the step count one more.

      "lo:" u̲se< "LoRA"
      ((_, _, B, _), _, _) ← 0.1 ˡᵒa̲dam ((ˡᵒa̲damStart (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 0.5)), (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 4.0 2.0))
      B
 0.9000000003333333
-0.9000000003333333
ˡa̲dam ← { lr (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)) →
  (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y)
  (A2, mA2, vA2) ← (lr, t) ᵒᵖa̲dam (A, mA, vA, dA)
  (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB)
  ((W, A2, B2, s), (mA2, vA2, mB2, vB2), t + 1)
}
Used in: q1, q3, ᵘl̲ora

LoRA-FA: A frozen

ˡs̲eeded : Num a => a -> Int -> Float

function · line 144

seed s̲eeded r n: an r x n matrix regenerated from a whole-number seed (LoRA-FA's frozen A, which a checkpoint then need not hold): sin(seed k k) for k = 1 .. r n, over the square root of n (the inputs' count, so A x keeps x's size). No chance: the same seed gives the same matrix everywhere.

      "lo:" u̲se< "LoRA"
      7 ˡᵒs̲eeded 2 4
0.32849329935939453 0.13545289415393452  0.08367785015140346 -0.44499780218341667
-0.4005672975890204  0.3115061055018265 -0.26829917759428185  0.47413628607807934
ˡs̲eeded ← { seed rn →
  k ← f̲loat r̲ange '× r̲/ rn
  rn r̲eshape (s̲in (f̲loat seed) × k × k) ÷ s̲qrt f̲loat 2 s̲elect rn
}

ˡg̲radsFA : Num a => (a, a, a, a) -> (a, a) -> (a, a)

function · line 152

(W, A, B, s) g̲radsFA (x, g): LoRA-FA's gradients, (dB, dx): A is frozen, so its gradient is skipped, but x's still flows back through A, dx = W' g + s A' (B' g).

ˡg̲radsFA ← { (W, A, B, s) (x, g) →
  (s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner (o̲\ B) '+ '× i̲nner g)
}

ˡs̲gdFA : Num a => a -> ((a, a, a, a), (a, a)) -> (a, a, a, a)

function · line 159

lr s̲gdFA ((W, A, B, s), (x, y)): LoRA-FA's step of gradient descent: B alone moves. Every update s B A it reaches has its rows in A's row space.

ˡs̲gdFA ← { lr ((W, A, B, s), (x, y)) →
  (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y)
  (W, A, lr ᵒᵖs̲gd (B, dB), s)
}

ˡa̲damStartFA : (Any a, Any b, Any c, Num d) => (a, b, Float, c) -> ((a, b, Float, c), (Float, Float), d)

function · line 167

a̲damStartFA (W, A, B, s): LoRA-FA's Adam state: the adapter, B's two running averages at 0 (A has none: half of LoRA's optimizer memory on a square layer), the step 1.

ˡa̲damStartFA ← { (W, A, B, s) → ((W, A, B, s), (0.0 × B, 0.0 × B), 1) }
Used in: q2, ᵘf̲a

ˡa̲damFA : Float -> (((Float, Float, Float, Float), (Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float), (Float, Float), Float)

function · line 171

lr a̲damFA (((W, A, B, s), (mB, vB), t), (x, y)): LoRA-FA's Adam step: B alone moves; the next state.

ˡa̲damFA ← { lr (((W, A, B, s), (mB, vB), t), (x, y)) →
  (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y)
  (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB)
  ((W, A, B2, s), (mB2, vB2), t + 1)
}
Used in: q2, ᵘf̲a

VeRA: shared random projections, two vectors trained

ˡp̲rojections : Num a => a -> Int -> (Float, Float)

function · line 188

seed p̲rojections m n r: VeRA's frozen projections for an m x n matrix at rank r, (A0, B0): A0 (r x n) and B0 (m x r) regenerated from the seed as s_eeded makes them (B0 from seed + 1), so a checkpoint need not hold them and every layer or expert of the same shape can share them.

      "lo:" u̲se< "LoRA"
      (A0, B0) ← 7 ˡᵒp̲rojections 3 4 2
      s̲hape B0
3 2
ˡp̲rojections ← { seed mnr →
  (m, n, r) ← (f̲irst mnr, 2 s̲elect mnr, 3 s̲elect mnr)
  (seed ˡs̲eeded r c̲at n, (seed + 1) ˡs̲eeded m c̲at r)
}

ˡv̲era : Num a => a -> (b, Int) -> (b, Float, Float, Float, Float)

function · line 203

seed v̲era (W, r): a VeRA adapter on the frozen W (m x n) at rank r, as the paper starts it (Kopiczko, Blankevoort and Asano, "VeRA: Vector-based Random Matrix Adaptation", 2024): (W, A0, B0, d, b), the projections from the seed, d (r numbers) at 0.1 and b (m numbers) at 0, so the adapted layer starts as the base. d and b are all it trains: m + r numbers.

      "lo:" u̲se< "LoRA"
      (_, _, _, d, b) ← 7 ˡᵒv̲era (2 3 r̲eshape 1.0, 4)
      d c̲at b
0.1 0.1 0.1 0.1 0.0 0.0
ˡv̲era ← { seed (W, r) →
  (A0, B0) ← seed ˡp̲rojections (s̲hape W) c̲at r
  (W, A0, B0, r r̲eshape 0.1, (f̲irst s̲hape W) r̲eshape 0.0)
}
Used in: A0, V, ᵘv̲era

ˡd̲eltaV : Num a => (a, a, a, a) -> a

function · line 216

(A0, B0, d, b) d̲eltaV: VeRA's update, diag(b) B0 diag(d) A0, an m x n matrix of rank r at most. The two diagonals need no matrix and no broadcasting: diag(b) B0 diag(d) is B0 times the table of b by d, item by item (an m x r matrix, B's size).

      "lo:" u̲se< "LoRA"
      ˡᵒd̲eltaV (1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0)
 1.0  1.0
-2.0 -2.0
ˡd̲eltaV ← { (A0, B0, d, b) → (B0 × b '× t̲able d) '+ '× i̲nner A0 }

ˡf̲orwardV : Num a => (a, a, a, a, a) -> a -> a

function · line 224

(W, A0, B0, d, b) f̲orwardV x: the VeRA layer, W x + b (B0 (d (A0 x))), the vectors scaling A0 x's rows and B0's output's: in factored form, A0 x first, then the m x r matrix diag(b) B0 diag(d) of that.

      "lo:" u̲se< "LoRA"
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) ˡᵒf̲orwardV 2.0 4.0
8.0 -8.0
ˡf̲orwardV ← { (W, A0, B0, d, b) x → (W '+ '× i̲nner x) + (B0 × b '× t̲able d) '+ '× i̲nner A0 '+ '× i̲nner x }

ˡm̲ergeV : Num a => (a, a, a, a, a) -> a

function · line 232

m̲ergeV (W, A0, B0, d, b): the VeRA adapter folded into the weights, W + diag(b) B0 diag(d) A0.

      "lo:" u̲se< "LoRA"
      ˡᵒm̲ergeV (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0)
 2.0  1.0
-2.0 -1.0
ˡm̲ergeV ← { (W, A0, B0, d, b) → W + ˡd̲eltaV (A0, B0, d, b) }

ˡl̲ossV : (Float, Float, Float, Float, Float) -> (Float, Float) -> Float

function · line 236

(W, A0, B0, d, b) l̲ossV (x, y): the squared error of the VeRA layer, as l_oss.

ˡl̲ossV ← { a (x, y) →
  e ← (a ˡf̲orwardV x) − y
  0.5 × '+ r̲/ r̲avel e × e
}
Used in: q4, ᵘv̲era

ˡg̲radsV : Num a => (a, a, a, a, a) -> (a, a) -> (a, a, a)

function · line 251

(W, A0, B0, d, b) g̲radsV (x, g): VeRA's gradients for the inputs x (n x N) and the output's gradient g (m x N), as (dd, db, dx). With z = A0 x and Q = B0 times g z' item by item (m x r): dd = b' Q (the research's (B0' (g b)) z, summed over the inputs), db = Q d (its g (B0 (d z))), dx = W' g + A0' (diag(b) B0 diag(d))' g. The projections are in every gradient and get none.

      "lo:" u̲se< "LoRA"
      (dd, db, dx) ← (2 2 r̲eshape 1.0 0.0 0.0 1.0, 1 2 r̲eshape 1.0 1.0, 2 1 r̲eshape 1.0 -1.0, 1 r̲eshape 0.5, 2.0 4.0) ˡᵒg̲radsV (2 1 r̲eshape 2.0 4.0, 2 1 r̲eshape 1.0 -1.0)
      dd c̲at db
36.0 3.0 3.0
ˡg̲radsV ← { (W, A0, B0, d, b) (x, g) →
  Q ← B0 × g '+ '× i̲nner o̲\ A0 '+ '× i̲nner x
  (b '+ '× i̲nner Q, Q '+ '× i̲nner d, ((o̲\ W) '+ '× i̲nner g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × b '× t̲able d) '+ '× i̲nner g)
}

ˡs̲gdV : Num a => a -> ((a, a, a, a, a), (a, a)) -> (a, a, a, a, a)

function · line 258

lr s̲gdV ((W, A0, B0, d, b), (x, y)): VeRA's step of gradient descent on the squared error: d and b move, the rest stays.

ˡs̲gdV ← { lr ((W, A0, B0, d, b), (x, y)) →
  (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y)
  (W, A0, B0, lr ᵒᵖs̲gd (d, dd), lr ᵒᵖs̲gd (b, db))
}

ˡa̲damStartV : (Any a, Any b, Any c, Num d) => (a, b, c, Float, Float) -> ((a, b, c, Float, Float), (Float, Float, Float, Float), d)

function · line 265

a̲damStartV (W, A0, B0, d, b): VeRA's Adam state: the adapter, the running averages of d and b at 0, the step 1.

ˡa̲damStartV ← { (W, A0, B0, d, b) → ((W, A0, B0, d, b), (0.0 × d, 0.0 × d, 0.0 × b, 0.0 × b), 1) }
Used in: q4, ᵘv̲era

ˡa̲damV : Float -> (((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float), (Float, Float)) -> ((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float)

function · line 269

lr a̲damV (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)): VeRA's Adam step: d and b move; the next state.

ˡa̲damV ← { lr (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)) →
  (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y)
  (d2, md2, vd2) ← (lr, t) ᵒᵖa̲dam (d, md, vd, dd)
  (b2, mb2, vb2) ← (lr, t) ᵒᵖa̲dam (b, mb, vb, db)
  ((W, A0, B0, d2, b2), (md2, vd2, mb2, vb2), t + 1)
}
Used in: q4, ᵘv̲era

QLoRA: the base in 4 bits

ˡq̲base : Int -> Float -> (Int, Float, Int)

function · line 285

k q̲base W: the base as QLoRA stores it: W in NF4, blocks of k (QLoRA's 64), as (codes, scales, shape): 4-bit codes, a scale per block, W's shape to come back to.

      "lo:" u̲se< "LoRA"
      (q, s, sh) ← 2 ˡᵒq̲base 2 2 r̲eshape 1.0 -4.0 2.0 0.5
      s c̲at f̲loat sh
4.0 2.0 2.0 2.0
ˡq̲base ← { k W →
  (q, s) ← k qz:n̲f4Codes W
  (q, s, s̲hape W)
}
Used in: ˡm̲ergeQ, Q, Wq, mm

ˡd̲equant : (Int, Float, Int) -> Float

function · line 298

d̲equant (q, s, shape): the stored base back to numbers, W~, the matrix QLoRA computes with: each code's level times its block's scale, in W's shape. It is not W: the difference is the 4-bit error.

      "lo:" u̲se< "LoRA"
      ˡᵒd̲equant 2 ˡᵒq̲base 2 2 r̲eshape 1.0 -4.0 2.0 0.5
0.9844492077827454               -4.0
               2.0 0.4922246038913727
ˡd̲equant ← { (q, s, sh) → sh r̲eshape qz:n̲f4Values (q, s) }

ˡf̲orwardQ : ((Int, Float, Int), Float, Float, Float) -> Float -> Float

function · line 304

(Q, A, B, s) f̲orwardQ x: QLoRA's adapted layer, W~ x + s B (A x): the 4-bit base dequantized when the layer runs, the adapter in full precision. Its gradients and steps are LoRA's with W~ for W: lr s̲gd ((d̲equant Q, A, B, s), (x, y)).

ˡf̲orwardQ ← { (Q, A, B, s) x → (ˡd̲equant Q, A, B, s) ˡf̲orward x }

ˡm̲ergeQ : Int -> ((Int, Float, Int), Float, Float, Float) -> (Int, Float, Int)

function · line 310

k m̲ergeQ (Q, A, B, s): the adapter merged into a 4-bit base: W~ + s B A quantized again (blocks of k), as a stored base. Not exact: d_equant of it is not W~ + s B A, since the merged numbers fall between NF4's levels.

ˡm̲ergeQ ← { k (Q, A, B, s) → k ˡq̲base (ˡd̲equant Q) + ˡd̲elta (A, B, s) }
Used in: Mq

ˡq̲bytes : (Num a, Num b) => (a, b) -> a -> Float

function · line 319

(r, k) q̲bytes m n: QLoRA's bytes for an m x n base at rank r, blocks of k: the base in NF4 (4 bits and a 32-bit scale per block), the adapter in BF16 (16 bits), Adam's two averages for it in FP32; against FP16's 2 m n for the base alone.

      "lo:" u̲se< "LoRA"
      (16, 64) ˡᵒq̲bytes 4096 14336
33030144.0 589824.0 2359296.0
ˡq̲bytes ← { (r, k) mn →
  n ← r ˡp̲arams mn
  ((f̲loat '× r̲/ mn) × (4.0 + 32.0 ÷ f̲loat k) ÷ 8.0) c̲at (16 ˡb̲ytes n) c̲at 2.0 × 32 ˡb̲ytes n
}

Rank

ʰs̲tep : Num a => Float -> (Float, Int, a) -> (Float, Int, a)

function (private) · line 329

One step of Gram-Schmidt on the rows: the k-th row's remainder R[k], if larger than tol, becomes a direction and is taken out of every row, and the count c goes up.

ʰs̲tep ← { tol (R, k, c) →
  v ← k s̲elect R
  n ← s̲qrt '+ r̲/ v × v
  n ≤ tol ? (R, k + 1, c)
  (R − (R '+ '× i̲nner v ÷ n) '× t̲able v ÷ n, k + 1, c + 1)
}
Used in: ˡr̲ank

ˡr̲ank : Num a => Float -> a

function · line 343

r̲ank M: the number of independent rows of the matrix M (its rank), by Gram-Schmidt: a row counts when what is left of it, after the directions of the rows before it are taken out, is larger than 1e-9 times the largest row.

      "lo:" u̲se< "LoRA"
      ˡᵒr̲ank 3 3 r̲eshape 1.0 2.0 3.0 2.0 4.0 6.0 0.0 1.0 1.0
2
ˡr̲ank ← { M →
  tol ← 0.000000001 × s̲qrt 'm̲ax r̲/ '+ r̲/₂ M × M
  (_, _, c) ← (f̲irst s̲hape M) '{ R → tol ʰs̲tep R } p̲ower (M, 1, 0)
  c
}

Counts

ˡp̲arams : Num a => a -> a -> a

function · line 357

r p̲arams m n: the numbers an adapter of rank r on an m x n matrix trains, r (m + n) (B's m r and A's r n), against the m n of the matrix itself.

      "lo:" u̲se< "LoRA"
      16 ˡᵒp̲arams 4096 14336
294912
ˡp̲arams ← { r mn → r × '+ r̲/ mn }

ˡe̲xpert : Num a => a -> a -> a

function · line 365

r e̲xpert d f: the numbers rank-r adapters train on one SwiGLU expert of hidden size d and inner size f: its three matrices (gate and up, f x d; down, d x f), 3 r (d + f).

      "lo:" u̲se< "LoRA"
      16 ˡᵒe̲xpert 4096 14336
884736
ˡe̲xpert ← { r df → 3 × r ˡp̲arams df }
Used in: ᵘe̲xpert, n, be

ˡe̲xpertFA : Num a => a -> a -> a

function · line 373

r e̲xpertFA d f: the same expert under LoRA-FA: the numbers it trains (the B's, r (2f + d)), then the frozen A's it regenerates (r (2d + f)); together r e_xpert d f.

      "lo:" u̲se< "LoRA"
      16 ˡᵒe̲xpertFA 4096 14336
524288 360448
ˡe̲xpertFA ← { r df →
  (d, f) ← (f̲irst df, 2 s̲elect df)
  (r × f + f + d) c̲at r × d + d + f
}
Used in: be

ˡb̲ytes : Num a => a -> a -> Float

function · line 383

b b̲ytes n: the bytes n numbers take at b bits each (16 for BF16 and FP16, 32 for FP32, 4 for NF4).

      "lo:" u̲se< "LoRA"
      16 ˡᵒb̲ytes 884736
1769472.0
ˡb̲ytes ← { b n → (f̲loat b × n) ÷ 8.0 }
Used in: ˡq̲bytes, mib, mib, be

ˡc̲osts : Num a => a -> a -> a

function · line 391

r c̲osts m n: the multiply-adds for one input through rank-r factors on an m x n matrix, B (A x) as r (m + n), then what making the matrix B A first costs, r m n.

      "lo:" u̲se< "LoRA"
      8 ˡᵒc̲osts 64 64
1024 32768
ˡc̲osts ← { r mn → (r ˡp̲arams mn) c̲at r × '× r̲/ mn }

ˡp̲aramsV : Num a => a -> a -> a

function · line 400

r p̲aramsV m n: the numbers VeRA trains on an m x n matrix at rank r, m + r (b's m and d's r), against LoRA's r (m + n); its frozen projections, r (m + n) numbers, are shared by every matrix of the shape (or regenerated from their seed).

      "lo:" u̲se< "LoRA"
      16 ˡᵒp̲aramsV 14336 4096
14352
ˡp̲aramsV ← { r mn → r + f̲irst mn }

ˡe̲xpertV : Num a => a -> a -> a

function · line 410

r e̲xpertV d f: the numbers VeRA trains on one SwiGLU expert of hidden size d and inner size f, its three matrices at rank r: m + r each, 2 f + d + 3 r. The projections are frozen and shared by every expert: two pairs (one for gate and up, f x d; one for down, d x f), 2 r (d + f) numbers, once.

      "lo:" u̲se< "LoRA"
      16 ˡᵒe̲xpertV 4096 14336
32816
ˡe̲xpertV ← { r df → (3 × r) + (2 × 2 s̲elect df) + f̲irst df }