sourcelibs/LoRA/src/LoRA.xtl

1⍝│ LoRA: low-rank adapters: the adapted forward pass in factored form, the merge, the counts, the gradients and steps, LoRA-FA, QLoRA, VeRA 2⍝│ 3⍝│ Import it with an alias of your choice: "lo:" u_se< "LoRA". 4⍝│ Put libs/LoRA/src on XETAL_PATH ("just path"); the reference is libs/LoRA/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ A frozen weight matrix W (m x n: m outputs, n inputs) is adapted by 8⍝│ a low-rank update s B A: A is r x n, B is m x r, the rank r much 9⍝│ smaller than m and n, s the scale alpha / r. An adapter is the tuple 10⍝│ (W, A, B, s). The inputs are columns: x is one input of n numbers or 11⍝│ an n x N matrix of N inputs, and W x is W '+ '× i̲nner x. The 12⍝│ forward pass keeps the factors apart, B (A x), as training does: 13⍝│ the merged matrix W + s B A is made only to deploy it. 14⍝│ 15⍝│ Training moves A and B by the gradients of the squared error 16⍝│ (g_rads, s_gd, a_dam through the Optim library); LoRA-FA freezes A 17⍝│ (regenerated from a seed, s_eeded) and trains B alone; QLoRA keeps W 18⍝│ in 4 bits (NF4, through the Quant library) and computes with it 19⍝│ dequantized; VeRA freezes A and B too (random, from a seed, shared by 20⍝│ every matrix of the shape) and trains two vectors, d and b. 21 22"op:" u̲se< "Optim" 23"qz:" u̲se< "Quant" 24 25⍝━━ The forward pass 26 27⍝│ alpha s̲caling r: the adapter's scale s, alpha / r (so that a 28⍝│ change of rank keeps the update's size). 29⍝│ >> "lo:" u_se< "LoRA" 30⍝│ >> 16.0 lo:s_caling 8 31⍝│ 2.0 32ˡs̲caling ← { alpha r → alpha ÷ f̲loat r } 33 34⍝│ alpha s̲calingRs r: rsLoRA's rank-stabilized scale, alpha / sqrt r 35⍝│ (Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning 36⍝│ with LoRA", 2023): with alpha / r the update's learning slows as r 37⍝│ grows; over sqrt r it does not. An adapter takes either, (W, A, B, s). 38⍝│ >> "lo:" u_se< "LoRA" 39⍝│ >> 16.0 lo:s_calingRs 4 40⍝│ 8.0 41ˡs̲calingRs ← { alpha r → alpha ÷ s̲qrt f̲loat r } 42 43⍝│ (W, A, B, s) f̲orward x: the adapted layer, W x + s B (A x), the 44⍝│ update in its factored form: A x (r numbers per input), then B of 45⍝│ that, never the m x n matrix B A. 46⍝│ >> "lo:" u_se< "LoRA" 47⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:f_orward 2.0 4.0 48⍝│ 5.0 1.0 49ˡf̲orward ← { (W, A, B, s) x → (W '+ '× i̲nner x) + s × B '+ '× i̲nner A '+ '× i̲nner x } 50 51⍝━━ The merge 52 53⍝│ d̲elta (A, B, s): the update itself, s B A, an m x n matrix of 54⍝│ rank r at most. 55⍝│ >> "lo:" u_se< "LoRA" 56⍝│ >> lo:d_elta (1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) 57⍝│ 0.5 0.5 58⍝│ -0.5 -0.5 59ˡd̲elta ← { (A, B, s) → s × B '+ '× i̲nner A } 60 61⍝│ m̲erge (W, A, B, s): the adapter folded into the weights, W + s B 62⍝│ A: one matrix again, as fast as the base layer, for one task. 63⍝│ >> "lo:" u_se< "LoRA" 64⍝│ >> lo:m_erge (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) 65⍝│ 1.5 0.5 66⍝│ -0.5 0.5 67ˡm̲erge ← { (W, A, B, s) → W + ˡd̲elta (A, B, s) } 68 69⍝━━ Gradients 70 71⍝│ (W, A, B, s) l̲oss (x, y): the squared error of the adapted layer 72⍝│ on the inputs x (columns) against the wanted outputs y, half the sum 73⍝│ of the squared differences, so that its gradient at the output, g, 74⍝│ is the difference itself. 75⍝│ >> "lo:" u_se< "LoRA" 76⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:l_oss (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0) 77⍝│ 1.0 78ˡl̲oss ← { a (x, y) → 79 e ← (a ˡf̲orward x) − y 80 0.5 × '+ r̲/ r̲avel e × e 81} 82 83⍝│ (W, A, B, s) g̲rads (x, g): the gradients for the inputs x (an n x 84⍝│ N matrix, one input per column) and g, the loss's gradient at the 85⍝│ output (m x N): with z = A x, dL/dA = s (B' g) x', dL/dB = s g z' and 86⍝│ dL/dx = W' g + s A' (B' g), as the tuple (dA, dB, dx). B' g is made 87⍝│ once and used twice; no m x n matrix but W's own is made. 88⍝│ >> "lo:" u_se< "LoRA" 89⍝│ >> (dA, dB, dx) := (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:g_rads (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 1.0 -1.0) 90⍝│ >> dB 91⍝│ 3.0 92⍝│ -3.0 93ˡg̲rads ← { (W, A, B, s) (x, g) → 94 b ← (o̲\ B) '+ '× i̲nner g 95 (s × b '+ '× i̲nner o̲\ x, s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner b) 96} 97 98⍝━━ Training steps 99 100⍝│ lr s̲gd ((W, A, B, s), (x, y)): one step of gradient descent on 101⍝│ the squared error (l_oss) at the rate lr, A and B moved by Optim's 102⍝│ s_gd, W and s left as they are: the adapter after the step. p_ower 103⍝│ repeats it. 104⍝│ >> "lo:" u_se< "LoRA" 105⍝│ >> (_, A, B, _) := 0.1 lo:s_gd ((2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5), (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0)) 106⍝│ >> B 107⍝│ 0.7 108⍝│ -0.7 109ˡs̲gd ← { lr ((W, A, B, s), (x, y)) → 110 (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y) 111 (W, lr ᵒᵖs̲gd (A, dA), lr ᵒᵖs̲gd (B, dB), s) 112} 113 114⍝│ a̲damStart (W, A, B, s): what Adam starts from: the adapter, its 115⍝│ four running averages at 0 (A's two, then B's) and the step count 1. 116ˡa̲damStart ← { (W, A, B, s) → ((W, A, B, s), (0.0 × A, 0.0 × A, 0.0 × B, 0.0 × B), 1) } 117 118⍝│ lr a̲dam (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)): Adam's step 119⍝│ t on the squared error, A and B moved by Optim's a_dam at the rate 120⍝│ lr; the next state, the step count one more. 121⍝│ >> "lo:" u_se< "LoRA" 122⍝│ >> ((_, _, B, _), _, _) := 0.1 lo:a_dam ((lo:a_damStart (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5)), (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0)) 123⍝│ >> B 124⍝│ 0.9000000003333333 125⍝│ -0.9000000003333333 126ˡa̲dam ← { lr (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)) → 127 (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y) 128 (A2, mA2, vA2) ← (lr, t) ᵒᵖa̲dam (A, mA, vA, dA) 129 (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB) 130 ((W, A2, B2, s), (mA2, vA2, mB2, vB2), t + 1) 131} 132 133⍝━━ LoRA-FA: A frozen 134 135⍝│ seed s̲eeded r n: an r x n matrix regenerated from a whole-number 136⍝│ seed (LoRA-FA's frozen A, which a checkpoint then need not hold): 137⍝│ sin(seed k k) for k = 1 .. r n, over the square root of n (the 138⍝│ inputs' count, so A x keeps x's size). No chance: the same seed 139⍝│ gives the same matrix everywhere. 140⍝│ >> "lo:" u_se< "LoRA" 141⍝│ >> 7 lo:s_eeded 2 4 142⍝│ 0.32849329935939453 0.13545289415393452 0.08367785015140346 -0.44499780218341667 143⍝│ -0.4005672975890204 0.3115061055018265 -0.26829917759428185 0.47413628607807934 144ˡs̲eeded ← { seed rn → 145 k ← f̲loat r̲ange '× r̲/ rn 146 rn r̲eshape (s̲in (f̲loat seed) × k × k) ÷ s̲qrt f̲loat 2 s̲elect rn 147} 148 149⍝│ (W, A, B, s) g̲radsFA (x, g): LoRA-FA's gradients, (dB, dx): A is 150⍝│ frozen, so its gradient is skipped, but x's still flows back through 151⍝│ A, dx = W' g + s A' (B' g). 152ˡg̲radsFA ← { (W, A, B, s) (x, g) → 153 (s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner (o̲\ B) '+ '× i̲nner g) 154} 155 156⍝│ lr s̲gdFA ((W, A, B, s), (x, y)): LoRA-FA's step of gradient 157⍝│ descent: B alone moves. Every update s B A it reaches has its rows 158⍝│ in A's row space. 159ˡs̲gdFA ← { lr ((W, A, B, s), (x, y)) → 160 (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y) 161 (W, A, lr ᵒᵖs̲gd (B, dB), s) 162} 163 164⍝│ a̲damStartFA (W, A, B, s): LoRA-FA's Adam state: the adapter, B's 165⍝│ two running averages at 0 (A has none: half of LoRA's optimizer 166⍝│ memory on a square layer), the step 1. 167ˡa̲damStartFA ← { (W, A, B, s) → ((W, A, B, s), (0.0 × B, 0.0 × B), 1) } 168 169⍝│ lr a̲damFA (((W, A, B, s), (mB, vB), t), (x, y)): LoRA-FA's Adam 170⍝│ step: B alone moves; the next state. 171ˡa̲damFA ← { lr (((W, A, B, s), (mB, vB), t), (x, y)) → 172 (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y) 173 (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB) 174 ((W, A, B2, s), (mB2, vB2), t + 1) 175} 176 177⍝━━ VeRA: shared random projections, two vectors trained 178 179⍝│ seed p̲rojections m n r: VeRA's frozen projections for an m x n 180⍝│ matrix at rank r, (A0, B0): A0 (r x n) and B0 (m x r) regenerated 181⍝│ from the seed as s_eeded makes them (B0 from seed + 1), so a 182⍝│ checkpoint need not hold them and every layer or expert of the same 183⍝│ shape can share them. 184⍝│ >> "lo:" u_se< "LoRA" 185⍝│ >> (A0, B0) := 7 lo:p_rojections 3 4 2 186⍝│ >> s_hape B0 187⍝│ 3 2 188ˡp̲rojections ← { seed mnr → 189 (m, n, r) ← (f̲irst mnr, 2 s̲elect mnr, 3 s̲elect mnr) 190 (seed ˡs̲eeded r c̲at n, (seed + 1) ˡs̲eeded m c̲at r) 191} 192 193⍝│ seed v̲era (W, r): a VeRA adapter on the frozen W (m x n) at rank 194⍝│ r, as the paper starts it (Kopiczko, Blankevoort and Asano, "VeRA: 195⍝│ Vector-based Random Matrix Adaptation", 2024): (W, A0, B0, d, b), 196⍝│ the projections from the seed, d (r numbers) at 0.1 and b (m 197⍝│ numbers) at 0, so the adapted layer starts as the base. d and b are 198⍝│ all it trains: m + r numbers. 199⍝│ >> "lo:" u_se< "LoRA" 200⍝│ >> (_, _, _, d, b) := 7 lo:v_era (2 3 r_eshape 1.0, 4) 201⍝│ >> d c_at b 202⍝│ 0.1 0.1 0.1 0.1 0.0 0.0 203ˡv̲era ← { seed (W, r) → 204 (A0, B0) ← seed ˡp̲rojections (s̲hape W) c̲at r 205 (W, A0, B0, r r̲eshape 0.1, (f̲irst s̲hape W) r̲eshape 0.0) 206} 207 208⍝│ (A0, B0, d, b) d̲eltaV: VeRA's update, diag(b) B0 diag(d) A0, an 209⍝│ m x n matrix of rank r at most. The two diagonals need no matrix and 210⍝│ no broadcasting: diag(b) B0 diag(d) is B0 times the table of b by d, 211⍝│ item by item (an m x r matrix, B's size). 212⍝│ >> "lo:" u_se< "LoRA" 213⍝│ >> lo:d_eltaV (1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) 214⍝│ 1.0 1.0 215⍝│ -2.0 -2.0 216ˡd̲eltaV ← { (A0, B0, d, b) → (B0 × b '× t̲able d) '+ '× i̲nner A0 } 217 218⍝│ (W, A0, B0, d, b) f̲orwardV x: the VeRA layer, W x + b (B0 (d (A0 219⍝│ x))), the vectors scaling A0 x's rows and B0's output's: in factored 220⍝│ form, A0 x first, then the m x r matrix diag(b) B0 diag(d) of that. 221⍝│ >> "lo:" u_se< "LoRA" 222⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) lo:f_orwardV 2.0 4.0 223⍝│ 8.0 -8.0 224ˡf̲orwardV ← { (W, A0, B0, d, b) x → (W '+ '× i̲nner x) + (B0 × b '× t̲able d) '+ '× i̲nner A0 '+ '× i̲nner x } 225 226⍝│ m̲ergeV (W, A0, B0, d, b): the VeRA adapter folded into the 227⍝│ weights, W + diag(b) B0 diag(d) A0. 228⍝│ >> "lo:" u_se< "LoRA" 229⍝│ >> lo:m_ergeV (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) 230⍝│ 2.0 1.0 231⍝│ -2.0 -1.0 232ˡm̲ergeV ← { (W, A0, B0, d, b) → W + ˡd̲eltaV (A0, B0, d, b) } 233 234⍝│ (W, A0, B0, d, b) l̲ossV (x, y): the squared error of the VeRA 235⍝│ layer, as l_oss. 236ˡl̲ossV ← { a (x, y) → 237 e ← (a ˡf̲orwardV x) − y 238 0.5 × '+ r̲/ r̲avel e × e 239} 240 241⍝│ (W, A0, B0, d, b) g̲radsV (x, g): VeRA's gradients for the inputs 242⍝│ x (n x N) and the output's gradient g (m x N), as (dd, db, dx). With 243⍝│ z = A0 x and Q = B0 times g z' item by item (m x r): dd = b' Q (the 244⍝│ research's (B0' (g b)) z, summed over the inputs), db = Q d (its g 245⍝│ (B0 (d z))), dx = W' g + A0' (diag(b) B0 diag(d))' g. The projections 246⍝│ are in every gradient and get none. 247⍝│ >> "lo:" u_se< "LoRA" 248⍝│ >> (dd, db, dx) := (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) lo:g_radsV (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 1.0 -1.0) 249⍝│ >> dd c_at db 250⍝│ 36.0 3.0 3.0 251ˡg̲radsV ← { (W, A0, B0, d, b) (x, g) → 252 Q ← B0 × g '+ '× i̲nner o̲\ A0 '+ '× i̲nner x 253 (b '+ '× i̲nner Q, Q '+ '× i̲nner d, ((o̲\ W) '+ '× i̲nner g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × b '× t̲able d) '+ '× i̲nner g) 254} 255 256⍝│ lr s̲gdV ((W, A0, B0, d, b), (x, y)): VeRA's step of gradient 257⍝│ descent on the squared error: d and b move, the rest stays. 258ˡs̲gdV ← { lr ((W, A0, B0, d, b), (x, y)) → 259 (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y) 260 (W, A0, B0, lr ᵒᵖs̲gd (d, dd), lr ᵒᵖs̲gd (b, db)) 261} 262 263⍝│ a̲damStartV (W, A0, B0, d, b): VeRA's Adam state: the adapter, the 264⍝│ running averages of d and b at 0, the step 1. 265ˡa̲damStartV ← { (W, A0, B0, d, b) → ((W, A0, B0, d, b), (0.0 × d, 0.0 × d, 0.0 × b, 0.0 × b), 1) } 266 267⍝│ lr a̲damV (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)): 268⍝│ VeRA's Adam step: d and b move; the next state. 269ˡa̲damV ← { lr (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)) → 270 (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y) 271 (d2, md2, vd2) ← (lr, t) ᵒᵖa̲dam (d, md, vd, dd) 272 (b2, mb2, vb2) ← (lr, t) ᵒᵖa̲dam (b, mb, vb, db) 273 ((W, A0, B0, d2, b2), (md2, vd2, mb2, vb2), t + 1) 274} 275 276⍝━━ QLoRA: the base in 4 bits 277 278⍝│ k q̲base W: the base as QLoRA stores it: W in NF4, blocks of k 279⍝│ (QLoRA's 64), as (codes, scales, shape): 4-bit codes, a scale per 280⍝│ block, W's shape to come back to. 281⍝│ >> "lo:" u_se< "LoRA" 282⍝│ >> (q, s, sh) := 2 lo:q_base 2 2 r_eshape 1.0 -4.0 2.0 0.5 283⍝│ >> s c_at f_loat sh 284⍝│ 4.0 2.0 2.0 2.0 285ˡq̲base ← { k W → 286 (q, s) ← k qz:n̲f4Codes W 287 (q, s, s̲hape W) 288} 289 290⍝│ d̲equant (q, s, shape): the stored base back to numbers, W~, the 291⍝│ matrix QLoRA computes with: each code's level times its block's 292⍝│ scale, in W's shape. It is not W: the difference is the 4-bit 293⍝│ error. 294⍝│ >> "lo:" u_se< "LoRA" 295⍝│ >> lo:d_equant 2 lo:q_base 2 2 r_eshape 1.0 -4.0 2.0 0.5 296⍝│ 0.9844492077827454 -4.0 297⍝│ 2.0 0.4922246038913727 298ˡd̲equant ← { (q, s, sh) → sh r̲eshape qz:n̲f4Values (q, s) } 299 300⍝│ (Q, A, B, s) f̲orwardQ x: QLoRA's adapted layer, W~ x + s B (A x): 301⍝│ the 4-bit base dequantized when the layer runs, the adapter in full 302⍝│ precision. Its gradients and steps are LoRA's with W~ for W: 303⍝│ lr s̲gd ((d̲equant Q, A, B, s), (x, y)). 304ˡf̲orwardQ ← { (Q, A, B, s) x → (ˡd̲equant Q, A, B, s) ˡf̲orward x } 305 306⍝│ k m̲ergeQ (Q, A, B, s): the adapter merged into a 4-bit base: W~ + 307⍝│ s B A quantized again (blocks of k), as a stored base. Not exact: 308⍝│ d_equant of it is not W~ + s B A, since the merged numbers fall 309⍝│ between NF4's levels. 310ˡm̲ergeQ ← { k (Q, A, B, s) → k ˡq̲base (ˡd̲equant Q) + ˡd̲elta (A, B, s) } 311 312⍝│ (r, k) q̲bytes m n: QLoRA's bytes for an m x n base at rank r, 313⍝│ blocks of k: the base in NF4 (4 bits and a 32-bit scale per block), 314⍝│ the adapter in BF16 (16 bits), Adam's two averages for it in FP32; 315⍝│ against FP16's 2 m n for the base alone. 316⍝│ >> "lo:" u_se< "LoRA" 317⍝│ >> (16, 64) lo:q_bytes 4096 14336 318⍝│ 33030144.0 589824.0 2359296.0 319ˡq̲bytes ← { (r, k) mn → 320 n ← r ˡp̲arams mn 321 ((f̲loat '× r̲/ mn) × (4.0 + 32.0 ÷ f̲loat k) ÷ 8.0) c̲at (16 ˡb̲ytes n) c̲at 2.0 × 32 ˡb̲ytes n 322} 323 324⍝━━ Rank 325 326⍝│ One step of Gram-Schmidt on the rows: the k-th row's remainder R[k], 327⍝│ if larger than tol, becomes a direction and is taken out of every 328⍝│ row, and the count c goes up. 329ʰs̲tep ← { tol (R, k, c) → 330 v ← k s̲elect R 331 n ← s̲qrt '+ r̲/ v × v 332 n ≤ tol ? (R, k + 1, c) 333 (R − (R '+ '× i̲nner v ÷ n) '× t̲able v ÷ n, k + 1, c + 1) 334} 335 336⍝│ r̲ank M: the number of independent rows of the matrix M (its 337⍝│ rank), by Gram-Schmidt: a row counts when what is left of it, after 338⍝│ the directions of the rows before it are taken out, is larger than 339⍝│ 1e-9 times the largest row. 340⍝│ >> "lo:" u_se< "LoRA" 341⍝│ >> lo:r_ank 3 3 r_eshape 1.0 2.0 3.0 2.0 4.0 6.0 0.0 1.0 1.0 342⍝│ 2 343ˡr̲ank ← { M → 344 tol ← 0.000000001 × s̲qrt 'm̲ax r̲/ '+ r̲/₂ M × M 345 (_, _, c) ← (f̲irst s̲hape M) '{ R → tol ʰs̲tep R } p̲ower (M, 1, 0) 346 c 347} 348 349⍝━━ Counts 350 351⍝│ r p̲arams m n: the numbers an adapter of rank r on an m x n matrix 352⍝│ trains, r (m + n) (B's m r and A's r n), against the m n of the 353⍝│ matrix itself. 354⍝│ >> "lo:" u_se< "LoRA" 355⍝│ >> 16 lo:p_arams 4096 14336 356⍝│ 294912 357ˡp̲arams ← { r mn → r × '+ r̲/ mn } 358 359⍝│ r e̲xpert d f: the numbers rank-r adapters train on one SwiGLU 360⍝│ expert of hidden size d and inner size f: its three matrices (gate 361⍝│ and up, f x d; down, d x f), 3 r (d + f). 362⍝│ >> "lo:" u_se< "LoRA" 363⍝│ >> 16 lo:e_xpert 4096 14336 364⍝│ 884736 365ˡe̲xpert ← { r df → 3 × r ˡp̲arams df } 366 367⍝│ r e̲xpertFA d f: the same expert under LoRA-FA: the numbers it 368⍝│ trains (the B's, r (2f + d)), then the frozen A's it regenerates 369⍝│ (r (2d + f)); together r e_xpert d f. 370⍝│ >> "lo:" u_se< "LoRA" 371⍝│ >> 16 lo:e_xpertFA 4096 14336 372⍝│ 524288 360448 373ˡe̲xpertFA ← { r df → 374 (d, f) ← (f̲irst df, 2 s̲elect df) 375 (r × f + f + d) c̲at r × d + d + f 376} 377 378⍝│ b b̲ytes n: the bytes n numbers take at b bits each (16 for BF16 379⍝│ and FP16, 32 for FP32, 4 for NF4). 380⍝│ >> "lo:" u_se< "LoRA" 381⍝│ >> 16 lo:b_ytes 884736 382⍝│ 1769472.0 383ˡb̲ytes ← { b n → (f̲loat b × n) ÷ 8.0 } 384 385⍝│ r c̲osts m n: the multiply-adds for one input through rank-r 386⍝│ factors on an m x n matrix, B (A x) as r (m + n), then what making 387⍝│ the matrix B A first costs, r m n. 388⍝│ >> "lo:" u_se< "LoRA" 389⍝│ >> 8 lo:c_osts 64 64 390⍝│ 1024 32768 391ˡc̲osts ← { r mn → (r ˡp̲arams mn) c̲at r × '× r̲/ mn } 392 393⍝│ r p̲aramsV m n: the numbers VeRA trains on an m x n matrix at rank 394⍝│ r, m + r (b's m and d's r), against LoRA's r (m + n); its frozen 395⍝│ projections, r (m + n) numbers, are shared by every matrix of the 396⍝│ shape (or regenerated from their seed). 397⍝│ >> "lo:" u_se< "LoRA" 398⍝│ >> 16 lo:p_aramsV 14336 4096 399⍝│ 14352 400ˡp̲aramsV ← { r mn → r + f̲irst mn } 401 402⍝│ r e̲xpertV d f: the numbers VeRA trains on one SwiGLU expert of 403⍝│ hidden size d and inner size f, its three matrices at rank r: m + r 404⍝│ each, 2 f + d + 3 r. The projections are frozen and shared by every 405⍝│ expert: two pairs (one for gate and up, f x d; one for down, d x f), 406⍝│ 2 r (d + f) numbers, once. 407⍝│ >> "lo:" u_se< "LoRA" 408⍝│ >> 16 lo:e_xpertV 4096 14336 409⍝│ 32816 410ˡe̲xpertV ← { r df → (3 × r) + (2 × 2 s̲elect df) + f̲irst df }