sourcelibs/LoRA/src/LoRA.xtl
1⍝│ LoRA: low-rank adapters: the adapted forward pass in factored form, the merge, the counts, the gradients and steps, LoRA-FA, QLoRA, VeRA
2⍝│
3⍝│ Import it with an alias of your choice: "lo:" u_se< "LoRA".
4⍝│ Put libs/LoRA/src on XETAL_PATH ("just path"); the reference is libs/LoRA/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ A frozen weight matrix W (m x n: m outputs, n inputs) is adapted by
8⍝│ a low-rank update s B A: A is r x n, B is m x r, the rank r much
9⍝│ smaller than m and n, s the scale alpha / r. An adapter is the tuple
10⍝│ (W, A, B, s). The inputs are columns: x is one input of n numbers or
11⍝│ an n x N matrix of N inputs, and W x is W '+ '× i̲nner x. The
12⍝│ forward pass keeps the factors apart, B (A x), as training does:
13⍝│ the merged matrix W + s B A is made only to deploy it.
14⍝│
15⍝│ Training moves A and B by the gradients of the squared error
16⍝│ (g_rads, s_gd, a_dam through the Optim library); LoRA-FA freezes A
17⍝│ (regenerated from a seed, s_eeded) and trains B alone; QLoRA keeps W
18⍝│ in 4 bits (NF4, through the Quant library) and computes with it
19⍝│ dequantized; VeRA freezes A and B too (random, from a seed, shared by
20⍝│ every matrix of the shape) and trains two vectors, d and b.
21
22"op:" u̲se< "Optim"
23"qz:" u̲se< "Quant"
24
25⍝━━ The forward pass
26
27⍝│ alpha s̲caling r: the adapter's scale s, alpha / r (so that a
28⍝│ change of rank keeps the update's size).
29⍝│ >> "lo:" u_se< "LoRA"
30⍝│ >> 16.0 lo:s_caling 8
31⍝│ 2.0
32ˡs̲caling ← { alpha r → alpha ÷ f̲loat r }
33
34⍝│ alpha s̲calingRs r: rsLoRA's rank-stabilized scale, alpha / sqrt r
35⍝│ (Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning
36⍝│ with LoRA", 2023): with alpha / r the update's learning slows as r
37⍝│ grows; over sqrt r it does not. An adapter takes either, (W, A, B, s).
38⍝│ >> "lo:" u_se< "LoRA"
39⍝│ >> 16.0 lo:s_calingRs 4
40⍝│ 8.0
41ˡs̲calingRs ← { alpha r → alpha ÷ s̲qrt f̲loat r }
42
43⍝│ (W, A, B, s) f̲orward x: the adapted layer, W x + s B (A x), the
44⍝│ update in its factored form: A x (r numbers per input), then B of
45⍝│ that, never the m x n matrix B A.
46⍝│ >> "lo:" u_se< "LoRA"
47⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:f_orward 2.0 4.0
48⍝│ 5.0 1.0
49ˡf̲orward ← { (W, A, B, s) x → (W '+ '× i̲nner x) + s × B '+ '× i̲nner A '+ '× i̲nner x }
50
51⍝━━ The merge
52
53⍝│ d̲elta (A, B, s): the update itself, s B A, an m x n matrix of
54⍝│ rank r at most.
55⍝│ >> "lo:" u_se< "LoRA"
56⍝│ >> lo:d_elta (1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5)
57⍝│ 0.5 0.5
58⍝│ -0.5 -0.5
59ˡd̲elta ← { (A, B, s) → s × B '+ '× i̲nner A }
60
61⍝│ m̲erge (W, A, B, s): the adapter folded into the weights, W + s B
62⍝│ A: one matrix again, as fast as the base layer, for one task.
63⍝│ >> "lo:" u_se< "LoRA"
64⍝│ >> lo:m_erge (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5)
65⍝│ 1.5 0.5
66⍝│ -0.5 0.5
67ˡm̲erge ← { (W, A, B, s) → W + ˡd̲elta (A, B, s) }
68
69⍝━━ Gradients
70
71⍝│ (W, A, B, s) l̲oss (x, y): the squared error of the adapted layer
72⍝│ on the inputs x (columns) against the wanted outputs y, half the sum
73⍝│ of the squared differences, so that its gradient at the output, g,
74⍝│ is the difference itself.
75⍝│ >> "lo:" u_se< "LoRA"
76⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:l_oss (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0)
77⍝│ 1.0
78ˡl̲oss ← { a (x, y) →
79 e ← (a ˡf̲orward x) − y
80 0.5 × '+ r̲/ r̲avel e × e
81}
82
83⍝│ (W, A, B, s) g̲rads (x, g): the gradients for the inputs x (an n x
84⍝│ N matrix, one input per column) and g, the loss's gradient at the
85⍝│ output (m x N): with z = A x, dL/dA = s (B' g) x', dL/dB = s g z' and
86⍝│ dL/dx = W' g + s A' (B' g), as the tuple (dA, dB, dx). B' g is made
87⍝│ once and used twice; no m x n matrix but W's own is made.
88⍝│ >> "lo:" u_se< "LoRA"
89⍝│ >> (dA, dB, dx) := (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5) lo:g_rads (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 1.0 -1.0)
90⍝│ >> dB
91⍝│ 3.0
92⍝│ -3.0
93ˡg̲rads ← { (W, A, B, s) (x, g) →
94 b ← (o̲\ B) '+ '× i̲nner g
95 (s × b '+ '× i̲nner o̲\ x, s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner b)
96}
97
98⍝━━ Training steps
99
100⍝│ lr s̲gd ((W, A, B, s), (x, y)): one step of gradient descent on
101⍝│ the squared error (l_oss) at the rate lr, A and B moved by Optim's
102⍝│ s_gd, W and s left as they are: the adapter after the step. p_ower
103⍝│ repeats it.
104⍝│ >> "lo:" u_se< "LoRA"
105⍝│ >> (_, A, B, _) := 0.1 lo:s_gd ((2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5), (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0))
106⍝│ >> B
107⍝│ 0.7
108⍝│ -0.7
109ˡs̲gd ← { lr ((W, A, B, s), (x, y)) →
110 (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y)
111 (W, lr ᵒᵖs̲gd (A, dA), lr ᵒᵖs̲gd (B, dB), s)
112}
113
114⍝│ a̲damStart (W, A, B, s): what Adam starts from: the adapter, its
115⍝│ four running averages at 0 (A's two, then B's) and the step count 1.
116ˡa̲damStart ← { (W, A, B, s) → ((W, A, B, s), (0.0 × A, 0.0 × A, 0.0 × B, 0.0 × B), 1) }
117
118⍝│ lr a̲dam (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)): Adam's step
119⍝│ t on the squared error, A and B moved by Optim's a_dam at the rate
120⍝│ lr; the next state, the step count one more.
121⍝│ >> "lo:" u_se< "LoRA"
122⍝│ >> ((_, _, B, _), _, _) := 0.1 lo:a_dam ((lo:a_damStart (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 0.5)), (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 4.0 2.0))
123⍝│ >> B
124⍝│ 0.9000000003333333
125⍝│ -0.9000000003333333
126ˡa̲dam ← { lr (((W, A, B, s), (mA, vA, mB, vB), t), (x, y)) →
127 (dA, dB, _) ← (W, A, B, s) ˡg̲rads (x, ((W, A, B, s) ˡf̲orward x) − y)
128 (A2, mA2, vA2) ← (lr, t) ᵒᵖa̲dam (A, mA, vA, dA)
129 (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB)
130 ((W, A2, B2, s), (mA2, vA2, mB2, vB2), t + 1)
131}
132
133⍝━━ LoRA-FA: A frozen
134
135⍝│ seed s̲eeded r n: an r x n matrix regenerated from a whole-number
136⍝│ seed (LoRA-FA's frozen A, which a checkpoint then need not hold):
137⍝│ sin(seed k k) for k = 1 .. r n, over the square root of n (the
138⍝│ inputs' count, so A x keeps x's size). No chance: the same seed
139⍝│ gives the same matrix everywhere.
140⍝│ >> "lo:" u_se< "LoRA"
141⍝│ >> 7 lo:s_eeded 2 4
142⍝│ 0.32849329935939453 0.13545289415393452 0.08367785015140346 -0.44499780218341667
143⍝│ -0.4005672975890204 0.3115061055018265 -0.26829917759428185 0.47413628607807934
144ˡs̲eeded ← { seed rn →
145 k ← f̲loat r̲ange '× r̲/ rn
146 rn r̲eshape (s̲in (f̲loat seed) × k × k) ÷ s̲qrt f̲loat 2 s̲elect rn
147}
148
149⍝│ (W, A, B, s) g̲radsFA (x, g): LoRA-FA's gradients, (dB, dx): A is
150⍝│ frozen, so its gradient is skipped, but x's still flows back through
151⍝│ A, dx = W' g + s A' (B' g).
152ˡg̲radsFA ← { (W, A, B, s) (x, g) →
153 (s × g '+ '× i̲nner o̲\ A '+ '× i̲nner x, ((o̲\ W) '+ '× i̲nner g) + s × (o̲\ A) '+ '× i̲nner (o̲\ B) '+ '× i̲nner g)
154}
155
156⍝│ lr s̲gdFA ((W, A, B, s), (x, y)): LoRA-FA's step of gradient
157⍝│ descent: B alone moves. Every update s B A it reaches has its rows
158⍝│ in A's row space.
159ˡs̲gdFA ← { lr ((W, A, B, s), (x, y)) →
160 (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y)
161 (W, A, lr ᵒᵖs̲gd (B, dB), s)
162}
163
164⍝│ a̲damStartFA (W, A, B, s): LoRA-FA's Adam state: the adapter, B's
165⍝│ two running averages at 0 (A has none: half of LoRA's optimizer
166⍝│ memory on a square layer), the step 1.
167ˡa̲damStartFA ← { (W, A, B, s) → ((W, A, B, s), (0.0 × B, 0.0 × B), 1) }
168
169⍝│ lr a̲damFA (((W, A, B, s), (mB, vB), t), (x, y)): LoRA-FA's Adam
170⍝│ step: B alone moves; the next state.
171ˡa̲damFA ← { lr (((W, A, B, s), (mB, vB), t), (x, y)) →
172 (dB, _) ← (W, A, B, s) ˡg̲radsFA (x, ((W, A, B, s) ˡf̲orward x) − y)
173 (B2, mB2, vB2) ← (lr, t) ᵒᵖa̲dam (B, mB, vB, dB)
174 ((W, A, B2, s), (mB2, vB2), t + 1)
175}
176
177⍝━━ VeRA: shared random projections, two vectors trained
178
179⍝│ seed p̲rojections m n r: VeRA's frozen projections for an m x n
180⍝│ matrix at rank r, (A0, B0): A0 (r x n) and B0 (m x r) regenerated
181⍝│ from the seed as s_eeded makes them (B0 from seed + 1), so a
182⍝│ checkpoint need not hold them and every layer or expert of the same
183⍝│ shape can share them.
184⍝│ >> "lo:" u_se< "LoRA"
185⍝│ >> (A0, B0) := 7 lo:p_rojections 3 4 2
186⍝│ >> s_hape B0
187⍝│ 3 2
188ˡp̲rojections ← { seed mnr →
189 (m, n, r) ← (f̲irst mnr, 2 s̲elect mnr, 3 s̲elect mnr)
190 (seed ˡs̲eeded r c̲at n, (seed + 1) ˡs̲eeded m c̲at r)
191}
192
193⍝│ seed v̲era (W, r): a VeRA adapter on the frozen W (m x n) at rank
194⍝│ r, as the paper starts it (Kopiczko, Blankevoort and Asano, "VeRA:
195⍝│ Vector-based Random Matrix Adaptation", 2024): (W, A0, B0, d, b),
196⍝│ the projections from the seed, d (r numbers) at 0.1 and b (m
197⍝│ numbers) at 0, so the adapted layer starts as the base. d and b are
198⍝│ all it trains: m + r numbers.
199⍝│ >> "lo:" u_se< "LoRA"
200⍝│ >> (_, _, _, d, b) := 7 lo:v_era (2 3 r_eshape 1.0, 4)
201⍝│ >> d c_at b
202⍝│ 0.1 0.1 0.1 0.1 0.0 0.0
203ˡv̲era ← { seed (W, r) →
204 (A0, B0) ← seed ˡp̲rojections (s̲hape W) c̲at r
205 (W, A0, B0, r r̲eshape 0.1, (f̲irst s̲hape W) r̲eshape 0.0)
206}
207
208⍝│ (A0, B0, d, b) d̲eltaV: VeRA's update, diag(b) B0 diag(d) A0, an
209⍝│ m x n matrix of rank r at most. The two diagonals need no matrix and
210⍝│ no broadcasting: diag(b) B0 diag(d) is B0 times the table of b by d,
211⍝│ item by item (an m x r matrix, B's size).
212⍝│ >> "lo:" u_se< "LoRA"
213⍝│ >> lo:d_eltaV (1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0)
214⍝│ 1.0 1.0
215⍝│ -2.0 -2.0
216ˡd̲eltaV ← { (A0, B0, d, b) → (B0 × b '× t̲able d) '+ '× i̲nner A0 }
217
218⍝│ (W, A0, B0, d, b) f̲orwardV x: the VeRA layer, W x + b (B0 (d (A0
219⍝│ x))), the vectors scaling A0 x's rows and B0's output's: in factored
220⍝│ form, A0 x first, then the m x r matrix diag(b) B0 diag(d) of that.
221⍝│ >> "lo:" u_se< "LoRA"
222⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) lo:f_orwardV 2.0 4.0
223⍝│ 8.0 -8.0
224ˡf̲orwardV ← { (W, A0, B0, d, b) x → (W '+ '× i̲nner x) + (B0 × b '× t̲able d) '+ '× i̲nner A0 '+ '× i̲nner x }
225
226⍝│ m̲ergeV (W, A0, B0, d, b): the VeRA adapter folded into the
227⍝│ weights, W + diag(b) B0 diag(d) A0.
228⍝│ >> "lo:" u_se< "LoRA"
229⍝│ >> lo:m_ergeV (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0)
230⍝│ 2.0 1.0
231⍝│ -2.0 -1.0
232ˡm̲ergeV ← { (W, A0, B0, d, b) → W + ˡd̲eltaV (A0, B0, d, b) }
233
234⍝│ (W, A0, B0, d, b) l̲ossV (x, y): the squared error of the VeRA
235⍝│ layer, as l_oss.
236ˡl̲ossV ← { a (x, y) →
237 e ← (a ˡf̲orwardV x) − y
238 0.5 × '+ r̲/ r̲avel e × e
239}
240
241⍝│ (W, A0, B0, d, b) g̲radsV (x, g): VeRA's gradients for the inputs
242⍝│ x (n x N) and the output's gradient g (m x N), as (dd, db, dx). With
243⍝│ z = A0 x and Q = B0 times g z' item by item (m x r): dd = b' Q (the
244⍝│ research's (B0' (g b)) z, summed over the inputs), db = Q d (its g
245⍝│ (B0 (d z))), dx = W' g + A0' (diag(b) B0 diag(d))' g. The projections
246⍝│ are in every gradient and get none.
247⍝│ >> "lo:" u_se< "LoRA"
248⍝│ >> (dd, db, dx) := (2 2 r_eshape 1.0 0.0 0.0 1.0, 1 2 r_eshape 1.0 1.0, 2 1 r_eshape 1.0 -1.0, 1 r_eshape 0.5, 2.0 4.0) lo:g_radsV (2 1 r_eshape 2.0 4.0, 2 1 r_eshape 1.0 -1.0)
249⍝│ >> dd c_at db
250⍝│ 36.0 3.0 3.0
251ˡg̲radsV ← { (W, A0, B0, d, b) (x, g) →
252 Q ← B0 × g '+ '× i̲nner o̲\ A0 '+ '× i̲nner x
253 (b '+ '× i̲nner Q, Q '+ '× i̲nner d, ((o̲\ W) '+ '× i̲nner g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × b '× t̲able d) '+ '× i̲nner g)
254}
255
256⍝│ lr s̲gdV ((W, A0, B0, d, b), (x, y)): VeRA's step of gradient
257⍝│ descent on the squared error: d and b move, the rest stays.
258ˡs̲gdV ← { lr ((W, A0, B0, d, b), (x, y)) →
259 (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y)
260 (W, A0, B0, lr ᵒᵖs̲gd (d, dd), lr ᵒᵖs̲gd (b, db))
261}
262
263⍝│ a̲damStartV (W, A0, B0, d, b): VeRA's Adam state: the adapter, the
264⍝│ running averages of d and b at 0, the step 1.
265ˡa̲damStartV ← { (W, A0, B0, d, b) → ((W, A0, B0, d, b), (0.0 × d, 0.0 × d, 0.0 × b, 0.0 × b), 1) }
266
267⍝│ lr a̲damV (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)):
268⍝│ VeRA's Adam step: d and b move; the next state.
269ˡa̲damV ← { lr (((W, A0, B0, d, b), (md, vd, mb, vb), t), (x, y)) →
270 (dd, db, _) ← (W, A0, B0, d, b) ˡg̲radsV (x, ((W, A0, B0, d, b) ˡf̲orwardV x) − y)
271 (d2, md2, vd2) ← (lr, t) ᵒᵖa̲dam (d, md, vd, dd)
272 (b2, mb2, vb2) ← (lr, t) ᵒᵖa̲dam (b, mb, vb, db)
273 ((W, A0, B0, d2, b2), (md2, vd2, mb2, vb2), t + 1)
274}
275
276⍝━━ QLoRA: the base in 4 bits
277
278⍝│ k q̲base W: the base as QLoRA stores it: W in NF4, blocks of k
279⍝│ (QLoRA's 64), as (codes, scales, shape): 4-bit codes, a scale per
280⍝│ block, W's shape to come back to.
281⍝│ >> "lo:" u_se< "LoRA"
282⍝│ >> (q, s, sh) := 2 lo:q_base 2 2 r_eshape 1.0 -4.0 2.0 0.5
283⍝│ >> s c_at f_loat sh
284⍝│ 4.0 2.0 2.0 2.0
285ˡq̲base ← { k W →
286 (q, s) ← k qz:n̲f4Codes W
287 (q, s, s̲hape W)
288}
289
290⍝│ d̲equant (q, s, shape): the stored base back to numbers, W~, the
291⍝│ matrix QLoRA computes with: each code's level times its block's
292⍝│ scale, in W's shape. It is not W: the difference is the 4-bit
293⍝│ error.
294⍝│ >> "lo:" u_se< "LoRA"
295⍝│ >> lo:d_equant 2 lo:q_base 2 2 r_eshape 1.0 -4.0 2.0 0.5
296⍝│ 0.9844492077827454 -4.0
297⍝│ 2.0 0.4922246038913727
298ˡd̲equant ← { (q, s, sh) → sh r̲eshape qz:n̲f4Values (q, s) }
299
300⍝│ (Q, A, B, s) f̲orwardQ x: QLoRA's adapted layer, W~ x + s B (A x):
301⍝│ the 4-bit base dequantized when the layer runs, the adapter in full
302⍝│ precision. Its gradients and steps are LoRA's with W~ for W:
303⍝│ lr s̲gd ((d̲equant Q, A, B, s), (x, y)).
304ˡf̲orwardQ ← { (Q, A, B, s) x → (ˡd̲equant Q, A, B, s) ˡf̲orward x }
305
306⍝│ k m̲ergeQ (Q, A, B, s): the adapter merged into a 4-bit base: W~ +
307⍝│ s B A quantized again (blocks of k), as a stored base. Not exact:
308⍝│ d_equant of it is not W~ + s B A, since the merged numbers fall
309⍝│ between NF4's levels.
310ˡm̲ergeQ ← { k (Q, A, B, s) → k ˡq̲base (ˡd̲equant Q) + ˡd̲elta (A, B, s) }
311
312⍝│ (r, k) q̲bytes m n: QLoRA's bytes for an m x n base at rank r,
313⍝│ blocks of k: the base in NF4 (4 bits and a 32-bit scale per block),
314⍝│ the adapter in BF16 (16 bits), Adam's two averages for it in FP32;
315⍝│ against FP16's 2 m n for the base alone.
316⍝│ >> "lo:" u_se< "LoRA"
317⍝│ >> (16, 64) lo:q_bytes 4096 14336
318⍝│ 33030144.0 589824.0 2359296.0
319ˡq̲bytes ← { (r, k) mn →
320 n ← r ˡp̲arams mn
321 ((f̲loat '× r̲/ mn) × (4.0 + 32.0 ÷ f̲loat k) ÷ 8.0) c̲at (16 ˡb̲ytes n) c̲at 2.0 × 32 ˡb̲ytes n
322}
323
324⍝━━ Rank
325
326⍝│ One step of Gram-Schmidt on the rows: the k-th row's remainder R[k],
327⍝│ if larger than tol, becomes a direction and is taken out of every
328⍝│ row, and the count c goes up.
329ʰs̲tep ← { tol (R, k, c) →
330 v ← k s̲elect R
331 n ← s̲qrt '+ r̲/ v × v
332 n ≤ tol ? (R, k + 1, c)
333 (R − (R '+ '× i̲nner v ÷ n) '× t̲able v ÷ n, k + 1, c + 1)
334}
335
336⍝│ r̲ank M: the number of independent rows of the matrix M (its
337⍝│ rank), by Gram-Schmidt: a row counts when what is left of it, after
338⍝│ the directions of the rows before it are taken out, is larger than
339⍝│ 1e-9 times the largest row.
340⍝│ >> "lo:" u_se< "LoRA"
341⍝│ >> lo:r_ank 3 3 r_eshape 1.0 2.0 3.0 2.0 4.0 6.0 0.0 1.0 1.0
342⍝│ 2
343ˡr̲ank ← { M →
344 tol ← 0.000000001 × s̲qrt 'm̲ax r̲/ '+ r̲/₂ M × M
345 (_, _, c) ← (f̲irst s̲hape M) '{ R → tol ʰs̲tep R } p̲ower (M, 1, 0)
346 c
347}
348
349⍝━━ Counts
350
351⍝│ r p̲arams m n: the numbers an adapter of rank r on an m x n matrix
352⍝│ trains, r (m + n) (B's m r and A's r n), against the m n of the
353⍝│ matrix itself.
354⍝│ >> "lo:" u_se< "LoRA"
355⍝│ >> 16 lo:p_arams 4096 14336
356⍝│ 294912
357ˡp̲arams ← { r mn → r × '+ r̲/ mn }
358
359⍝│ r e̲xpert d f: the numbers rank-r adapters train on one SwiGLU
360⍝│ expert of hidden size d and inner size f: its three matrices (gate
361⍝│ and up, f x d; down, d x f), 3 r (d + f).
362⍝│ >> "lo:" u_se< "LoRA"
363⍝│ >> 16 lo:e_xpert 4096 14336
364⍝│ 884736
365ˡe̲xpert ← { r df → 3 × r ˡp̲arams df }
366
367⍝│ r e̲xpertFA d f: the same expert under LoRA-FA: the numbers it
368⍝│ trains (the B's, r (2f + d)), then the frozen A's it regenerates
369⍝│ (r (2d + f)); together r e_xpert d f.
370⍝│ >> "lo:" u_se< "LoRA"
371⍝│ >> 16 lo:e_xpertFA 4096 14336
372⍝│ 524288 360448
373ˡe̲xpertFA ← { r df →
374 (d, f) ← (f̲irst df, 2 s̲elect df)
375 (r × f + f + d) c̲at r × d + d + f
376}
377
378⍝│ b b̲ytes n: the bytes n numbers take at b bits each (16 for BF16
379⍝│ and FP16, 32 for FP32, 4 for NF4).
380⍝│ >> "lo:" u_se< "LoRA"
381⍝│ >> 16 lo:b_ytes 884736
382⍝│ 1769472.0
383ˡb̲ytes ← { b n → (f̲loat b × n) ÷ 8.0 }
384
385⍝│ r c̲osts m n: the multiply-adds for one input through rank-r
386⍝│ factors on an m x n matrix, B (A x) as r (m + n), then what making
387⍝│ the matrix B A first costs, r m n.
388⍝│ >> "lo:" u_se< "LoRA"
389⍝│ >> 8 lo:c_osts 64 64
390⍝│ 1024 32768
391ˡc̲osts ← { r mn → (r ˡp̲arams mn) c̲at r × '× r̲/ mn }
392
393⍝│ r p̲aramsV m n: the numbers VeRA trains on an m x n matrix at rank
394⍝│ r, m + r (b's m and d's r), against LoRA's r (m + n); its frozen
395⍝│ projections, r (m + n) numbers, are shared by every matrix of the
396⍝│ shape (or regenerated from their seed).
397⍝│ >> "lo:" u_se< "LoRA"
398⍝│ >> 16 lo:p_aramsV 14336 4096
399⍝│ 14352
400ˡp̲aramsV ← { r mn → r + f̲irst mn }
401
402⍝│ r e̲xpertV d f: the numbers VeRA trains on one SwiGLU expert of
403⍝│ hidden size d and inner size f, its three matrices at rank r: m + r
404⍝│ each, 2 f + d + 3 r. The projections are frozen and shared by every
405⍝│ expert: two pairs (one for gate and up, f x d; one for down, d x f),
406⍝│ 2 r (d + f) numbers, once.
407⍝│ >> "lo:" u_se< "LoRA"
408⍝│ >> 16 lo:e_xpertV 4096 14336
409⍝│ 32816
410ˡe̲xpertV ← { r df → (3 × r) + (2 × 2 s̲elect df) + f̲irst df }