sourcelibs/LoRA/src/LoRA.xtlm
1⍝│ LoRA's macro: an adapted layer written from a short spec -- its forward and backward passes, for LoRA or LoRA-FA
2⍝│
3⍝│ Import it with the library: "lo:" u_se< "LoRA" reaches LoRA.xtl's
4⍝│ functions and this file's macro (X_eTaL MC11).
5⍝│ Names with m: are macros; h: names are private to this file.
6⍝│
7⍝│ A macro is a function from the source text written left and right of
8⍝│ its call to the source that replaces the call, before the program is
9⍝│ compiled (X_eTaL MC10); xetal expand FILE shows the result. The spec
10⍝│ names the adapter's arrays, its rank and what trains:
11⍝│ "W A B s | r 3 | train A B". The arrays are the frozen W (m x n), A
12⍝│ (r x n), B (m x r) and the scale s, as LoRA.xtl's adapter (W, A, B,
13⍝│ s), or for VeRA W, its frozen projections A and B and its vectors d
14⍝│ (r numbers) and b (m), as LoRA.xtl's (W, A0, B0, d, b); "W" alone
15⍝│ means "W A B s", or "W A B d b" when d and b train. The rank is a
16⍝│ number or a name, and the expansion checks A's and B's shapes against
17⍝│ it when the program runs. "train A B" is LoRA, "train B" is LoRA-FA
18⍝│ (A frozen), "train d b" is VeRA (A and B frozen); nothing else
19⍝│ trains. The expansion is ordinary code with no call to the library:
20⍝│ the same formulas as lo:f_orward, lo:g_rads, lo:g_radsFA,
21⍝│ lo:f_orwardV and lo:g_radsV, written out for these names.
22
23⍝━━ Reading a spec (private)
24
25⍝│ The words of t: runs of characters other than white space, boxed
26⍝│ (after X_eTaL's Combinators.xtlm).
27ʰw̲ords ← { t → (n̲ot t m̲ember? " \n\t") p̲artition t }
28⍝│ The fields of t between bars, boxed.
29ʰf̲ields ← { t → (n̲ot t m̲ember? "|") p̲artition t }
30⍝│ The boxed texts b joined with a space between ("" for none).
31ʰj̲oin ← { b →
32 0 = t̲ally b ? ""
33 d̲isclose '{ x y → e̲nclose (d̲isclose x) c̲at " " c̲at d̲isclose y } r̲/ b
34}
35⍝│ Whether the text w is a name a program can write (letters, digits,
36⍝│ _ and an alias's colon, not starting with a digit).
37ʰn̲ame? ← { w →
38 0 = t̲ally w ? 0
39 ('∧ r̲/ w m̲ember? "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_:") ∧ n̲ot (f̲irst w) m̲ember? "0123456789"
40}
41⍝│ Whether the text w is a whole number.
42ʰn̲umber? ← { w → (0 < t̲ally w) ∧ '∧ r̲/ w m̲ember? ⎕D }
43⍝│ The i-th of the boxed words ws, as text.
44ʰn̲th ← { ws i → d̲isclose i s̲elect ws }
45⍝│ Whether the boxed words ws hold the text w.
46ʰh̲as? ← { ws w → '∨ r̲/ '{ v → w m̲atch d̲isclose v } e̲ach ws }
47⍝│ The arrays' names: those written, or W with A, B and s, or with A,
48⍝│ B, d and b when the trained names ts are d and b.
49ʰa̲rrays ← { ws ts →
50 (1 = t̲ally ws) ∧ ts ʰh̲as? "d" ? ws c̲at ʰw̲ords "A B d b"
51 1 = t̲ally ws ? ws c̲at ʰw̲ords "A B s"
52 ws
53}
54⍝│ What a spec looks like, for the messages.
55specShape ← "a spec is the arrays, the rank and what trains: \"W A B s | r 3 | train A B\" (LoRA), \"W A B s | r 3 | train B\" (LoRA-FA) or \"W A B d b | r 3 | train d b\" (VeRA)"
56⍝│ What is wrong with the spec's fields fs, or "".
57ʰw̲rong ← { fs →
58 3 ≠ t̲ally fs ? specShape
59 ns ← ʰw̲ords d̲isclose 1 s̲elect fs
60 rs ← ʰw̲ords d̲isclose 2 s̲elect fs
61 ts ← ʰw̲ords d̲isclose 3 s̲elect fs
62 n̲ot (1 = t̲ally ns) ∨ (4 = t̲ally ns) ∨ 5 = t̲ally ns ? "name the arrays W A B s (W A B d b for VeRA), or W alone: " c̲at specShape
63 n̲ot '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach ns ? "not a name: " c̲at ʰj̲oin ns
64 n̲ot (2 = t̲ally rs) ? "write the rank as r and a number or a name: \"r 3\""
65 n̲ot ("r" m̲atch d̲isclose f̲irst rs) ∧ (ʰn̲umber? rs ʰn̲th 2) ∨ ʰn̲ame? rs ʰn̲th 2 ? "write the rank as r and a number or a name: \"r 3\""
66 n̲ot (1 < t̲ally ts) ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)"
67 n̲ot "train" m̲atch d̲isclose f̲irst ts ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)"
68 ""
69}
70⍝│ "vera" if the trained names ts are the fourth and fifth of the
71⍝│ arrays as, d and b, else "".
72ʰv̲era? ← { as ts →
73 (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 4) ∧ ts ʰh̲as? as ʰn̲th 5 ? "vera"
74 ""
75}
76⍝│ The method the trained names ts (after "train") pick for the arrays
77⍝│ as: "lora" for A and B, "fa" for B alone, "vera" for d and b (five
78⍝│ arrays), "" for anything else.
79ʰm̲ethod ← { as ts →
80 5 = t̲ally as ? as ʰv̲era? ts
81 (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 2) ∧ ts ʰh̲as? as ʰn̲th 3 ? "lora"
82 (1 = t̲ally ts) ∧ ts ʰh̲as? as ʰn̲th 3 ? "fa"
83 ""
84}
85
86⍝━━ Writing the passes (private)
87
88⍝│ The rank check, a statement: A has r rows and B r columns, or the
89⍝│ program stops naming the spec.
90ʰc̲heck ← { (W, A, B, r) spec →
91 "_ := { v -> v ? 0; []P_ANIC \"lo:a_dapter<: A must be " c̲at r c̲at " x n and B m x " c̲at r c̲at " for \\\"" c̲at spec c̲at "\\\"\" } (" c̲at r c̲at " = f_irst s_hape " c̲at A c̲at ") & " c̲at r c̲at " = 2 s_elect s_hape " c̲at B
92}
93⍝│ The forward pass: W x + s B (A x), right to left, A x first.
94ʰf̲orward ← { (W, A, B, s) x →
95 "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + " c̲at s c̲at " * " c̲at B c̲at " '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }"
96}
97⍝│ LoRA's backward pass for inputs x and the output's gradient g:
98⍝│ (dA, dB, dx) = (s (B' g) x', s g (A x)', W' g + s A' (B' g)), B' g
99⍝│ bound once under b.
100ʰb̲ackLoRA ← { (W, A, B, s) (x, g, b) →
101 "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at b c̲at " := (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at "\n (" c̲at s c̲at " * " c̲at b c̲at " '+ '* i_nner o_\\ " c̲at x c̲at ", " c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner " c̲at b c̲at ")\n}"
102}
103⍝│ LoRA-FA's: A frozen, its gradient skipped; (dB, dx), x's still back
104⍝│ through A.
105ʰb̲ackFA ← { (W, A, B, s) (x, g) →
106 "{ (" c̲at x c̲at ", " c̲at g c̲at ") -> (" c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at ") }"
107}
108
109⍝│ VeRA's forward pass: W x + b (B (d (A x))), the m x r matrix
110⍝│ diag(b) B diag(d) as B times the table of b by d.
111ʰf̲orwardV ← { (W, A, B, d, b) x →
112 "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + (" c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }"
113}
114⍝│ VeRA's backward pass: (dd, db, dx) = (b' Q, Q d, W' g + A' (diag(b)
115⍝│ B diag(d))' g), Q = B times g (A x)' item by item, bound once under q.
116ʰb̲ackV ← { (W, A, B, d, b) (x, g, q) →
117 "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at q c̲at " := " c̲at B c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at "\n (" c̲at b c̲at " '+ '* i_nner " c̲at q c̲at ", " c̲at q c̲at " '+ '* i_nner " c̲at d c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at g c̲at ")\n}"
118}
119
120⍝│ The backward pass the method how picks, LoRA's or LoRA-FA's.
121ʰb̲ack ← { how (arrays, x, g, b) →
122 "lora" m̲atch how ? arrays ʰb̲ackLoRA (x, g, b)
123 arrays ʰb̲ackFA (x, g)
124}
125
126⍝│ The statements: the rank check, then the forward and the backward
127⍝│ functions under the names fn.
128ʰp̲asses ← { (fn, check, fwd, back) → check c̲at "\n" c̲at (fn ʰn̲th 1) c̲at " := " c̲at fwd c̲at "\n" c̲at (fn ʰn̲th 2) c̲at " := " c̲at back }
129
130⍝━━ Adapters
131
132⍝│ An adapted layer from the spec on the right, as statements: the rank
133⍝│ check, then the forward function and the backward function under the
134⍝│ two names on the left ("u:f_wd u:b_wd"). The forward function takes
135⍝│ the inputs (columns) and gives W x + s B (A x); the backward one
136⍝│ takes the inputs and the loss's gradient at the output, (x, g), and
137⍝│ gives (dA, dB, dx) for "train A B" (LoRA), (dB, dx) for "train B"
138⍝│ (LoRA-FA) or (dd, db, dx) for "train d b" (VeRA, whose forward
139⍝│ function is W x + b (B (d (A x)))). The lambdas' own names are fresh ([]G_ENSYM, X_eTaL
140⍝│ MC30), so a caller's array named x, g or b is not captured.
141⍝│ uses: parsed
142ᵐa̲dapter< ← { what spec →
143 fs ← ʰf̲ields spec
144 0 < t̲ally ʰw̲rong fs ? "bad-macro-argument right" ⎕R̲EJECT ʰw̲rong fs
145 ts ← 1 d̲rop ʰw̲ords d̲isclose 3 s̲elect fs
146 as ← (ʰw̲ords d̲isclose 1 s̲elect fs) ʰa̲rrays ts
147 r ← (ʰw̲ords d̲isclose 2 s̲elect fs) ʰn̲th 2
148 how ← as ʰm̲ethod ts
149 0 = t̲ally how ? "bad-macro-argument right" ⎕R̲EJECT "LoRA trains A and B (\"train " c̲at (as ʰn̲th 2) c̲at " " c̲at (as ʰn̲th 3) c̲at "\"), LoRA-FA B alone (\"train " c̲at (as ʰn̲th 3) c̲at "\"), VeRA d and b of five arrays (\"W A B d b | r 3 | train d b\"); nothing else trains here"
150 fn ← ʰw̲ords what
151 n̲ot (2 = t̲ally fn) ∧ '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach fn ? "bad-macro-argument left" ⎕R̲EJECT "name the forward and the backward functions: \"u:f_wd u:b_wd\""
152 (x, g, b) ← (⎕G̲ENSYM "x", ⎕G̲ENSYM "g", ⎕G̲ENSYM "b")
153 check ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, r) ʰc̲heck spec
154 "vera" m̲atch how ? ʰp̲asses (fn, check, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰf̲orwardV x, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰb̲ackV (x, g, b))
155 arrays ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4)
156 ʰp̲asses (fn, check, arrays ʰf̲orward x, how ʰb̲ack (arrays, x, g, b))
157}