sourcelibs/LoRA/src/LoRA.xtlm

1⍝│ LoRA's macro: an adapted layer written from a short spec -- its forward and backward passes, for LoRA or LoRA-FA 2⍝│ 3⍝│ Import it with the library: "lo:" u_se< "LoRA" reaches LoRA.xtl's 4⍝│ functions and this file's macro (X_eTaL MC11). 5⍝│ Names with m: are macros; h: names are private to this file. 6⍝│ 7⍝│ A macro is a function from the source text written left and right of 8⍝│ its call to the source that replaces the call, before the program is 9⍝│ compiled (X_eTaL MC10); xetal expand FILE shows the result. The spec 10⍝│ names the adapter's arrays, its rank and what trains: 11⍝│ "W A B s | r 3 | train A B". The arrays are the frozen W (m x n), A 12⍝│ (r x n), B (m x r) and the scale s, as LoRA.xtl's adapter (W, A, B, 13⍝│ s), or for VeRA W, its frozen projections A and B and its vectors d 14⍝│ (r numbers) and b (m), as LoRA.xtl's (W, A0, B0, d, b); "W" alone 15⍝│ means "W A B s", or "W A B d b" when d and b train. The rank is a 16⍝│ number or a name, and the expansion checks A's and B's shapes against 17⍝│ it when the program runs. "train A B" is LoRA, "train B" is LoRA-FA 18⍝│ (A frozen), "train d b" is VeRA (A and B frozen); nothing else 19⍝│ trains. The expansion is ordinary code with no call to the library: 20⍝│ the same formulas as lo:f_orward, lo:g_rads, lo:g_radsFA, 21⍝│ lo:f_orwardV and lo:g_radsV, written out for these names. 22 23⍝━━ Reading a spec (private) 24 25⍝│ The words of t: runs of characters other than white space, boxed 26⍝│ (after X_eTaL's Combinators.xtlm). 27ʰw̲ords ← { t → (n̲ot t m̲ember? " \n\t") p̲artition t } 28⍝│ The fields of t between bars, boxed. 29ʰf̲ields ← { t → (n̲ot t m̲ember? "|") p̲artition t } 30⍝│ The boxed texts b joined with a space between ("" for none). 31ʰj̲oin ← { b → 32 0 = t̲ally b ? "" 33 d̲isclose '{ x y → e̲nclose (d̲isclose x) c̲at " " c̲at d̲isclose y } r̲/ b 34} 35⍝│ Whether the text w is a name a program can write (letters, digits, 36⍝│ _ and an alias's colon, not starting with a digit). 37ʰn̲ame? ← { w → 38 0 = t̲ally w ? 0 39 ('∧ r̲/ w m̲ember? "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_:") ∧ n̲ot (f̲irst w) m̲ember? "0123456789" 40} 41⍝│ Whether the text w is a whole number. 42ʰn̲umber? ← { w → (0 < t̲ally w) ∧ '∧ r̲/ w m̲ember? ⎕D } 43⍝│ The i-th of the boxed words ws, as text. 44ʰn̲th ← { ws i → d̲isclose i s̲elect ws } 45⍝│ Whether the boxed words ws hold the text w. 46ʰh̲as? ← { ws w → '∨ r̲/ '{ v → w m̲atch d̲isclose v } e̲ach ws } 47⍝│ The arrays' names: those written, or W with A, B and s, or with A, 48⍝│ B, d and b when the trained names ts are d and b. 49ʰa̲rrays ← { ws ts → 50 (1 = t̲ally ws) ∧ ts ʰh̲as? "d" ? ws c̲at ʰw̲ords "A B d b" 51 1 = t̲ally ws ? ws c̲at ʰw̲ords "A B s" 52 ws 53} 54⍝│ What a spec looks like, for the messages. 55specShape ← "a spec is the arrays, the rank and what trains: \"W A B s | r 3 | train A B\" (LoRA), \"W A B s | r 3 | train B\" (LoRA-FA) or \"W A B d b | r 3 | train d b\" (VeRA)" 56⍝│ What is wrong with the spec's fields fs, or "". 57ʰw̲rong ← { fs → 58 3 ≠ t̲ally fs ? specShape 59 ns ← ʰw̲ords d̲isclose 1 s̲elect fs 60 rs ← ʰw̲ords d̲isclose 2 s̲elect fs 61 ts ← ʰw̲ords d̲isclose 3 s̲elect fs 62 n̲ot (1 = t̲ally ns) ∨ (4 = t̲ally ns) ∨ 5 = t̲ally ns ? "name the arrays W A B s (W A B d b for VeRA), or W alone: " c̲at specShape 63 n̲ot '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach ns ? "not a name: " c̲at ʰj̲oin ns 64 n̲ot (2 = t̲ally rs) ? "write the rank as r and a number or a name: \"r 3\"" 65 n̲ot ("r" m̲atch d̲isclose f̲irst rs) ∧ (ʰn̲umber? rs ʰn̲th 2) ∨ ʰn̲ame? rs ʰn̲th 2 ? "write the rank as r and a number or a name: \"r 3\"" 66 n̲ot (1 < t̲ally ts) ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)" 67 n̲ot "train" m̲atch d̲isclose f̲irst ts ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)" 68 "" 69} 70⍝│ "vera" if the trained names ts are the fourth and fifth of the 71⍝│ arrays as, d and b, else "". 72ʰv̲era? ← { as ts → 73 (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 4) ∧ ts ʰh̲as? as ʰn̲th 5 ? "vera" 74 "" 75} 76⍝│ The method the trained names ts (after "train") pick for the arrays 77⍝│ as: "lora" for A and B, "fa" for B alone, "vera" for d and b (five 78⍝│ arrays), "" for anything else. 79ʰm̲ethod ← { as ts → 80 5 = t̲ally as ? as ʰv̲era? ts 81 (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 2) ∧ ts ʰh̲as? as ʰn̲th 3 ? "lora" 82 (1 = t̲ally ts) ∧ ts ʰh̲as? as ʰn̲th 3 ? "fa" 83 "" 84} 85 86⍝━━ Writing the passes (private) 87 88⍝│ The rank check, a statement: A has r rows and B r columns, or the 89⍝│ program stops naming the spec. 90ʰc̲heck ← { (W, A, B, r) spec → 91 "_ := { v -> v ? 0; []P_ANIC \"lo:a_dapter<: A must be " c̲at r c̲at " x n and B m x " c̲at r c̲at " for \\\"" c̲at spec c̲at "\\\"\" } (" c̲at r c̲at " = f_irst s_hape " c̲at A c̲at ") & " c̲at r c̲at " = 2 s_elect s_hape " c̲at B 92} 93⍝│ The forward pass: W x + s B (A x), right to left, A x first. 94ʰf̲orward ← { (W, A, B, s) x → 95 "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + " c̲at s c̲at " * " c̲at B c̲at " '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }" 96} 97⍝│ LoRA's backward pass for inputs x and the output's gradient g: 98⍝│ (dA, dB, dx) = (s (B' g) x', s g (A x)', W' g + s A' (B' g)), B' g 99⍝│ bound once under b. 100ʰb̲ackLoRA ← { (W, A, B, s) (x, g, b) → 101 "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at b c̲at " := (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at "\n (" c̲at s c̲at " * " c̲at b c̲at " '+ '* i_nner o_\\ " c̲at x c̲at ", " c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner " c̲at b c̲at ")\n}" 102} 103⍝│ LoRA-FA's: A frozen, its gradient skipped; (dB, dx), x's still back 104⍝│ through A. 105ʰb̲ackFA ← { (W, A, B, s) (x, g) → 106 "{ (" c̲at x c̲at ", " c̲at g c̲at ") -> (" c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at ") }" 107} 108 109⍝│ VeRA's forward pass: W x + b (B (d (A x))), the m x r matrix 110⍝│ diag(b) B diag(d) as B times the table of b by d. 111ʰf̲orwardV ← { (W, A, B, d, b) x → 112 "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + (" c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }" 113} 114⍝│ VeRA's backward pass: (dd, db, dx) = (b' Q, Q d, W' g + A' (diag(b) 115⍝│ B diag(d))' g), Q = B times g (A x)' item by item, bound once under q. 116ʰb̲ackV ← { (W, A, B, d, b) (x, g, q) → 117 "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at q c̲at " := " c̲at B c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at "\n (" c̲at b c̲at " '+ '* i_nner " c̲at q c̲at ", " c̲at q c̲at " '+ '* i_nner " c̲at d c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at g c̲at ")\n}" 118} 119 120⍝│ The backward pass the method how picks, LoRA's or LoRA-FA's. 121ʰb̲ack ← { how (arrays, x, g, b) → 122 "lora" m̲atch how ? arrays ʰb̲ackLoRA (x, g, b) 123 arrays ʰb̲ackFA (x, g) 124} 125 126⍝│ The statements: the rank check, then the forward and the backward 127⍝│ functions under the names fn. 128ʰp̲asses ← { (fn, check, fwd, back) → check c̲at "\n" c̲at (fn ʰn̲th 1) c̲at " := " c̲at fwd c̲at "\n" c̲at (fn ʰn̲th 2) c̲at " := " c̲at back } 129 130⍝━━ Adapters 131 132⍝│ An adapted layer from the spec on the right, as statements: the rank 133⍝│ check, then the forward function and the backward function under the 134⍝│ two names on the left ("u:f_wd u:b_wd"). The forward function takes 135⍝│ the inputs (columns) and gives W x + s B (A x); the backward one 136⍝│ takes the inputs and the loss's gradient at the output, (x, g), and 137⍝│ gives (dA, dB, dx) for "train A B" (LoRA), (dB, dx) for "train B" 138⍝│ (LoRA-FA) or (dd, db, dx) for "train d b" (VeRA, whose forward 139⍝│ function is W x + b (B (d (A x)))). The lambdas' own names are fresh ([]G_ENSYM, X_eTaL 140⍝│ MC30), so a caller's array named x, g or b is not captured. 141⍝│ uses: parsed 142ᵐa̲dapter< ← { what spec → 143 fs ← ʰf̲ields spec 144 0 < t̲ally ʰw̲rong fs ? "bad-macro-argument right" ⎕R̲EJECT ʰw̲rong fs 145 ts ← 1 d̲rop ʰw̲ords d̲isclose 3 s̲elect fs 146 as ← (ʰw̲ords d̲isclose 1 s̲elect fs) ʰa̲rrays ts 147 r ← (ʰw̲ords d̲isclose 2 s̲elect fs) ʰn̲th 2 148 how ← as ʰm̲ethod ts 149 0 = t̲ally how ? "bad-macro-argument right" ⎕R̲EJECT "LoRA trains A and B (\"train " c̲at (as ʰn̲th 2) c̲at " " c̲at (as ʰn̲th 3) c̲at "\"), LoRA-FA B alone (\"train " c̲at (as ʰn̲th 3) c̲at "\"), VeRA d and b of five arrays (\"W A B d b | r 3 | train d b\"); nothing else trains here" 150 fn ← ʰw̲ords what 151 n̲ot (2 = t̲ally fn) ∧ '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach fn ? "bad-macro-argument left" ⎕R̲EJECT "name the forward and the backward functions: \"u:f_wd u:b_wd\"" 152 (x, g, b) ← (⎕G̲ENSYM "x", ⎕G̲ENSYM "g", ⎕G̲ENSYM "b") 153 check ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, r) ʰc̲heck spec 154 "vera" m̲atch how ? ʰp̲asses (fn, check, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰf̲orwardV x, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰb̲ackV (x, g, b)) 155 arrays ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4) 156 ʰp̲asses (fn, check, arrays ʰf̲orward x, how ʰb̲ack (arrays, x, g, b)) 157}