macro librarylibs/LoRA/src/LoRA.xtlm
LoRA's macro: an adapted layer written from a short spec -- its forward and backward passes, for LoRA or LoRA-FA
Import it with the library: "lo:" u_se< "LoRA" reaches LoRA.xtl's functions and this file's macro (X_eTaL MC11). Names with m: are macros; h: names are private to this file.
A macro is a function from the source text written left and right of its call to the source that replaces the call, before the program is compiled (X_eTaL MC10); xetal expand FILE shows the result. The spec names the adapter's arrays, its rank and what trains: "W A B s | r 3 | train A B". The arrays are the frozen W (m x n), A (r x n), B (m x r) and the scale s, as LoRA.xtl's adapter (W, A, B, s), or for VeRA W, its frozen projections A and B and its vectors d (r numbers) and b (m), as LoRA.xtl's (W, A0, B0, d, b); "W" alone means "W A B s", or "W A B d b" when d and b train. The rank is a number or a name, and the expansion checks A's and B's shapes against it when the program runs. "train A B" is LoRA, "train B" is LoRA-FA (A frozen), "train d b" is VeRA (A and B frozen); nothing else trains. The expansion is ordinary code with no call to the library: the same formulas as lo:f_orward, lo:g_rads, lo:g_radsFA, lo:f_orwardV and lo:g_radsV, written out for these names.
Reading a spec (private)
ʰw̲ords : Char -> Box Char
The words of t: runs of characters other than white space, boxed (after X_eTaL's Combinators.xtlm).
ʰw̲ords ← { t → (n̲ot t m̲ember? " \n\t") p̲artition t }
ʰf̲ields : Char -> Box Char
The fields of t between bars, boxed.
ʰf̲ields ← { t → (n̲ot t m̲ember? "|") p̲artition t }
ʰj̲oin : Box Char -> Char
The boxed texts b joined with a space between ("" for none).
ʰj̲oin ← { b → 0 = t̲ally b ? "" d̲isclose '{ x y → e̲nclose (d̲isclose x) c̲at " " c̲at d̲isclose y } r̲/ b }
ʰn̲ame? : (Num a, Truthy a) => Char -> a
Whether the text w is a name a program can write (letters, digits, _ and an alias's colon, not starting with a digit).
ʰn̲ame? ← { w → 0 = t̲ally w ? 0 ('∧ r̲/ w m̲ember? "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_:") ∧ n̲ot (f̲irst w) m̲ember? "0123456789" }
ʰn̲umber? : Truthy a => Char -> a
Whether the text w is a whole number.
ʰn̲umber? ← { w → (0 < t̲ally w) ∧ '∧ r̲/ w m̲ember? ⎕D }
ʰn̲th : Any a => Box a -> Int -> a
The i-th of the boxed words ws, as text.
ʰn̲th ← { ws i → d̲isclose i s̲elect ws }
ʰh̲as? : (Match a, Truthy b) => Box a -> a -> b
Whether the boxed words ws hold the text w.
ʰh̲as? ← { ws w → '∨ r̲/ '{ v → w m̲atch d̲isclose v } e̲ach ws }
ʰa̲rrays : Box Char -> Box Char -> Box Char
The arrays' names: those written, or W with A, B and s, or with A, B, d and b when the trained names ts are d and b.
ʰa̲rrays ← { ws ts → (1 = t̲ally ws) ∧ ts ʰh̲as? "d" ? ws c̲at ʰw̲ords "A B d b" 1 = t̲ally ws ? ws c̲at ʰw̲ords "A B s" ws }
specShape : Char
What a spec looks like, for the messages.
specShape ← "a spec is the arrays, the rank and what trains: \"W A B s | r 3 | train A B\" (LoRA), \"W A B s | r 3 | train B\" (LoRA-FA) or \"W A B d b | r 3 | train d b\" (VeRA)"
ʰw̲rong : Box Char -> Char
What is wrong with the spec's fields fs, or "".
ʰw̲rong ← { fs → 3 ≠ t̲ally fs ? specShape ns ← ʰw̲ords d̲isclose 1 s̲elect fs rs ← ʰw̲ords d̲isclose 2 s̲elect fs ts ← ʰw̲ords d̲isclose 3 s̲elect fs n̲ot (1 = t̲ally ns) ∨ (4 = t̲ally ns) ∨ 5 = t̲ally ns ? "name the arrays W A B s (W A B d b for VeRA), or W alone: " c̲at specShape n̲ot '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach ns ? "not a name: " c̲at ʰj̲oin ns n̲ot (2 = t̲ally rs) ? "write the rank as r and a number or a name: \"r 3\"" n̲ot ("r" m̲atch d̲isclose f̲irst rs) ∧ (ʰn̲umber? rs ʰn̲th 2) ∨ ʰn̲ame? rs ʰn̲th 2 ? "write the rank as r and a number or a name: \"r 3\"" n̲ot (1 < t̲ally ts) ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)" n̲ot "train" m̲atch d̲isclose f̲irst ts ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)" "" }
ʰv̲era? : Match a => Box a -> Box a -> Char
"vera" if the trained names ts are the fourth and fifth of the arrays as, d and b, else "".
ʰv̲era? ← { as ts → (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 4) ∧ ts ʰh̲as? as ʰn̲th 5 ? "vera" "" }
ʰm̲ethod : Match a => Box a -> Box a -> Char
The method the trained names ts (after "train") pick for the arrays as: "lora" for A and B, "fa" for B alone, "vera" for d and b (five arrays), "" for anything else.
ʰm̲ethod ← { as ts → 5 = t̲ally as ? as ʰv̲era? ts (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 2) ∧ ts ʰh̲as? as ʰn̲th 3 ? "lora" (1 = t̲ally ts) ∧ ts ʰh̲as? as ʰn̲th 3 ? "fa" "" }
Writing the passes (private)
ʰc̲heck : Any a => (a, Char, Char, Char) -> Char -> Char
The rank check, a statement: A has r rows and B r columns, or the program stops naming the spec.
ʰc̲heck ← { (W, A, B, r) spec → "_ := { v -> v ? 0; []P_ANIC \"lo:a_dapter<: A must be " c̲at r c̲at " x n and B m x " c̲at r c̲at " for \\\"" c̲at spec c̲at "\\\"\" } (" c̲at r c̲at " = f_irst s_hape " c̲at A c̲at ") & " c̲at r c̲at " = 2 s_elect s_hape " c̲at B }
ʰf̲orward : (Char, Char, Char, Char) -> Char -> Char
The forward pass: W x + s B (A x), right to left, A x first.
ʰf̲orward ← { (W, A, B, s) x → "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + " c̲at s c̲at " * " c̲at B c̲at " '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }" }
ʰb̲ackLoRA : (Char, Char, Char, Char) -> (Char, Char, Char) -> Char
LoRA's backward pass for inputs x and the output's gradient g: (dA, dB, dx) = (s (B' g) x', s g (A x)', W' g + s A' (B' g)), B' g bound once under b.
ʰb̲ackLoRA ← { (W, A, B, s) (x, g, b) → "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at b c̲at " := (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at "\n (" c̲at s c̲at " * " c̲at b c̲at " '+ '* i_nner o_\\ " c̲at x c̲at ", " c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner " c̲at b c̲at ")\n}" }
ʰb̲ackFA : (Char, Char, Char, Char) -> (Char, Char) -> Char
LoRA-FA's: A frozen, its gradient skipped; (dB, dx), x's still back through A.
ʰb̲ackFA ← { (W, A, B, s) (x, g) → "{ (" c̲at x c̲at ", " c̲at g c̲at ") -> (" c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at ") }" }
ʰf̲orwardV : (Char, Char, Char, Char, Char) -> Char -> Char
VeRA's forward pass: W x + b (B (d (A x))), the m x r matrix diag(b) B diag(d) as B times the table of b by d.
ʰf̲orwardV ← { (W, A, B, d, b) x → "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + (" c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }" }
ʰb̲ackV : (Char, Char, Char, Char, Char) -> (Char, Char, Char) -> Char
VeRA's backward pass: (dd, db, dx) = (b' Q, Q d, W' g + A' (diag(b) B diag(d))' g), Q = B times g (A x)' item by item, bound once under q.
ʰb̲ackV ← { (W, A, B, d, b) (x, g, q) → "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n " c̲at q c̲at " := " c̲at B c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at "\n (" c̲at b c̲at " '+ '* i_nner " c̲at q c̲at ", " c̲at q c̲at " '+ '* i_nner " c̲at d c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at g c̲at ")\n}" }
ʰb̲ack : Char -> ((Char, Char, Char, Char), Char, Char, Char) -> Char
The backward pass the method how picks, LoRA's or LoRA-FA's.
ʰb̲ack ← { how (arrays, x, g, b) → "lora" m̲atch how ? arrays ʰb̲ackLoRA (x, g, b) arrays ʰb̲ackFA (x, g) }
ʰp̲asses : (Box Char, Char, Char, Char) -> Char
The statements: the rank check, then the forward and the backward functions under the names fn.
ʰp̲asses ← { (fn, check, fwd, back) → check c̲at "\n" c̲at (fn ʰn̲th 1) c̲at " := " c̲at fwd c̲at "\n" c̲at (fn ʰn̲th 2) c̲at " := " c̲at back }
Adapters
ᵐa̲dapter< : Char -> Char -> Char
An adapted layer from the spec on the right, as statements: the rank check, then the forward function and the backward function under the two names on the left ("u:f_wd u:b_wd"). The forward function takes the inputs (columns) and gives W x + s B (A x); the backward one takes the inputs and the loss's gradient at the output, (x, g), and gives (dA, dB, dx) for "train A B" (LoRA), (dB, dx) for "train B" (LoRA-FA) or (dd, db, dx) for "train d b" (VeRA, whose forward function is W x + b (B (d (A x)))). The lambdas' own names are fresh ([]G_ENSYM, X_eTaL MC30), so a caller's array named x, g or b is not captured.
ᵐa̲dapter< ← { what spec → fs ← ʰf̲ields spec 0 < t̲ally ʰw̲rong fs ? "bad-macro-argument right" ⎕R̲EJECT ʰw̲rong fs ts ← 1 d̲rop ʰw̲ords d̲isclose 3 s̲elect fs as ← (ʰw̲ords d̲isclose 1 s̲elect fs) ʰa̲rrays ts r ← (ʰw̲ords d̲isclose 2 s̲elect fs) ʰn̲th 2 how ← as ʰm̲ethod ts 0 = t̲ally how ? "bad-macro-argument right" ⎕R̲EJECT "LoRA trains A and B (\"train " c̲at (as ʰn̲th 2) c̲at " " c̲at (as ʰn̲th 3) c̲at "\"), LoRA-FA B alone (\"train " c̲at (as ʰn̲th 3) c̲at "\"), VeRA d and b of five arrays (\"W A B d b | r 3 | train d b\"); nothing else trains here" fn ← ʰw̲ords what n̲ot (2 = t̲ally fn) ∧ '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach fn ? "bad-macro-argument left" ⎕R̲EJECT "name the forward and the backward functions: \"u:f_wd u:b_wd\"" (x, g, b) ← (⎕G̲ENSYM "x", ⎕G̲ENSYM "g", ⎕G̲ENSYM "b") check ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, r) ʰc̲heck spec "vera" m̲atch how ? ʰp̲asses (fn, check, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰf̲orwardV x, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰb̲ackV (x, g, b)) arrays ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4) ʰp̲asses (fn, check, arrays ʰf̲orward x, how ʰb̲ack (arrays, x, g, b)) }