macro librarylibs/LoRA/src/LoRA.xtlm

LoRA's macro: an adapted layer written from a short spec -- its forward and backward passes, for LoRA or LoRA-FA

Import it with the library: "lo:" u_se< "LoRA" reaches LoRA.xtl's functions and this file's macro (X_eTaL MC11). Names with m: are macros; h: names are private to this file.

A macro is a function from the source text written left and right of its call to the source that replaces the call, before the program is compiled (X_eTaL MC10); xetal expand FILE shows the result. The spec names the adapter's arrays, its rank and what trains: "W A B s | r 3 | train A B". The arrays are the frozen W (m x n), A (r x n), B (m x r) and the scale s, as LoRA.xtl's adapter (W, A, B, s), or for VeRA W, its frozen projections A and B and its vectors d (r numbers) and b (m), as LoRA.xtl's (W, A0, B0, d, b); "W" alone means "W A B s", or "W A B d b" when d and b train. The rank is a number or a name, and the expansion checks A's and B's shapes against it when the program runs. "train A B" is LoRA, "train B" is LoRA-FA (A frozen), "train d b" is VeRA (A and B frozen); nothing else trains. The expansion is ordinary code with no call to the library: the same formulas as lo:f_orward, lo:g_rads, lo:g_radsFA, lo:f_orwardV and lo:g_radsV, written out for these names.

source

Reading a spec (private)

ʰw̲ords : Char -> Box Char

function (private) · line 27

The words of t: runs of characters other than white space, boxed (after X_eTaL's Combinators.xtlm).

ʰw̲ords ← { t → (n̲ot t m̲ember? " \n\t") p̲artition t }

ʰf̲ields : Char -> Box Char

function (private) · line 29

The fields of t between bars, boxed.

ʰf̲ields ← { t → (n̲ot t m̲ember? "|") p̲artition t }
Used in: ᵐa̲dapter<

ʰj̲oin : Box Char -> Char

function (private) · line 31

The boxed texts b joined with a space between ("" for none).

ʰj̲oin ← { b →
  0 = t̲ally b ? ""
  d̲isclose '{ x y → e̲nclose (d̲isclose x) c̲at " " c̲at d̲isclose y } r̲/ b
}
Used in: ʰw̲rong

ʰn̲ame? : (Num a, Truthy a) => Char -> a

function (private) · line 37

Whether the text w is a name a program can write (letters, digits, _ and an alias's colon, not starting with a digit).

ʰn̲ame? ← { w →
  0 = t̲ally w ? 0
  ('∧ r̲/ w m̲ember? "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_:") ∧ n̲ot (f̲irst w) m̲ember? "0123456789"
}

ʰn̲umber? : Truthy a => Char -> a

function (private) · line 42

Whether the text w is a whole number.

ʰn̲umber? ← { w → (0 < t̲ally w) ∧ '∧ r̲/ w m̲ember? ⎕D }
Used in: ʰw̲rong

ʰn̲th : Any a => Box a -> Int -> a

function (private) · line 44

The i-th of the boxed words ws, as text.

ʰn̲th ← { ws i → d̲isclose i s̲elect ws }

ʰh̲as? : (Match a, Truthy b) => Box a -> a -> b

function (private) · line 46

Whether the boxed words ws hold the text w.

ʰh̲as? ← { ws w → '∨ r̲/ '{ v → w m̲atch d̲isclose v } e̲ach ws }

ʰa̲rrays : Box Char -> Box Char -> Box Char

function (private) · line 49

The arrays' names: those written, or W with A, B and s, or with A, B, d and b when the trained names ts are d and b.

ʰa̲rrays ← { ws ts →
  (1 = t̲ally ws) ∧ ts ʰh̲as? "d" ? ws c̲at ʰw̲ords "A B d b"
  1 = t̲ally ws ? ws c̲at ʰw̲ords "A B s"
  ws
}
Used in: ᵐa̲dapter<

specShape : Char

value (private) · line 55

What a spec looks like, for the messages.

specShape ← "a spec is the arrays, the rank and what trains: \"W A B s | r 3 | train A B\" (LoRA), \"W A B s | r 3 | train B\" (LoRA-FA) or \"W A B d b | r 3 | train d b\" (VeRA)"
Used in: ʰw̲rong

ʰw̲rong : Box Char -> Char

function (private) · line 57

What is wrong with the spec's fields fs, or "".

ʰw̲rong ← { fs →
  3 ≠ t̲ally fs ? specShape
  ns ← ʰw̲ords d̲isclose 1 s̲elect fs
  rs ← ʰw̲ords d̲isclose 2 s̲elect fs
  ts ← ʰw̲ords d̲isclose 3 s̲elect fs
  n̲ot (1 = t̲ally ns) ∨ (4 = t̲ally ns) ∨ 5 = t̲ally ns ? "name the arrays W A B s (W A B d b for VeRA), or W alone: " c̲at specShape
  n̲ot '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach ns ? "not a name: " c̲at ʰj̲oin ns
  n̲ot (2 = t̲ally rs) ? "write the rank as r and a number or a name: \"r 3\""
  n̲ot ("r" m̲atch d̲isclose f̲irst rs) ∧ (ʰn̲umber? rs ʰn̲th 2) ∨ ʰn̲ame? rs ʰn̲th 2 ? "write the rank as r and a number or a name: \"r 3\""
  n̲ot (1 < t̲ally ts) ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)"
  n̲ot "train" m̲atch d̲isclose f̲irst ts ? "say what trains: \"train A B\" (LoRA), \"train B\" (LoRA-FA) or \"train d b\" (VeRA)"
  ""
}
Used in: ᵐa̲dapter<

ʰv̲era? : Match a => Box a -> Box a -> Char

function (private) · line 72

"vera" if the trained names ts are the fourth and fifth of the arrays as, d and b, else "".

ʰv̲era? ← { as ts →
  (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 4) ∧ ts ʰh̲as? as ʰn̲th 5 ? "vera"
  ""
}
Used in: ʰm̲ethod

ʰm̲ethod : Match a => Box a -> Box a -> Char

function (private) · line 79

The method the trained names ts (after "train") pick for the arrays as: "lora" for A and B, "fa" for B alone, "vera" for d and b (five arrays), "" for anything else.

ʰm̲ethod ← { as ts →
  5 = t̲ally as ? as ʰv̲era? ts
  (2 = t̲ally ts) ∧ (ts ʰh̲as? as ʰn̲th 2) ∧ ts ʰh̲as? as ʰn̲th 3 ? "lora"
  (1 = t̲ally ts) ∧ ts ʰh̲as? as ʰn̲th 3 ? "fa"
  ""
}
Used in: ᵐa̲dapter<

Writing the passes (private)

ʰc̲heck : Any a => (a, Char, Char, Char) -> Char -> Char

function (private) · line 90

The rank check, a statement: A has r rows and B r columns, or the program stops naming the spec.

ʰc̲heck ← { (W, A, B, r) spec →
  "_ := { v -> v ? 0; []P_ANIC \"lo:a_dapter<: A must be " c̲at r c̲at " x n and B m x " c̲at r c̲at " for \\\"" c̲at spec c̲at "\\\"\" } (" c̲at r c̲at " = f_irst s_hape " c̲at A c̲at ") & " c̲at r c̲at " = 2 s_elect s_hape " c̲at B
}
Used in: ᵐa̲dapter<

ʰf̲orward : (Char, Char, Char, Char) -> Char -> Char

function (private) · line 94

The forward pass: W x + s B (A x), right to left, A x first.

ʰf̲orward ← { (W, A, B, s) x →
  "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + " c̲at s c̲at " * " c̲at B c̲at " '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }"
}
Used in: ᵐa̲dapter<

ʰb̲ackLoRA : (Char, Char, Char, Char) -> (Char, Char, Char) -> Char

function (private) · line 100

LoRA's backward pass for inputs x and the output's gradient g: (dA, dB, dx) = (s (B' g) x', s g (A x)', W' g + s A' (B' g)), B' g bound once under b.

ʰb̲ackLoRA ← { (W, A, B, s) (x, g, b) →
  "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n  " c̲at b c̲at " := (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at "\n  (" c̲at s c̲at " * " c̲at b c̲at " '+ '* i_nner o_\\ " c̲at x c̲at ", " c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner " c̲at b c̲at ")\n}"
}
Used in: ʰb̲ack

ʰb̲ackFA : (Char, Char, Char, Char) -> (Char, Char) -> Char

function (private) · line 105

LoRA-FA's: A frozen, its gradient skipped; (dB, dx), x's still back through A.

ʰb̲ackFA ← { (W, A, B, s) (x, g) →
  "{ (" c̲at x c̲at ", " c̲at g c̲at ") -> (" c̲at s c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + " c̲at s c̲at " * (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at ") '+ '* i_nner " c̲at g c̲at ") }"
}
Used in: ʰb̲ack

ʰf̲orwardV : (Char, Char, Char, Char, Char) -> Char -> Char

function (private) · line 111

VeRA's forward pass: W x + b (B (d (A x))), the m x r matrix diag(b) B diag(d) as B times the table of b by d.

ʰf̲orwardV ← { (W, A, B, d, b) x →
  "{ " c̲at x c̲at " -> (" c̲at W c̲at " '+ '* i_nner " c̲at x c̲at ") + (" c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at " }"
}
Used in: ᵐa̲dapter<

ʰb̲ackV : (Char, Char, Char, Char, Char) -> (Char, Char, Char) -> Char

function (private) · line 116

VeRA's backward pass: (dd, db, dx) = (b' Q, Q d, W' g + A' (diag(b) B diag(d))' g), Q = B times g (A x)' item by item, bound once under q.

ʰb̲ackV ← { (W, A, B, d, b) (x, g, q) →
  "{ (" c̲at x c̲at ", " c̲at g c̲at ") ->\n  " c̲at q c̲at " := " c̲at B c̲at " * " c̲at g c̲at " '+ '* i_nner o_\\ " c̲at A c̲at " '+ '* i_nner " c̲at x c̲at "\n  (" c̲at b c̲at " '+ '* i_nner " c̲at q c̲at ", " c̲at q c̲at " '+ '* i_nner " c̲at d c̲at ", ((o_\\ " c̲at W c̲at ") '+ '* i_nner " c̲at g c̲at ") + (o_\\ " c̲at A c̲at ") '+ '* i_nner (o_\\ " c̲at B c̲at " * " c̲at b c̲at " '* t_able " c̲at d c̲at ") '+ '* i_nner " c̲at g c̲at ")\n}"
}
Used in: ᵐa̲dapter<

ʰb̲ack : Char -> ((Char, Char, Char, Char), Char, Char, Char) -> Char

function (private) · line 121

The backward pass the method how picks, LoRA's or LoRA-FA's.

ʰb̲ack ← { how (arrays, x, g, b) →
  "lora" m̲atch how ? arrays ʰb̲ackLoRA (x, g, b)
  arrays ʰb̲ackFA (x, g)
}
Used in: ᵐa̲dapter<

ʰp̲asses : (Box Char, Char, Char, Char) -> Char

function (private) · line 128

The statements: the rank check, then the forward and the backward functions under the names fn.

ʰp̲asses ← { (fn, check, fwd, back) → check c̲at "\n" c̲at (fn ʰn̲th 1) c̲at " := " c̲at fwd c̲at "\n" c̲at (fn ʰn̲th 2) c̲at " := " c̲at back }
Used in: ᵐa̲dapter<

Adapters

ᵐa̲dapter< : Char -> Char -> Char

macro · line 142

An adapted layer from the spec on the right, as statements: the rank check, then the forward function and the backward function under the two names on the left ("u:f_wd u:b_wd"). The forward function takes the inputs (columns) and gives W x + s B (A x); the backward one takes the inputs and the loss's gradient at the output, (x, g), and gives (dA, dB, dx) for "train A B" (LoRA), (dB, dx) for "train B" (LoRA-FA) or (dd, db, dx) for "train d b" (VeRA, whose forward function is W x + b (B (d (A x)))). The lambdas' own names are fresh ([]G_ENSYM, X_eTaL MC30), so a caller's array named x, g or b is not captured.

ᵐa̲dapter< ← { what spec →
  fs ← ʰf̲ields spec
  0 < t̲ally ʰw̲rong fs ? "bad-macro-argument right" ⎕R̲EJECT ʰw̲rong fs
  ts ← 1 d̲rop ʰw̲ords d̲isclose 3 s̲elect fs
  as ← (ʰw̲ords d̲isclose 1 s̲elect fs) ʰa̲rrays ts
  r ← (ʰw̲ords d̲isclose 2 s̲elect fs) ʰn̲th 2
  how ← as ʰm̲ethod ts
  0 = t̲ally how ? "bad-macro-argument right" ⎕R̲EJECT "LoRA trains A and B (\"train " c̲at (as ʰn̲th 2) c̲at " " c̲at (as ʰn̲th 3) c̲at "\"), LoRA-FA B alone (\"train " c̲at (as ʰn̲th 3) c̲at "\"), VeRA d and b of five arrays (\"W A B d b | r 3 | train d b\"); nothing else trains here"
  fn ← ʰw̲ords what
  n̲ot (2 = t̲ally fn) ∧ '∧ r̲/ '{ w → ʰn̲ame? d̲isclose w } e̲ach fn ? "bad-macro-argument left" ⎕R̲EJECT "name the forward and the backward functions: \"u:f_wd u:b_wd\""
  (x, g, b) ← (⎕G̲ENSYM "x", ⎕G̲ENSYM "g", ⎕G̲ENSYM "b")
  check ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, r) ʰc̲heck spec
  "vera" m̲atch how ? ʰp̲asses (fn, check, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰf̲orwardV x, (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4, as ʰn̲th 5) ʰb̲ackV (x, g, b))
  arrays ← (as ʰn̲th 1, as ʰn̲th 2, as ʰn̲th 3, as ʰn̲th 4)
  ʰp̲asses (fn, check, arrays ʰf̲orward x, how ʰb̲ack (arrays, x, g, b))
}
Used in: _