programdemos/lora-explorer/lora-explorer.xtl

LoRA explorer: one 8 x 8 projection adapted by a rank-r update. The frozen W, the trainable A (r x 8) and B (8 x r); the adapted layer y = W x + s B (A x) as a pipeline of equations, drawn as math, its factors kept apart; the update s B A, its rank and the merge; what adapters cost on a large mixture-of-experts model; a backward step, stage by stage; LoRA and LoRA-FA trained side by side to fit a known update, against the best any rank-r update can do (the truncated SVD); W in 4 bits, uniform and NF4 (QLoRA's base); then the rank sweep.

source · imports e: work/libs/Equations/src/Equations.xtl work/libs/Equations/src/Equations.xtlm; flow: libs/Flow/src/Flow.xtlm; lo: libs/LoRA/src/LoRA.xtl libs/LoRA/src/LoRA.xtlm; qz: libs/Quant/src/Quant.xtl

Data

r : Int

value · line 21

The rank, alpha (the scale s is alpha / r), and the sizes for the costs: a SwiGLU expert (d 4096, f 14336), 64 experts a layer, 24 layers.

r ← 3

alpha : Float

value · line 22
alpha ← 6.0
Used in: s, ᵘs̲tart

d : Int

value · line 23
d ← 4096

f : Int

value · line 24
f ← 14336

experts : Int

value · line 25
experts ← 64

layers : Int

value · line 26
layers ← 24

i : Float

value · line 30

W, A, B and the input x by a fixed ripple, the sine of a square (no chance). A real adapter starts with B at 0 and learns it; these are made up, to show the arrays.

i ← f̲loat r̲ange 64
Used in: W, X

W : Float

value · line 31
W ← 8 8 r̲eshape s̲in 0.7 × i × i

j : Float

value · line 32
j ← f̲loat r̲ange r × 8
Used in: A, B

A : Float

value · line 33
A ← (r c̲at 8) r̲eshape 0.4 × c̲os 1.1 × j × j

B : Float

value · line 34
B ← (8 c̲at r) r̲eshape 0.3 × s̲in 0.9 × j × j

X : Float

value · line 35
X ← c̲os 0.4 × 8 t̲ake i

s : Float

value · line 36
s ← alpha ˡᵒs̲caling r

q : Int

value · line 37
q ← r − 1
Used in: y, back, yv, backV

p : Float

value · line 42

The known update to learn, T = P Q (8 x 8, of rank 4), by ripple; the training inputs are the 8 unit vectors (I), so the loss is half the squared distance between s B A and T, entry by entry. Adam at the rate lr, 30 times 10 steps.

p ← f̲loat r̲ange 32
Used in: T

T : Float

value · line 43
T ← (8 4 r̲eshape 0.5 × s̲in 1.3 × p × p) '+ '× i̲nner 4 8 r̲eshape 0.5 × c̲os 0.6 × p × p

I : Float

value · line 44
I ← 8 8 r̲eshape 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

lr : Float

value · line 45
lr ← 0.05

chunks : Int

value · line 46
chunks ← 30
Used in: q1, q2, q3, q4

every : Int

value · line 47
every ← 10
Used in: q1, q2, q3, q4

qk : Int

value · line 49

The quantization's block: 16 numbers (QLoRA's is 64; W has 64).

qk ← 16

The adapted layer

y : Float

value · line 55

x goes two ways: through the frozen W, and through the adapter, A first (r numbers), then B (8); the two are added.

y ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  x ← X
  h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A[k,j] * x[j]"
  u ← "u[i] | 8" ᵉe̲q< "sum k:=0..q | B[i,k] * h[k]"
  g1:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]"
  g1:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g4:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g3:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g2:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g5:grid ← r
  g6:g̲rid ← { k → g5:grid r̲eshape (o̲ffsets k s̲elect g5:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g5:grid) r̲eshape 0 }
  g5:grid r̲eshape (({ @ →
  g7:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
  '+ r̲/₂ g7:grid r̲eshape ((
    ((g2:r̲ank (A, "A", 2, "A[k,j]")) g4:a̲t ((
      ((A, "A", 2, "j") g3:c̲heck (0 + g8:g̲rid 2)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "k") g3:c̲heck (0 + g8:g̲rid 1))))) × 
    ((g2:r̲ank (x, "x", 1, "x[j]")) g4:a̲t (
      ((x, "x", 1, "j") g3:c̲heck (0 + g8:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g11:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g10:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g9:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g12:grid ← 8
  g13:g̲rid ← { k → g12:grid r̲eshape (o̲ffsets k s̲elect g12:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g12:grid) r̲eshape 0 }
  g12:grid r̲eshape (({ @ →
  g14:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
  '+ r̲/₂ g14:grid r̲eshape ((
    ((g9:r̲ank (B, "B", 2, "B[i,k]")) g11:a̲t ((
      ((B, "B", 2, "k") g10:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape B) × 
      ((B, "B", 1, "i") g10:c̲heck (0 + g15:g̲rid 1))))) × 
    ((g9:r̲ank (h, "h", 1, "h[k]")) g11:a̲t (
      ((h, "h", 1, "k") g10:c̲heck (0 + g15:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g18:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g17:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g16:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g19:grid ← 8
  g20:g̲rid ← { k → g19:grid r̲eshape (o̲ffsets k s̲elect g19:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g19:grid) r̲eshape 0 }
  g19:grid r̲eshape ((({ @ →
  g21:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
  '+ r̲/₂ g21:grid r̲eshape ((
    ((g16:r̲ank (W, "W", 2, "W[i,j]")) g18:a̲t ((
      ((W, "W", 2, "j") g17:c̲heck (0 + g22:g̲rid 2)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g17:c̲heck (0 + g22:g̲rid 1))))) × 
    ((g16:r̲ank (x, "x", 1, "x[j]")) g18:a̲t (
      ((x, "x", 1, "j") g17:c̲heck (0 + g22:g̲rid 2))))))
} @) + (s × 
    ((g16:r̲ank (u, "u", 1, "u[i]")) g18:a̲t (
      ((u, "u", 1, "i") g17:c̲heck (0 + g20:g̲rid 1)))))))
} @))

drawn : Char

value · line 61

The same pipeline, drawn as math, a row a stage.

drawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[i] | 8" ᵉt̲ex< "sum k:=0..q | B[i,k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A_{k,j} \\, x_{j}")
ᵉt̲ex< expands to
("u_{i} = \\sum_{k=0}^{q} B_{i,k} \\, h_{k}")
ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + s \\, u_{i}")

The update and the merge

delta : Float

value · line 73

The update s B A (8 x 8, of rank r at most), the merged matrix W + s B A, the update's rank, and how far the merged layer's output is from the factored one (under 1e-12: 1).

delta ← ˡᵒd̲elta (A, B, s)

merged : Float

value · line 74
merged ← ˡᵒm̲erge (W, A, B, s)

On a large model

n : Int

value · line 86

One expert's adapters (gate, up, down), then their MiB in BF16: one expert, a layer of experts, every layer.

n ← r ˡᵒe̲xpert d c̲at f

mib : Float

value · line 87
mib ← (16 ˡᵒb̲ytes n) ÷ 1048576.0

A backward step

h : Float

value · line 97

The loss's gradient at the output for x is g = y - (W + T) x (the known update's output wanted). Back through the layer: B' g once (r numbers), then the gradients of A and B (outer products, s b x' and s g h') and of x, through W and through the adapter.

h ← A '+ '× i̲nner X
Used in: y, back, yv

G : Float

value · line 98
G ← y − (W + T) '+ '× i̲nner X
Used in: back, dx

back : Float

value · line 99
back ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g ← G
  b ← "b[k] | r" ᵉe̲q< "sum i:=0..7 | B[i,k] * g[i]"
  dA ← "dA[k,j] | r 8" ᵉe̲q< "s * b[k] * X[j]"
  dB ← "dB[i,k] | 8 r" ᵉe̲q< "s * g[i] * h[k]"
  g23:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])"
  g23:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g26:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g25:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g24:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g27:grid ← r
  g28:g̲rid ← { k → g27:grid r̲eshape (o̲ffsets k s̲elect g27:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g27:grid) r̲eshape 0 }
  g27:grid r̲eshape (({ @ →
  g29:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g30:g̲rid ← { k → g29:grid r̲eshape (o̲ffsets k s̲elect g29:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g29:grid) r̲eshape 0 }
  '+ r̲/₂ g29:grid r̲eshape ((
    ((g24:r̲ank (B, "B", 2, "B[i,k]")) g26:a̲t ((
      ((B, "B", 2, "k") g25:c̲heck (0 + g30:g̲rid 1)) + (2 s̲elect s̲hape B) × 
      ((B, "B", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))) × 
    ((g24:r̲ank (g, "g", 1, "g[i]")) g26:a̲t (
      ((g, "g", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g33:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g32:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g31:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g34:grid ← r c̲at 8
  g35:g̲rid ← { k → g34:grid r̲eshape (o̲ffsets k s̲elect g34:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g34:grid) r̲eshape 0 }
  g34:grid r̲eshape (((s × 
    ((g31:r̲ank (b, "b", 1, "b[k]")) g33:a̲t (
      ((b, "b", 1, "k") g32:c̲heck (0 + g35:g̲rid 1))))) × 
    ((g31:r̲ank (X, "X", 1, "X[j]")) g33:a̲t (
      ((X, "X", 1, "j") g32:c̲heck (0 + g35:g̲rid 2))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g38:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g37:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g36:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g39:grid ← 8 c̲at r
  g40:g̲rid ← { k → g39:grid r̲eshape (o̲ffsets k s̲elect g39:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g39:grid) r̲eshape 0 }
  g39:grid r̲eshape (((s × 
    ((g36:r̲ank (g, "g", 1, "g[i]")) g38:a̲t (
      ((g, "g", 1, "i") g37:c̲heck (0 + g40:g̲rid 1))))) × 
    ((g36:r̲ank (h, "h", 1, "h[k]")) g38:a̲t (
      ((h, "h", 1, "k") g37:c̲heck (0 + g40:g̲rid 2))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g43:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g42:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g41:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g44:grid ← 8
  g45:g̲rid ← { k → g44:grid r̲eshape (o̲ffsets k s̲elect g44:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g44:grid) r̲eshape 0 }
  g44:grid r̲eshape ((({ @ →
  g46:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g47:g̲rid ← { k → g46:grid r̲eshape (o̲ffsets k s̲elect g46:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g46:grid) r̲eshape 0 }
  '+ r̲/₂ g46:grid r̲eshape ((
    ((g41:r̲ank (W, "W", 2, "W[i,j]")) g43:a̲t ((
      ((W, "W", 2, "j") g42:c̲heck (0 + g47:g̲rid 1)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))) × 
    ((g41:r̲ank (g, "g", 1, "g[i]")) g43:a̲t (
      ((g, "g", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))))
} @) + (s × ({ @ →
  g48:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g49:g̲rid ← { k → g48:grid r̲eshape (o̲ffsets k s̲elect g48:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g48:grid) r̲eshape 0 }
  '+ r̲/₂ g48:grid r̲eshape ((
    ((g41:r̲ank (A, "A", 2, "A[k,j]")) g43:a̲t ((
      ((A, "A", 2, "j") g42:c̲heck (0 + g49:g̲rid 1)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))) × 
    ((g41:r̲ank (b, "b", 1, "b[k]")) g43:a̲t (
      ((b, "b", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))))
} @))))
} @))

backDrawn : Char

value · line 106

The same, drawn as math.

backDrawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"G"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("b[k] | r" ᵉt̲ex< "sum i:=0..7 | B[i,k] * g[i]") c̲at r#" \qquad \triangleright \, b \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dA[k,j] | r 8" ᵉt̲ex< "s * b[k] * X[j]") c̲at r#" \qquad \triangleright \, dA \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dB[i,k] | 8 r" ᵉt̲ex< "s * g[i] * h[k]") c̲at r#" \qquad \triangleright \, dB \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("b_{k} = \\sum_{i=0}^{7} B_{i,k} \\, g_{i}")
ᵉt̲ex< expands to
("dA_{k,j} = s \\, b_{k} \\, X_{j}")
ᵉt̲ex< expands to
("dB_{i,k} = s \\, g_{i} \\, h_{k}")
ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + s \\, \\left(\\sum_{k=0}^{q} A_{k,j} \\, b_{k}\\right)")

_ : Int

function · line 115

The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.

"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
  g50:b ← (o̲\ B) '+ '× i̲nner g51:g
  (s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}

ᵘf̲orward : Float -> Float

function · line 115

The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.

"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
  g50:b ← (o̲\ B) '+ '× i̲nner g51:g
  (s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}

ᵘb̲ackward : (Float, Float) -> (Float, Float, Float)

function · line 115

The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.

"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
  g50:b ← (o̲\ B) '+ '× i̲nner g51:g
  (s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}
Used in: dx

dx : Float

value · line 116
(_, _, dx) ← ᵘb̲ackward (8 1 r̲eshape X, 8 1 r̲eshape G)

Training: LoRA beside LoRA-FA

a0 : (Float, Float, Float, Float)

value · line 123

Both start as LoRA does, B at 0, from the same A (seeded); LoRA trains A and B, LoRA-FA B alone. The loss after every 10 steps.

a0 ← (W, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s)
Used in: q1, q2

Y : Float

value · line 124
Y ← (W + T) '+ '× i̲nner I

ᵘl̲oss : (Float, Float, Float, Float) -> Float

function · line 125
ᵘl̲oss ← { a → a ˡᵒl̲oss (I, Y) }
Used in: q1, q2, q3, ᵘl̲ora, ᵘf̲a

q1 : ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)

value · line 126
(q1, L1) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStart a0, ᵘl̲oss a0)
Used in: q1, A1

L1 : Float

value · line 126
(q1, L1) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStart a0, ᵘl̲oss a0)

q2 : ((Float, Float, Float, Float), (Float, Float), Float)

value · line 131
(q2, L2) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStartFA a0, ᵘl̲oss a0)
Used in: q2, A2

L2 : Float

value · line 131
(q2, L2) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStartFA a0, ᵘl̲oss a0)

u0 : Float

value · line 141

The best any update of rank k can do (Eckart-Young): half the sum of T's squared singular values past the k-th. They are the eigenvalues of T' T, found by power iteration, each taken out once found.

u0 ← 1.0 + 0.1 × f̲loat r̲ange 8
Used in: ᵘb̲est

ᵘb̲est : Float -> Float

function · line 142
ᵘb̲est ← { D →
  (_, ev) ← 8 '{ (M, e) →
    v ← 300 '{ w → m ← M '+ '× i̲nner w
      m ÷ 0.000000000000000001 + s̲qrt '+ r̲/ m × m } p̲ower u0
    lam ← '+ r̲/ v × M '+ '× i̲nner v
    (M − lam × v '× t̲able v, e c̲at lam)
  } p̲ower ((o̲\ D) '+ '× i̲nner D, 0 r̲eshape 0.0)
  0.0 m̲ax 0.5 × '{ k → '+ r̲/ k d̲rop ev } e̲ach r̲ange 8
}

best : Float

value · line 151
best ← ᵘb̲est T

A1 : Float

value · line 154

What each learned: LoRA's update, LoRA-FA's.

((_, A1, B1, s1), _, _) ← q1
Used in: A1, learned

B1 : Float

value · line 154

What each learned: LoRA's update, LoRA-FA's.

((_, A1, B1, s1), _, _) ← q1
Used in: A1, learned

s1 : Float

value · line 154

What each learned: LoRA's update, LoRA-FA's.

((_, A1, B1, s1), _, _) ← q1
Used in: A1, learned

A2 : Float

value · line 155
((_, A2, B2, s2), _, _) ← q2
Used in: A2, learnedFA

B2 : Float

value · line 155
((_, A2, B2, s2), _, _) ← q2
Used in: A2, learnedFA

s2 : Float

value · line 155
((_, A2, B2, s2), _, _) ← q2
Used in: A2, learnedFA

learned : Float

value · line 156
learned ← ˡᵒd̲elta (A1, B1, s1)

learnedFA : Float

value · line 157
learnedFA ← ˡᵒd̲elta (A2, B2, s2)

W in 4 bits

Wu : Float

value · line 168

QLoRA keeps the frozen W in 4 bits and computes with it dequantized. W in blocks of qk, each with its scale (its largest size): uniform 4 bits (rounded to -7 .. 7 steps of the scale / 7) and NF4 (the nearest of 16 levels at the normal's quantiles). Their RMSE against W, then against a normal 8 x 8 matrix Z (Box-Muller on the ripple): W's numbers are a sine, piled up near its ends, so uniform does better; on normal numbers NF4 does.

Wu ← (4, qk) qz:u̲niform W

Wn : Float

value · line 169
Wn ← qk qz:n̲f4 W

k2 : Float

value · line 170
k2 ← f̲loat r̲ange 64
Used in: Z

Z : Float

value · line 171
Z ← 8 8 r̲eshape (s̲qrt -2.0 × l̲og 0.0001 m̲ax 0.5 + 0.5 × s̲in 78.233 × k2 × k2) × c̲os 6.283185307179586 × 0.5 + 0.5 × s̲in 37.719 × k2 × k2

ᵘr̲mse : Float -> Float -> Float

function · line 172
ᵘr̲mse ← { a b → s̲qrt ('+ r̲/ r̲avel (a − b) × a − b) ÷ f̲loat t̲ally r̲avel a }

QLoRA: the adapter on the 4-bit base

Q : (Int, Float, Int)

value · line 184

The base stored in NF4 (blocks of qk) and trained on as LoRA is, the same start and steps, the base dequantized (W~, Wn above): its loss every 10 steps. W~ is W plus the 4-bit error, so what the adapter has to learn is T less it; the best rank r can do for that.

Q ← qk ˡᵒq̲base W

q3 : ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)

value · line 185
(q3, L3) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStart (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s), ᵘl̲oss (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s))
Used in: q3, A3

L3 : Float

value · line 185
(q3, L3) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at ᵘl̲oss a)
} p̲ower (ˡᵒa̲damStart (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s), ᵘl̲oss (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s))

A3 : Float

value · line 194

The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.

((_, A3, B3, s3), _, _) ← q3

B3 : Float

value · line 194

The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.

((_, A3, B3, s3), _, _) ← q3

s3 : Float

value · line 194

The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.

((_, A3, B3, s3), _, _) ← q3

learnedQ : Float

value · line 195
learnedQ ← ˡᵒd̲elta (A3, B3, s3)

Mq : Float

value · line 196
Mq ← ˡᵒd̲equant qk ˡᵒm̲ergeQ (Q, A3, B3, s3)

bits : Float

value · line 201

Bits a weight: 4-bit codes with a 32-bit scale per block, then the scales double-quantized (8 bits, blocks of 256); a 4096 x 14336 projection's MiB in FP32, FP16, NF4 and NF4 double-quantized.

bits ← (qk, 256) qz:b̲its 4

VeRA: shared random projections, two vectors

A0 : Float

value · line 212

VeRA (Kopiczko, Blankevoort and Asano, 2024) freezes A and B too: A0 (r x 8) and B0 (8 x r) are random, regenerated from a seed and shared by every layer of the shape, and it trains two vectors, d (r numbers) scaling A0 x and b (8) scaling B0's output: y = W x + b (B0 (d (A0 x))). Forward and back as pipelines, at made-up vectors dv and bv (a real adapter starts with b at 0).

(_, A0, B0, _, _) ← 1 ˡᵒv̲era (W, r)
Used in: A0, yv, hv, backV, _, learnedV

B0 : Float

value · line 212

VeRA (Kopiczko, Blankevoort and Asano, 2024) freezes A and B too: A0 (r x 8) and B0 (8 x r) are random, regenerated from a seed and shared by every layer of the shape, and it trains two vectors, d (r numbers) scaling A0 x and b (8) scaling B0's output: y = W x + b (B0 (d (A0 x))). Forward and back as pipelines, at made-up vectors dv and bv (a real adapter starts with b at 0).

(_, A0, B0, _, _) ← 1 ˡᵒv̲era (W, r)
Used in: A0, yv, vv, backV, _, learnedV

dv : Float

value · line 213
dv ← 0.5 + 0.3 × c̲os 0.8 × f̲loat r̲ange r
Used in: yv, vv, backV, _

bv : Float

value · line 214
bv ← 0.6 × s̲in 1.7 × f̲loat r̲ange 8
Used in: yv, backV, _

yv : Float

value · line 215
yv ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  x ← X
  h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A0[k,j] * x[j]"
  u ← "u[k] | r" ᵉe̲q< "dv[k] * h[k]"
  v ← "v[i] | 8" ᵉe̲q< "sum k:=0..q | B0[i,k] * u[k]"
  g53:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]"
  g53:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g56:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g55:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g54:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g57:grid ← r
  g58:g̲rid ← { k → g57:grid r̲eshape (o̲ffsets k s̲elect g57:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g57:grid) r̲eshape 0 }
  g57:grid r̲eshape (({ @ →
  g59:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g60:g̲rid ← { k → g59:grid r̲eshape (o̲ffsets k s̲elect g59:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g59:grid) r̲eshape 0 }
  '+ r̲/₂ g59:grid r̲eshape ((
    ((g54:r̲ank (A0, "A0", 2, "A0[k,j]")) g56:a̲t ((
      ((A0, "A0", 2, "j") g55:c̲heck (0 + g60:g̲rid 2)) + (2 s̲elect s̲hape A0) × 
      ((A0, "A0", 1, "k") g55:c̲heck (0 + g60:g̲rid 1))))) × 
    ((g54:r̲ank (x, "x", 1, "x[j]")) g56:a̲t (
      ((x, "x", 1, "j") g55:c̲heck (0 + g60:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g63:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g62:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g61:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g64:grid ← r
  g65:g̲rid ← { k → g64:grid r̲eshape (o̲ffsets k s̲elect g64:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g64:grid) r̲eshape 0 }
  g64:grid r̲eshape ((
    ((g61:r̲ank (dv, "dv", 1, "dv[k]")) g63:a̲t (
      ((dv, "dv", 1, "k") g62:c̲heck (0 + g65:g̲rid 1)))) × 
    ((g61:r̲ank (h, "h", 1, "h[k]")) g63:a̲t (
      ((h, "h", 1, "k") g62:c̲heck (0 + g65:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g68:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g67:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g66:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g69:grid ← 8
  g70:g̲rid ← { k → g69:grid r̲eshape (o̲ffsets k s̲elect g69:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g69:grid) r̲eshape 0 }
  g69:grid r̲eshape (({ @ →
  g71:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g72:g̲rid ← { k → g71:grid r̲eshape (o̲ffsets k s̲elect g71:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g71:grid) r̲eshape 0 }
  '+ r̲/₂ g71:grid r̲eshape ((
    ((g66:r̲ank (B0, "B0", 2, "B0[i,k]")) g68:a̲t ((
      ((B0, "B0", 2, "k") g67:c̲heck (0 + g72:g̲rid 2)) + (2 s̲elect s̲hape B0) × 
      ((B0, "B0", 1, "i") g67:c̲heck (0 + g72:g̲rid 1))))) × 
    ((g66:r̲ank (u, "u", 1, "u[k]")) g68:a̲t (
      ((u, "u", 1, "k") g67:c̲heck (0 + g72:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g75:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g74:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g73:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g76:grid ← 8
  g77:g̲rid ← { k → g76:grid r̲eshape (o̲ffsets k s̲elect g76:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g76:grid) r̲eshape 0 }
  g76:grid r̲eshape ((({ @ →
  g78:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g79:g̲rid ← { k → g78:grid r̲eshape (o̲ffsets k s̲elect g78:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g78:grid) r̲eshape 0 }
  '+ r̲/₂ g78:grid r̲eshape ((
    ((g73:r̲ank (W, "W", 2, "W[i,j]")) g75:a̲t ((
      ((W, "W", 2, "j") g74:c̲heck (0 + g79:g̲rid 2)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g74:c̲heck (0 + g79:g̲rid 1))))) × 
    ((g73:r̲ank (x, "x", 1, "x[j]")) g75:a̲t (
      ((x, "x", 1, "j") g74:c̲heck (0 + g79:g̲rid 2))))))
} @) + (
    ((g73:r̲ank (bv, "bv", 1, "bv[i]")) g75:a̲t (
      ((bv, "bv", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))) × 
    ((g73:r̲ank (v, "v", 1, "v[i]")) g75:a̲t (
      ((v, "v", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))))))
} @))

drawnV : Char

value · line 222

The same, drawn as math.

drawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A0[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[k] | r" ᵉt̲ex< "dv[k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("v[i] | 8" ᵉt̲ex< "sum k:=0..q | B0[i,k] * u[k]") c̲at r#" \qquad \triangleright \, v \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A0_{k,j} \\, x_{j}")
ᵉt̲ex< expands to
("u_{k} = dv_{k} \\, h_{k}")
ᵉt̲ex< expands to
("v_{i} = \\sum_{k=0}^{q} B0_{i,k} \\, u_{k}")
ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + bv_{i} \\, v_{i}")

hv : Float

value · line 233

Back, for the gradient g at the output: c = B0' (b g) once (r numbers), then dd = c (A0 x) and db = g (B0 (d (A0 x))), item by item, and dx through W and through the adapter. The projections are in every gradient and get none.

hv ← A0 '+ '× i̲nner X
Used in: vv, backV

vv : Float

value · line 234
vv ← B0 '+ '× i̲nner dv × hv
Used in: backV

Gv : Float

value · line 235
Gv ← yv − (W + T) '+ '× i̲nner X
Used in: backV, dxv

backV : Float

value · line 236
backV ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g ← Gv
  c ← "c[k] | r" ᵉe̲q< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]"
  dd ← "dd[k] | r" ᵉe̲q< "c[k] * hv[k]"
  db ← "db[i] | 8" ᵉe̲q< "g[i] * vv[i]"
  g80:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])"
  g80:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g83:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g82:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g81:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g84:grid ← r
  g85:g̲rid ← { k → g84:grid r̲eshape (o̲ffsets k s̲elect g84:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g84:grid) r̲eshape 0 }
  g84:grid r̲eshape (({ @ →
  g86:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g87:g̲rid ← { k → g86:grid r̲eshape (o̲ffsets k s̲elect g86:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g86:grid) r̲eshape 0 }
  '+ r̲/₂ g86:grid r̲eshape (((
    ((g81:r̲ank (B0, "B0", 2, "B0[i,k]")) g83:a̲t ((
      ((B0, "B0", 2, "k") g82:c̲heck (0 + g87:g̲rid 1)) + (2 s̲elect s̲hape B0) × 
      ((B0, "B0", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) × 
    ((g81:r̲ank (bv, "bv", 1, "bv[i]")) g83:a̲t (
      ((bv, "bv", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) × 
    ((g81:r̲ank (g, "g", 1, "g[i]")) g83:a̲t (
      ((g, "g", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g90:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g89:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g88:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g91:grid ← r
  g92:g̲rid ← { k → g91:grid r̲eshape (o̲ffsets k s̲elect g91:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g91:grid) r̲eshape 0 }
  g91:grid r̲eshape ((
    ((g88:r̲ank (c, "c", 1, "c[k]")) g90:a̲t (
      ((c, "c", 1, "k") g89:c̲heck (0 + g92:g̲rid 1)))) × 
    ((g88:r̲ank (hv, "hv", 1, "hv[k]")) g90:a̲t (
      ((hv, "hv", 1, "k") g89:c̲heck (0 + g92:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g95:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g94:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g93:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g96:grid ← 8
  g97:g̲rid ← { k → g96:grid r̲eshape (o̲ffsets k s̲elect g96:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g96:grid) r̲eshape 0 }
  g96:grid r̲eshape ((
    ((g93:r̲ank (g, "g", 1, "g[i]")) g95:a̲t (
      ((g, "g", 1, "i") g94:c̲heck (0 + g97:g̲rid 1)))) × 
    ((g93:r̲ank (vv, "vv", 1, "vv[i]")) g95:a̲t (
      ((vv, "vv", 1, "i") g94:c̲heck (0 + g97:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g100:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g99:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g98:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g101:grid ← 8
  g102:g̲rid ← { k → g101:grid r̲eshape (o̲ffsets k s̲elect g101:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g101:grid) r̲eshape 0 }
  g101:grid r̲eshape ((({ @ →
  g103:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g104:g̲rid ← { k → g103:grid r̲eshape (o̲ffsets k s̲elect g103:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g103:grid) r̲eshape 0 }
  '+ r̲/₂ g103:grid r̲eshape ((
    ((g98:r̲ank (W, "W", 2, "W[i,j]")) g100:a̲t ((
      ((W, "W", 2, "j") g99:c̲heck (0 + g104:g̲rid 1)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))) × 
    ((g98:r̲ank (g, "g", 1, "g[i]")) g100:a̲t (
      ((g, "g", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))))
} @) + ({ @ →
  g105:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g106:g̲rid ← { k → g105:grid r̲eshape (o̲ffsets k s̲elect g105:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g105:grid) r̲eshape 0 }
  '+ r̲/₂ g105:grid r̲eshape (((
    ((g98:r̲ank (A0, "A0", 2, "A0[k,j]")) g100:a̲t ((
      ((A0, "A0", 2, "j") g99:c̲heck (0 + g106:g̲rid 1)) + (2 s̲elect s̲hape A0) × 
      ((A0, "A0", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) × 
    ((g98:r̲ank (dv, "dv", 1, "dv[k]")) g100:a̲t (
      ((dv, "dv", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) × 
    ((g98:r̲ank (c, "c", 1, "c[k]")) g100:a̲t (
      ((c, "c", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))))
} @)))
} @))

backDrawnV : Char

value · line 243

The same, drawn as math.

backDrawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"Gv"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("c[k] | r" ᵉt̲ex< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]") c̲at r#" \qquad \triangleright \, c \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dd[k] | r" ᵉt̲ex< "c[k] * hv[k]") c̲at r#" \qquad \triangleright \, dd \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("db[i] | 8" ᵉt̲ex< "g[i] * vv[i]") c̲at r#" \qquad \triangleright \, db \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("c_{k} = \\sum_{i=0}^{7} B0_{i,k} \\, bv_{i} \\, g_{i}")
ᵉt̲ex< expands to
("dd_{k} = c_{k} \\, hv_{k}")
ᵉt̲ex< expands to
("db_{i} = g_{i} \\, vv_{i}")
ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + \\left(\\sum_{k=0}^{q} A0_{k,j} \\, dv_{k} \\, c_{k}\\right)")

_ : Int

function · line 251

The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.

"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
  g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
  (bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}

ᵘf̲orwardV : Float -> Float

function · line 251

The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.

"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
  g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
  (bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}

ᵘb̲ackwardV : (Float, Float) -> (Float, Float, Float)

function · line 251

The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.

"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
  g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
  (bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}
Used in: dxv

dxv : Float

value · line 252
(_, _, dxv) ← ᵘb̲ackwardV (8 1 r̲eshape X, 8 1 r̲eshape Gv)

V : (Float, Float, Float, Float, Float)

value · line 256

VeRA trained beside LoRA, from its start (b at 0, d at 0.1), on the same known update and steps: its loss every 10 steps.

V ← 1 ˡᵒv̲era (W, r)
Used in: q4

q4 : ((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float)

value · line 257
(q4, L4) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at a ˡᵒl̲ossV (I, Y))
} p̲ower (ˡᵒa̲damStartV V, V ˡᵒl̲ossV (I, Y))
Used in: q4, d4

L4 : Float

value · line 257
(q4, L4) ← chunks '{ (z, L) →
  z2 ← every '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower z
  (a, _, _) ← z2
  (z2, L c̲at a ˡᵒl̲ossV (I, Y))
} p̲ower (ˡᵒa̲damStartV V, V ˡᵒl̲ossV (I, Y))

d4 : Float

value · line 263
((_, _, _, d4, b4), _, _) ← q4
Used in: d4, learnedV

b4 : Float

value · line 263
((_, _, _, d4, b4), _, _) ← q4
Used in: d4, learnedV

learnedV : Float

value · line 264
learnedV ← ˡᵒd̲eltaV (A0, B0, d4, b4)

The rank sweep

ᵘs̲tart : Int -> (Float, Float, Float, Float)

function · line 280

LoRA and LoRA-FA trained at each rank 1 .. 8 (300 steps each) from the same start: LoRA reaches the best above at every rank; LoRA-FA, its update held in A's row space, does not; nor does VeRA, which trains 8 + k numbers.

ᵘs̲tart ← { k → (W, 1 ˡᵒs̲eeded k c̲at 8, (8 c̲at k) r̲eshape 0.0, alpha ˡᵒs̲caling k) }
Used in: ᵘl̲ora, ᵘf̲a

ᵘl̲ora : Int -> Float

function · line 281
ᵘl̲ora ← { k →
  (a, _, _) ← 300 '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower ˡᵒa̲damStart ᵘs̲tart k
  ᵘl̲oss a
}

ᵘv̲era : Int -> Float

function · line 285
ᵘv̲era ← { k →
  (a, _, _) ← 300 '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower ˡᵒa̲damStartV 1 ˡᵒv̲era (W, k)
  a ˡᵒl̲ossV (I, Y)
}

ᵘf̲a : Int -> Float

function · line 289
ᵘf̲a ← { k →
  (a, _, _) ← 300 '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower ˡᵒa̲damStartFA ᵘs̲tart k
  ᵘl̲oss a
}