programdemos/lora-explorer/lora-explorer.xtl
LoRA explorer: one 8 x 8 projection adapted by a rank-r update. The frozen W, the trainable A (r x 8) and B (8 x r); the adapted layer y = W x + s B (A x) as a pipeline of equations, drawn as math, its factors kept apart; the update s B A, its rank and the merge; what adapters cost on a large mixture-of-experts model; a backward step, stage by stage; LoRA and LoRA-FA trained side by side to fit a known update, against the best any rank-r update can do (the truncated SVD); W in 4 bits, uniform and NF4 (QLoRA's base); then the rank sweep.
Data
r : Int
The rank, alpha (the scale s is alpha / r), and the sizes for the costs: a SwiGLU expert (d 4096, f 14336), 64 experts a layer, 24 layers.
r ← 3
experts : Int
experts ← 64
i : Float
W, A, B and the input x by a fixed ripple, the sine of a square (no chance). A real adapter starts with B at 0 and learns it; these are made up, to show the arrays.
i ← f̲loat r̲ange 64
s : Float
s ← alpha ˡᵒs̲caling r
p : Float
The known update to learn, T = P Q (8 x 8, of rank 4), by ripple; the training inputs are the 8 unit vectors (I), so the loss is half the squared distance between s B A and T, entry by entry. Adam at the rate lr, 30 times 10 steps.
p ← f̲loat r̲ange 32
T : Float
T ← (8 4 r̲eshape 0.5 × s̲in 1.3 × p × p) '+ '× i̲nner 4 8 r̲eshape 0.5 × c̲os 0.6 × p × p
The adapted layer
y : Float
x goes two ways: through the frozen W, and through the adapter, A first (r numbers), then B (8); the two are added.
y ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
x ← X
h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A[k,j] * x[j]"
u ← "u[i] | 8" ᵉe̲q< "sum k:=0..q | B[i,k] * h[k]"
g1:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]"
g1:stage
} @)ᵉe̲q< expands to
(({ @ →
g4:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g3:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g2:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g5:grid ← r
g6:g̲rid ← { k → g5:grid r̲eshape (o̲ffsets k s̲elect g5:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g5:grid) r̲eshape 0 }
g5:grid r̲eshape (({ @ →
g7:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
'+ r̲/₂ g7:grid r̲eshape ((
((g2:r̲ank (A, "A", 2, "A[k,j]")) g4:a̲t ((
((A, "A", 2, "j") g3:c̲heck (0 + g8:g̲rid 2)) + (2 s̲elect s̲hape A) ×
((A, "A", 1, "k") g3:c̲heck (0 + g8:g̲rid 1))))) ×
((g2:r̲ank (x, "x", 1, "x[j]")) g4:a̲t (
((x, "x", 1, "j") g3:c̲heck (0 + g8:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g11:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g10:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g9:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g12:grid ← 8
g13:g̲rid ← { k → g12:grid r̲eshape (o̲ffsets k s̲elect g12:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g12:grid) r̲eshape 0 }
g12:grid r̲eshape (({ @ →
g14:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
'+ r̲/₂ g14:grid r̲eshape ((
((g9:r̲ank (B, "B", 2, "B[i,k]")) g11:a̲t ((
((B, "B", 2, "k") g10:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape B) ×
((B, "B", 1, "i") g10:c̲heck (0 + g15:g̲rid 1))))) ×
((g9:r̲ank (h, "h", 1, "h[k]")) g11:a̲t (
((h, "h", 1, "k") g10:c̲heck (0 + g15:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g18:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g17:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g16:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g19:grid ← 8
g20:g̲rid ← { k → g19:grid r̲eshape (o̲ffsets k s̲elect g19:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g19:grid) r̲eshape 0 }
g19:grid r̲eshape ((({ @ →
g21:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
'+ r̲/₂ g21:grid r̲eshape ((
((g16:r̲ank (W, "W", 2, "W[i,j]")) g18:a̲t ((
((W, "W", 2, "j") g17:c̲heck (0 + g22:g̲rid 2)) + (2 s̲elect s̲hape W) ×
((W, "W", 1, "i") g17:c̲heck (0 + g22:g̲rid 1))))) ×
((g16:r̲ank (x, "x", 1, "x[j]")) g18:a̲t (
((x, "x", 1, "j") g17:c̲heck (0 + g22:g̲rid 2))))))
} @) + (s ×
((g16:r̲ank (u, "u", 1, "u[i]")) g18:a̲t (
((u, "u", 1, "i") g17:c̲heck (0 + g20:g̲rid 1)))))))
} @))drawn : Char
The same pipeline, drawn as math, a row a stage.
drawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[i] | 8" ᵉt̲ex< "sum k:=0..q | B[i,k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A_{k,j} \\, x_{j}")ᵉt̲ex< expands to
("u_{i} = \\sum_{k=0}^{q} B_{i,k} \\, h_{k}")ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + s \\, u_{i}")The update and the merge
delta : Float
The update s B A (8 x 8, of rank r at most), the merged matrix W + s B A, the update's rank, and how far the merged layer's output is from the factored one (under 1e-12: 1).
delta ← ˡᵒd̲elta (A, B, s)
On a large model
n : Int
One expert's adapters (gate, up, down), then their MiB in BF16: one expert, a layer of experts, every layer.
n ← r ˡᵒe̲xpert d c̲at f
A backward step
h : Float
The loss's gradient at the output for x is g = y - (W + T) x (the known update's output wanted). Back through the layer: B' g once (r numbers), then the gradients of A and B (outer products, s b x' and s g h') and of x, through W and through the adapter.
h ← A '+ '× i̲nner X
back : Float
back ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
g ← G
b ← "b[k] | r" ᵉe̲q< "sum i:=0..7 | B[i,k] * g[i]"
dA ← "dA[k,j] | r 8" ᵉe̲q< "s * b[k] * X[j]"
dB ← "dB[i,k] | 8 r" ᵉe̲q< "s * g[i] * h[k]"
g23:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])"
g23:stage
} @)ᵉe̲q< expands to
(({ @ →
g26:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g25:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g24:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g27:grid ← r
g28:g̲rid ← { k → g27:grid r̲eshape (o̲ffsets k s̲elect g27:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g27:grid) r̲eshape 0 }
g27:grid r̲eshape (({ @ →
g29:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g30:g̲rid ← { k → g29:grid r̲eshape (o̲ffsets k s̲elect g29:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g29:grid) r̲eshape 0 }
'+ r̲/₂ g29:grid r̲eshape ((
((g24:r̲ank (B, "B", 2, "B[i,k]")) g26:a̲t ((
((B, "B", 2, "k") g25:c̲heck (0 + g30:g̲rid 1)) + (2 s̲elect s̲hape B) ×
((B, "B", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))) ×
((g24:r̲ank (g, "g", 1, "g[i]")) g26:a̲t (
((g, "g", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g33:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g32:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g31:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g34:grid ← r c̲at 8
g35:g̲rid ← { k → g34:grid r̲eshape (o̲ffsets k s̲elect g34:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g34:grid) r̲eshape 0 }
g34:grid r̲eshape (((s ×
((g31:r̲ank (b, "b", 1, "b[k]")) g33:a̲t (
((b, "b", 1, "k") g32:c̲heck (0 + g35:g̲rid 1))))) ×
((g31:r̲ank (X, "X", 1, "X[j]")) g33:a̲t (
((X, "X", 1, "j") g32:c̲heck (0 + g35:g̲rid 2))))))
} @))ᵉe̲q< expands to
(({ @ →
g38:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g37:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g36:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g39:grid ← 8 c̲at r
g40:g̲rid ← { k → g39:grid r̲eshape (o̲ffsets k s̲elect g39:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g39:grid) r̲eshape 0 }
g39:grid r̲eshape (((s ×
((g36:r̲ank (g, "g", 1, "g[i]")) g38:a̲t (
((g, "g", 1, "i") g37:c̲heck (0 + g40:g̲rid 1))))) ×
((g36:r̲ank (h, "h", 1, "h[k]")) g38:a̲t (
((h, "h", 1, "k") g37:c̲heck (0 + g40:g̲rid 2))))))
} @))ᵉe̲q< expands to
(({ @ →
g43:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g42:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g41:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g44:grid ← 8
g45:g̲rid ← { k → g44:grid r̲eshape (o̲ffsets k s̲elect g44:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g44:grid) r̲eshape 0 }
g44:grid r̲eshape ((({ @ →
g46:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g47:g̲rid ← { k → g46:grid r̲eshape (o̲ffsets k s̲elect g46:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g46:grid) r̲eshape 0 }
'+ r̲/₂ g46:grid r̲eshape ((
((g41:r̲ank (W, "W", 2, "W[i,j]")) g43:a̲t ((
((W, "W", 2, "j") g42:c̲heck (0 + g47:g̲rid 1)) + (2 s̲elect s̲hape W) ×
((W, "W", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))) ×
((g41:r̲ank (g, "g", 1, "g[i]")) g43:a̲t (
((g, "g", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))))
} @) + (s × ({ @ →
g48:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
g49:g̲rid ← { k → g48:grid r̲eshape (o̲ffsets k s̲elect g48:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g48:grid) r̲eshape 0 }
'+ r̲/₂ g48:grid r̲eshape ((
((g41:r̲ank (A, "A", 2, "A[k,j]")) g43:a̲t ((
((A, "A", 2, "j") g42:c̲heck (0 + g49:g̲rid 1)) + (2 s̲elect s̲hape A) ×
((A, "A", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))) ×
((g41:r̲ank (b, "b", 1, "b[k]")) g43:a̲t (
((b, "b", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))))
} @))))
} @))backDrawn : Char
The same, drawn as math.
backDrawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"G"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("b[k] | r" ᵉt̲ex< "sum i:=0..7 | B[i,k] * g[i]") c̲at r#" \qquad \triangleright \, b \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dA[k,j] | r 8" ᵉt̲ex< "s * b[k] * X[j]") c̲at r#" \qquad \triangleright \, dA \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dB[i,k] | 8 r" ᵉt̲ex< "s * g[i] * h[k]") c̲at r#" \qquad \triangleright \, dB \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("b_{k} = \\sum_{i=0}^{7} B_{i,k} \\, g_{i}")ᵉt̲ex< expands to
("dA_{k,j} = s \\, b_{k} \\, X_{j}")ᵉt̲ex< expands to
("dB_{i,k} = s \\, g_{i} \\, h_{k}")ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + s \\, \\left(\\sum_{k=0}^{q} A_{k,j} \\, b_{k}\\right)")_ : Int
The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.
"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
g50:b ← (o̲\ B) '+ '× i̲nner g51:g
(s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}ᵘf̲orward : Float -> Float
The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.
"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
g50:b ← (o̲\ B) '+ '× i̲nner g51:g
(s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}ᵘb̲ackward : (Float, Float) -> (Float, Float, Float)
The same layer written by the library's macro from a spec (lo:a_dapter<: its forward and backward functions, xetal expand shows them); the pipeline's dx is the macro's, within 1e-12: 1.
"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
g50:b ← (o̲\ B) '+ '× i̲nner g51:g
(s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}dx : Float
(_, _, dx) ← ᵘb̲ackward (8 1 r̲eshape X, 8 1 r̲eshape G)
Training: LoRA beside LoRA-FA
a0 : (Float, Float, Float, Float)
Both start as LoRA does, B at 0, from the same A (seeded); LoRA trains A and B, LoRA-FA B alone. The loss after every 10 steps.
a0 ← (W, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s)
q1 : ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)
(q1, L1) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStart a0, ᵘl̲oss a0)
L1 : Float
(q1, L1) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStart a0, ᵘl̲oss a0)
q2 : ((Float, Float, Float, Float), (Float, Float), Float)
(q2, L2) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStartFA a0, ᵘl̲oss a0)
L2 : Float
(q2, L2) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStartFA a0, ᵘl̲oss a0)
u0 : Float
The best any update of rank k can do (Eckart-Young): half the sum of T's squared singular values past the k-th. They are the eigenvalues of T' T, found by power iteration, each taken out once found.
u0 ← 1.0 + 0.1 × f̲loat r̲ange 8
ᵘb̲est : Float -> Float
ᵘb̲est ← { D → (_, ev) ← 8 '{ (M, e) → v ← 300 '{ w → m ← M '+ '× i̲nner w m ÷ 0.000000000000000001 + s̲qrt '+ r̲/ m × m } p̲ower u0 lam ← '+ r̲/ v × M '+ '× i̲nner v (M − lam × v '× t̲able v, e c̲at lam) } p̲ower ((o̲\ D) '+ '× i̲nner D, 0 r̲eshape 0.0) 0.0 m̲ax 0.5 × '{ k → '+ r̲/ k d̲rop ev } e̲ach r̲ange 8 }
W in 4 bits
Wu : Float
QLoRA keeps the frozen W in 4 bits and computes with it dequantized. W in blocks of qk, each with its scale (its largest size): uniform 4 bits (rounded to -7 .. 7 steps of the scale / 7) and NF4 (the nearest of 16 levels at the normal's quantiles). Their RMSE against W, then against a normal 8 x 8 matrix Z (Box-Muller on the ripple): W's numbers are a sine, piled up near its ends, so uniform does better; on normal numbers NF4 does.
Wu ← (4, qk) qz:u̲niform W
Z : Float
Z ← 8 8 r̲eshape (s̲qrt -2.0 × l̲og 0.0001 m̲ax 0.5 + 0.5 × s̲in 78.233 × k2 × k2) × c̲os 6.283185307179586 × 0.5 + 0.5 × s̲in 37.719 × k2 × k2
ᵘr̲mse : Float -> Float -> Float
ᵘr̲mse ← { a b → s̲qrt ('+ r̲/ r̲avel (a − b) × a − b) ÷ f̲loat t̲ally r̲avel a }
QLoRA: the adapter on the 4-bit base
Q : (Int, Float, Int)
The base stored in NF4 (blocks of qk) and trained on as LoRA is, the same start and steps, the base dequantized (W~, Wn above): its loss every 10 steps. W~ is W plus the 4-bit error, so what the adapter has to learn is T less it; the best rank r can do for that.
Q ← qk ˡᵒq̲base W
q3 : ((Float, Float, Float, Float), (Float, Float, Float, Float), Float)
(q3, L3) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStart (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s), ᵘl̲oss (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s))
L3 : Float
(q3, L3) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at ᵘl̲oss a) } p̲ower (ˡᵒa̲damStart (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s), ᵘl̲oss (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s))
A3 : Float
The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.
((_, A3, B3, s3), _, _) ← q3
B3 : Float
The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.
((_, A3, B3, s3), _, _) ← q3
s3 : Float
The merge into a 4-bit base: W~ + s B A quantized again. Its largest miss of W~ + s B A, and of the adapter's output.
((_, A3, B3, s3), _, _) ← q3
bits : Float
Bits a weight: 4-bit codes with a 32-bit scale per block, then the scales double-quantized (8 bits, blocks of 256); a 4096 x 14336 projection's MiB in FP32, FP16, NF4 and NF4 double-quantized.
bits ← (qk, 256) qz:b̲its 4
VeRA: shared random projections, two vectors
A0 : Float
VeRA (Kopiczko, Blankevoort and Asano, 2024) freezes A and B too: A0 (r x 8) and B0 (8 x r) are random, regenerated from a seed and shared by every layer of the shape, and it trains two vectors, d (r numbers) scaling A0 x and b (8) scaling B0's output: y = W x + b (B0 (d (A0 x))). Forward and back as pipelines, at made-up vectors dv and bv (a real adapter starts with b at 0).
(_, A0, B0, _, _) ← 1 ˡᵒv̲era (W, r)
B0 : Float
VeRA (Kopiczko, Blankevoort and Asano, 2024) freezes A and B too: A0 (r x 8) and B0 (8 x r) are random, regenerated from a seed and shared by every layer of the shape, and it trains two vectors, d (r numbers) scaling A0 x and b (8) scaling B0's output: y = W x + b (B0 (d (A0 x))). Forward and back as pipelines, at made-up vectors dv and bv (a real adapter starts with b at 0).
(_, A0, B0, _, _) ← 1 ˡᵒv̲era (W, r)
yv : Float
yv ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
x ← X
h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A0[k,j] * x[j]"
u ← "u[k] | r" ᵉe̲q< "dv[k] * h[k]"
v ← "v[i] | 8" ᵉe̲q< "sum k:=0..q | B0[i,k] * u[k]"
g53:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]"
g53:stage
} @)ᵉe̲q< expands to
(({ @ →
g56:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g55:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g54:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g57:grid ← r
g58:g̲rid ← { k → g57:grid r̲eshape (o̲ffsets k s̲elect g57:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g57:grid) r̲eshape 0 }
g57:grid r̲eshape (({ @ →
g59:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g60:g̲rid ← { k → g59:grid r̲eshape (o̲ffsets k s̲elect g59:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g59:grid) r̲eshape 0 }
'+ r̲/₂ g59:grid r̲eshape ((
((g54:r̲ank (A0, "A0", 2, "A0[k,j]")) g56:a̲t ((
((A0, "A0", 2, "j") g55:c̲heck (0 + g60:g̲rid 2)) + (2 s̲elect s̲hape A0) ×
((A0, "A0", 1, "k") g55:c̲heck (0 + g60:g̲rid 1))))) ×
((g54:r̲ank (x, "x", 1, "x[j]")) g56:a̲t (
((x, "x", 1, "j") g55:c̲heck (0 + g60:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g63:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g62:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g61:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g64:grid ← r
g65:g̲rid ← { k → g64:grid r̲eshape (o̲ffsets k s̲elect g64:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g64:grid) r̲eshape 0 }
g64:grid r̲eshape ((
((g61:r̲ank (dv, "dv", 1, "dv[k]")) g63:a̲t (
((dv, "dv", 1, "k") g62:c̲heck (0 + g65:g̲rid 1)))) ×
((g61:r̲ank (h, "h", 1, "h[k]")) g63:a̲t (
((h, "h", 1, "k") g62:c̲heck (0 + g65:g̲rid 1))))))
} @))ᵉe̲q< expands to
(({ @ →
g68:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g67:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g66:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g69:grid ← 8
g70:g̲rid ← { k → g69:grid r̲eshape (o̲ffsets k s̲elect g69:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g69:grid) r̲eshape 0 }
g69:grid r̲eshape (({ @ →
g71:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
g72:g̲rid ← { k → g71:grid r̲eshape (o̲ffsets k s̲elect g71:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g71:grid) r̲eshape 0 }
'+ r̲/₂ g71:grid r̲eshape ((
((g66:r̲ank (B0, "B0", 2, "B0[i,k]")) g68:a̲t ((
((B0, "B0", 2, "k") g67:c̲heck (0 + g72:g̲rid 2)) + (2 s̲elect s̲hape B0) ×
((B0, "B0", 1, "i") g67:c̲heck (0 + g72:g̲rid 1))))) ×
((g66:r̲ank (u, "u", 1, "u[k]")) g68:a̲t (
((u, "u", 1, "k") g67:c̲heck (0 + g72:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g75:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g74:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g73:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g76:grid ← 8
g77:g̲rid ← { k → g76:grid r̲eshape (o̲ffsets k s̲elect g76:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g76:grid) r̲eshape 0 }
g76:grid r̲eshape ((({ @ →
g78:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g79:g̲rid ← { k → g78:grid r̲eshape (o̲ffsets k s̲elect g78:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g78:grid) r̲eshape 0 }
'+ r̲/₂ g78:grid r̲eshape ((
((g73:r̲ank (W, "W", 2, "W[i,j]")) g75:a̲t ((
((W, "W", 2, "j") g74:c̲heck (0 + g79:g̲rid 2)) + (2 s̲elect s̲hape W) ×
((W, "W", 1, "i") g74:c̲heck (0 + g79:g̲rid 1))))) ×
((g73:r̲ank (x, "x", 1, "x[j]")) g75:a̲t (
((x, "x", 1, "j") g74:c̲heck (0 + g79:g̲rid 2))))))
} @) + (
((g73:r̲ank (bv, "bv", 1, "bv[i]")) g75:a̲t (
((bv, "bv", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))) ×
((g73:r̲ank (v, "v", 1, "v[i]")) g75:a̲t (
((v, "v", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))))))
} @))drawnV : Char
The same, drawn as math.
drawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A0[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[k] | r" ᵉt̲ex< "dv[k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("v[i] | 8" ᵉt̲ex< "sum k:=0..q | B0[i,k] * u[k]") c̲at r#" \qquad \triangleright \, v \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A0_{k,j} \\, x_{j}")ᵉt̲ex< expands to
("u_{k} = dv_{k} \\, h_{k}")ᵉt̲ex< expands to
("v_{i} = \\sum_{k=0}^{q} B0_{i,k} \\, u_{k}")ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + bv_{i} \\, v_{i}")hv : Float
Back, for the gradient g at the output: c = B0' (b g) once (r numbers), then dd = c (A0 x) and db = g (B0 (d (A0 x))), item by item, and dx through W and through the adapter. The projections are in every gradient and get none.
hv ← A0 '+ '× i̲nner X
backV : Float
backV ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
g ← Gv
c ← "c[k] | r" ᵉe̲q< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]"
dd ← "dd[k] | r" ᵉe̲q< "c[k] * hv[k]"
db ← "db[i] | 8" ᵉe̲q< "g[i] * vv[i]"
g80:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])"
g80:stage
} @)ᵉe̲q< expands to
(({ @ →
g83:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g82:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g81:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g84:grid ← r
g85:g̲rid ← { k → g84:grid r̲eshape (o̲ffsets k s̲elect g84:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g84:grid) r̲eshape 0 }
g84:grid r̲eshape (({ @ →
g86:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g87:g̲rid ← { k → g86:grid r̲eshape (o̲ffsets k s̲elect g86:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g86:grid) r̲eshape 0 }
'+ r̲/₂ g86:grid r̲eshape (((
((g81:r̲ank (B0, "B0", 2, "B0[i,k]")) g83:a̲t ((
((B0, "B0", 2, "k") g82:c̲heck (0 + g87:g̲rid 1)) + (2 s̲elect s̲hape B0) ×
((B0, "B0", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) ×
((g81:r̲ank (bv, "bv", 1, "bv[i]")) g83:a̲t (
((bv, "bv", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) ×
((g81:r̲ank (g, "g", 1, "g[i]")) g83:a̲t (
((g, "g", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))))
} @))
} @))ᵉe̲q< expands to
(({ @ →
g90:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g89:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g88:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g91:grid ← r
g92:g̲rid ← { k → g91:grid r̲eshape (o̲ffsets k s̲elect g91:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g91:grid) r̲eshape 0 }
g91:grid r̲eshape ((
((g88:r̲ank (c, "c", 1, "c[k]")) g90:a̲t (
((c, "c", 1, "k") g89:c̲heck (0 + g92:g̲rid 1)))) ×
((g88:r̲ank (hv, "hv", 1, "hv[k]")) g90:a̲t (
((hv, "hv", 1, "k") g89:c̲heck (0 + g92:g̲rid 1))))))
} @))ᵉe̲q< expands to
(({ @ →
g95:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g94:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g93:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g96:grid ← 8
g97:g̲rid ← { k → g96:grid r̲eshape (o̲ffsets k s̲elect g96:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g96:grid) r̲eshape 0 }
g96:grid r̲eshape ((
((g93:r̲ank (g, "g", 1, "g[i]")) g95:a̲t (
((g, "g", 1, "i") g94:c̲heck (0 + g97:g̲rid 1)))) ×
((g93:r̲ank (vv, "vv", 1, "vv[i]")) g95:a̲t (
((vv, "vv", 1, "i") g94:c̲heck (0 + g97:g̲rid 1))))))
} @))ᵉe̲q< expands to
(({ @ →
g100:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g99:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g98:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g101:grid ← 8
g102:g̲rid ← { k → g101:grid r̲eshape (o̲ffsets k s̲elect g101:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g101:grid) r̲eshape 0 }
g101:grid r̲eshape ((({ @ →
g103:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
g104:g̲rid ← { k → g103:grid r̲eshape (o̲ffsets k s̲elect g103:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g103:grid) r̲eshape 0 }
'+ r̲/₂ g103:grid r̲eshape ((
((g98:r̲ank (W, "W", 2, "W[i,j]")) g100:a̲t ((
((W, "W", 2, "j") g99:c̲heck (0 + g104:g̲rid 1)) + (2 s̲elect s̲hape W) ×
((W, "W", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))) ×
((g98:r̲ank (g, "g", 1, "g[i]")) g100:a̲t (
((g, "g", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))))
} @) + ({ @ →
g105:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
g106:g̲rid ← { k → g105:grid r̲eshape (o̲ffsets k s̲elect g105:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g105:grid) r̲eshape 0 }
'+ r̲/₂ g105:grid r̲eshape (((
((g98:r̲ank (A0, "A0", 2, "A0[k,j]")) g100:a̲t ((
((A0, "A0", 2, "j") g99:c̲heck (0 + g106:g̲rid 1)) + (2 s̲elect s̲hape A0) ×
((A0, "A0", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) ×
((g98:r̲ank (dv, "dv", 1, "dv[k]")) g100:a̲t (
((dv, "dv", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) ×
((g98:r̲ank (c, "c", 1, "c[k]")) g100:a̲t (
((c, "c", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))))
} @)))
} @))backDrawnV : Char
The same, drawn as math.
backDrawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"Gv"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("c[k] | r" ᵉt̲ex< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]") c̲at r#" \qquad \triangleright \, c \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dd[k] | r" ᵉt̲ex< "c[k] * hv[k]") c̲at r#" \qquad \triangleright \, dd \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("db[i] | 8" ᵉt̲ex< "g[i] * vv[i]") c̲at r#" \qquad \triangleright \, db \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("c_{k} = \\sum_{i=0}^{7} B0_{i,k} \\, bv_{i} \\, g_{i}")ᵉt̲ex< expands to
("dd_{k} = c_{k} \\, hv_{k}")ᵉt̲ex< expands to
("db_{i} = g_{i} \\, vv_{i}")ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + \\left(\\sum_{k=0}^{q} A0_{k,j} \\, dv_{k} \\, c_{k}\\right)")_ : Int
The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.
"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
(bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}ᵘf̲orwardV : Float -> Float
The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.
"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
(bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}ᵘb̲ackwardV : (Float, Float) -> (Float, Float, Float)
The macro's VeRA layer from its spec: the pipelines' y and dx are its forward and backward functions', within 1e-12: 1 1.
"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
(bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}dxv : Float
(_, _, dxv) ← ᵘb̲ackwardV (8 1 r̲eshape X, 8 1 r̲eshape Gv)
V : (Float, Float, Float, Float, Float)
VeRA trained beside LoRA, from its start (b at 0, d at 0.1), on the same known update and steps: its loss every 10 steps.
V ← 1 ˡᵒv̲era (W, r)
q4 : ((Float, Float, Float, Float, Float), (Float, Float, Float, Float), Float)
(q4, L4) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at a ˡᵒl̲ossV (I, Y)) } p̲ower (ˡᵒa̲damStartV V, V ˡᵒl̲ossV (I, Y))
L4 : Float
(q4, L4) ← chunks '{ (z, L) → z2 ← every '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower z (a, _, _) ← z2 (z2, L c̲at a ˡᵒl̲ossV (I, Y)) } p̲ower (ˡᵒa̲damStartV V, V ˡᵒl̲ossV (I, Y))
The rank sweep
ᵘs̲tart : Int -> (Float, Float, Float, Float)
LoRA and LoRA-FA trained at each rank 1 .. 8 (300 steps each) from the same start: LoRA reaches the best above at every rank; LoRA-FA, its update held in A's row space, does not; nor does VeRA, which trains 8 + k numbers.
ᵘs̲tart ← { k → (W, 1 ˡᵒs̲eeded k c̲at 8, (8 c̲at k) r̲eshape 0.0, alpha ˡᵒs̲caling k) }
ᵘl̲ora : Int -> Float
ᵘl̲ora ← { k → (a, _, _) ← 300 '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower ˡᵒa̲damStart ᵘs̲tart k ᵘl̲oss a }
ᵘv̲era : Int -> Float
ᵘv̲era ← { k → (a, _, _) ← 300 '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower ˡᵒa̲damStartV 1 ˡᵒv̲era (W, k) a ˡᵒl̲ossV (I, Y) }