sourcedemos/lora-explorer/lora-explorer.xtl

1⍝!/usr/bin/env xetal 2⍝│ LoRA explorer: one 8 x 8 projection adapted by a rank-r update. The 3⍝│ frozen W, the trainable A (r x 8) and B (8 x r); the adapted layer 4⍝│ y = W x + s B (A x) as a pipeline of equations, drawn as math, its 5⍝│ factors kept apart; the update s B A, its rank and the merge; what 6⍝│ adapters cost on a large mixture-of-experts model; a backward step, 7⍝│ stage by stage; LoRA and LoRA-FA trained side by side to fit a known 8⍝│ update, against the best any rank-r update can do (the truncated 9⍝│ SVD); W in 4 bits, uniform and NF4 (QLoRA's base); then the rank 10⍝│ sweep. 11 12"e:" u̲se< "Equations" 13"flow:" u̲se< "Flow" 14"lo:" u̲se< "LoRA" 15"qz:" u̲se< "Quant" 16 17⍝━━ Data 18 19⍝│ The rank, alpha (the scale s is alpha / r), and the sizes for the 20⍝│ costs: a SwiGLU expert (d 4096, f 14336), 64 experts a layer, 24 layers. 21r ← 3 22alpha ← 6.0 23d ← 4096 24f ← 14336 25experts ← 64 26layers ← 24 27⍝│ W, A, B and the input x by a fixed ripple, the sine of a square (no 28⍝│ chance). A real adapter starts with B at 0 and learns it; these are 29⍝│ made up, to show the arrays. 30i ← f̲loat r̲ange 64 31W ← 8 8 r̲eshape s̲in 0.7 × i × i 32j ← f̲loat r̲ange r × 8 33A ← (r c̲at 8) r̲eshape 0.4 × c̲os 1.1 × j × j 34B ← (8 c̲at r) r̲eshape 0.3 × s̲in 0.9 × j × j 35X ← c̲os 0.4 × 8 t̲ake i 36s ← alpha ˡᵒs̲caling r 37q ← r − 1 38⍝│ The known update to learn, T = P Q (8 x 8, of rank 4), by ripple; 39⍝│ the training inputs are the 8 unit vectors (I), so the loss is half 40⍝│ the squared distance between s B A and T, entry by entry. Adam at 41⍝│ the rate lr, 30 times 10 steps. 42p ← f̲loat r̲ange 32 43T ← (8 4 r̲eshape 0.5 × s̲in 1.3 × p × p) '+ '× i̲nner 4 8 r̲eshape 0.5 × c̲os 0.6 × p × p 44I ← 8 8 r̲eshape 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 45lr ← 0.05 46chunks ← 30 47every ← 10 48⍝│ The quantization's block: 16 numbers (QLoRA's is 64; W has 64). 49qk ← 16 50 51⍝━━ The adapted layer 52 53⍝│ x goes two ways: through the frozen W, and through the adapter, A 54⍝│ first (r numbers), then B (8); the two are added. 55y ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  x ← X
  h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A[k,j] * x[j]"
  u ← "u[i] | 8" ᵉe̲q< "sum k:=0..q | B[i,k] * h[k]"
  g1:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]"
  g1:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g4:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g3:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g2:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g5:grid ← r
  g6:g̲rid ← { k → g5:grid r̲eshape (o̲ffsets k s̲elect g5:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g5:grid) r̲eshape 0 }
  g5:grid r̲eshape (({ @ →
  g7:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g8:g̲rid ← { k → g7:grid r̲eshape (o̲ffsets k s̲elect g7:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g7:grid) r̲eshape 0 }
  '+ r̲/₂ g7:grid r̲eshape ((
    ((g2:r̲ank (A, "A", 2, "A[k,j]")) g4:a̲t ((
      ((A, "A", 2, "j") g3:c̲heck (0 + g8:g̲rid 2)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "k") g3:c̲heck (0 + g8:g̲rid 1))))) × 
    ((g2:r̲ank (x, "x", 1, "x[j]")) g4:a̲t (
      ((x, "x", 1, "j") g3:c̲heck (0 + g8:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g11:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g10:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g9:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g12:grid ← 8
  g13:g̲rid ← { k → g12:grid r̲eshape (o̲ffsets k s̲elect g12:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g12:grid) r̲eshape 0 }
  g12:grid r̲eshape (({ @ →
  g14:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g15:g̲rid ← { k → g14:grid r̲eshape (o̲ffsets k s̲elect g14:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g14:grid) r̲eshape 0 }
  '+ r̲/₂ g14:grid r̲eshape ((
    ((g9:r̲ank (B, "B", 2, "B[i,k]")) g11:a̲t ((
      ((B, "B", 2, "k") g10:c̲heck (0 + g15:g̲rid 2)) + (2 s̲elect s̲hape B) × 
      ((B, "B", 1, "i") g10:c̲heck (0 + g15:g̲rid 1))))) × 
    ((g9:r̲ank (h, "h", 1, "h[k]")) g11:a̲t (
      ((h, "h", 1, "k") g10:c̲heck (0 + g15:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g18:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g17:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g16:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g19:grid ← 8
  g20:g̲rid ← { k → g19:grid r̲eshape (o̲ffsets k s̲elect g19:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g19:grid) r̲eshape 0 }
  g19:grid r̲eshape ((({ @ →
  g21:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g22:g̲rid ← { k → g21:grid r̲eshape (o̲ffsets k s̲elect g21:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g21:grid) r̲eshape 0 }
  '+ r̲/₂ g21:grid r̲eshape ((
    ((g16:r̲ank (W, "W", 2, "W[i,j]")) g18:a̲t ((
      ((W, "W", 2, "j") g17:c̲heck (0 + g22:g̲rid 2)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g17:c̲heck (0 + g22:g̲rid 1))))) × 
    ((g16:r̲ank (x, "x", 1, "x[j]")) g18:a̲t (
      ((x, "x", 1, "j") g17:c̲heck (0 + g22:g̲rid 2))))))
} @) + (s × 
    ((g16:r̲ank (u, "u", 1, "u[i]")) g18:a̲t (
      ((u, "u", 1, "i") g17:c̲heck (0 + g20:g̲rid 1)))))))
} @))
56 X ▹x 57▷ "h[k] | r" ᵉe̲q< "sum j:=0..7 | A[k,j] * x[j]" ▹h 58▷ "u[i] | 8" ᵉe̲q< "sum k:=0..q | B[i,k] * h[k]" ▹u 59▷ "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]""# 60⍝│ The same pipeline, drawn as math, a row a stage. 61drawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[i] | 8" ᵉt̲ex< "sum k:=0..q | B[i,k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A_{k,j} \\, x_{j}")
ᵉt̲ex< expands to
("u_{i} = \\sum_{k=0}^{q} B_{i,k} \\, h_{k}")
ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + s \\, u_{i}")
62 X ▹x 63▷ "h[k] | r" ᵉe̲q< "sum j:=0..7 | A[k,j] * x[j]" ▹h 64▷ "u[i] | 8" ᵉe̲q< "sum k:=0..q | B[i,k] * h[k]" ▹u 65▷ "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + s * u[i]""# 66y 67 68⍝━━ The update and the merge 69 70⍝│ The update s B A (8 x 8, of rank r at most), the merged matrix W + s 71⍝│ B A, the update's rank, and how far the merged layer's output is from 72⍝│ the factored one (under 1e-12: 1). 73delta ← ˡᵒd̲elta (A, B, s) 74merged ← ˡᵒm̲erge (W, A, B, s) 75ˡᵒr̲ank delta 760.000000000001 > 'm̲ax r̲/ a̲bs y − merged '+ '× i̲nner X 77⍝│ The numbers trained, against W's 64; the multiply-adds for one input, 78⍝│ factored, and to make B A first. 79(r ˡᵒp̲arams 8 8) c̲at 64 80r ˡᵒc̲osts 8 8 81 82⍝━━ On a large model 83 84⍝│ One expert's adapters (gate, up, down), then their MiB in BF16: one 85⍝│ expert, a layer of experts, every layer. 86n ← r ˡᵒe̲xpert d c̲at f 87mib ← (16 ˡᵒb̲ytes n) ÷ 1048576.0 88n 89mib c̲at (mib × f̲loat experts) c̲at mib × f̲loat experts × layers 90 91⍝━━ A backward step 92 93⍝│ The loss's gradient at the output for x is g = y - (W + T) x (the 94⍝│ known update's output wanted). Back through the layer: B' g once 95⍝│ (r numbers), then the gradients of A and B (outer products, s b x' 96⍝│ and s g h') and of x, through W and through the adapter. 97h ← A '+ '× i̲nner X 98G ← y − (W + T) '+ '× i̲nner X 99back ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g ← G
  b ← "b[k] | r" ᵉe̲q< "sum i:=0..7 | B[i,k] * g[i]"
  dA ← "dA[k,j] | r 8" ᵉe̲q< "s * b[k] * X[j]"
  dB ← "dB[i,k] | 8 r" ᵉe̲q< "s * g[i] * h[k]"
  g23:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])"
  g23:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g26:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g25:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g24:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g27:grid ← r
  g28:g̲rid ← { k → g27:grid r̲eshape (o̲ffsets k s̲elect g27:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g27:grid) r̲eshape 0 }
  g27:grid r̲eshape (({ @ →
  g29:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g30:g̲rid ← { k → g29:grid r̲eshape (o̲ffsets k s̲elect g29:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g29:grid) r̲eshape 0 }
  '+ r̲/₂ g29:grid r̲eshape ((
    ((g24:r̲ank (B, "B", 2, "B[i,k]")) g26:a̲t ((
      ((B, "B", 2, "k") g25:c̲heck (0 + g30:g̲rid 1)) + (2 s̲elect s̲hape B) × 
      ((B, "B", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))) × 
    ((g24:r̲ank (g, "g", 1, "g[i]")) g26:a̲t (
      ((g, "g", 1, "i") g25:c̲heck (0 + g30:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g33:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g32:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g31:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g34:grid ← r c̲at 8
  g35:g̲rid ← { k → g34:grid r̲eshape (o̲ffsets k s̲elect g34:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g34:grid) r̲eshape 0 }
  g34:grid r̲eshape (((s × 
    ((g31:r̲ank (b, "b", 1, "b[k]")) g33:a̲t (
      ((b, "b", 1, "k") g32:c̲heck (0 + g35:g̲rid 1))))) × 
    ((g31:r̲ank (X, "X", 1, "X[j]")) g33:a̲t (
      ((X, "X", 1, "j") g32:c̲heck (0 + g35:g̲rid 2))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g38:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g37:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g36:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g39:grid ← 8 c̲at r
  g40:g̲rid ← { k → g39:grid r̲eshape (o̲ffsets k s̲elect g39:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g39:grid) r̲eshape 0 }
  g39:grid r̲eshape (((s × 
    ((g36:r̲ank (g, "g", 1, "g[i]")) g38:a̲t (
      ((g, "g", 1, "i") g37:c̲heck (0 + g40:g̲rid 1))))) × 
    ((g36:r̲ank (h, "h", 1, "h[k]")) g38:a̲t (
      ((h, "h", 1, "k") g37:c̲heck (0 + g40:g̲rid 2))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g43:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g42:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g41:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g44:grid ← 8
  g45:g̲rid ← { k → g44:grid r̲eshape (o̲ffsets k s̲elect g44:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g44:grid) r̲eshape 0 }
  g44:grid r̲eshape ((({ @ →
  g46:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g47:g̲rid ← { k → g46:grid r̲eshape (o̲ffsets k s̲elect g46:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g46:grid) r̲eshape 0 }
  '+ r̲/₂ g46:grid r̲eshape ((
    ((g41:r̲ank (W, "W", 2, "W[i,j]")) g43:a̲t ((
      ((W, "W", 2, "j") g42:c̲heck (0 + g47:g̲rid 1)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))) × 
    ((g41:r̲ank (g, "g", 1, "g[i]")) g43:a̲t (
      ((g, "g", 1, "i") g42:c̲heck (0 + g47:g̲rid 2))))))
} @) + (s × ({ @ →
  g48:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g49:g̲rid ← { k → g48:grid r̲eshape (o̲ffsets k s̲elect g48:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g48:grid) r̲eshape 0 }
  '+ r̲/₂ g48:grid r̲eshape ((
    ((g41:r̲ank (A, "A", 2, "A[k,j]")) g43:a̲t ((
      ((A, "A", 2, "j") g42:c̲heck (0 + g49:g̲rid 1)) + (2 s̲elect s̲hape A) × 
      ((A, "A", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))) × 
    ((g41:r̲ank (b, "b", 1, "b[k]")) g43:a̲t (
      ((b, "b", 1, "k") g42:c̲heck (0 + g49:g̲rid 2))))))
} @))))
} @))
100 G ▹g 101▷ "b[k] | r" ᵉe̲q< "sum i:=0..7 | B[i,k] * g[i]" ▹b 102▷ "dA[k,j] | r 8" ᵉe̲q< "s * b[k] * X[j]" ▹dA 103▷ "dB[i,k] | 8 r" ᵉe̲q< "s * g[i] * h[k]" ▹dB 104▷ "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])""# 105⍝│ The same, drawn as math. 106backDrawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"G"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("b[k] | r" ᵉt̲ex< "sum i:=0..7 | B[i,k] * g[i]") c̲at r#" \qquad \triangleright \, b \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dA[k,j] | r 8" ᵉt̲ex< "s * b[k] * X[j]") c̲at r#" \qquad \triangleright \, dA \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dB[i,k] | 8 r" ᵉt̲ex< "s * g[i] * h[k]") c̲at r#" \qquad \triangleright \, dB \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("b_{k} = \\sum_{i=0}^{7} B_{i,k} \\, g_{i}")
ᵉt̲ex< expands to
("dA_{k,j} = s \\, b_{k} \\, X_{j}")
ᵉt̲ex< expands to
("dB_{i,k} = s \\, g_{i} \\, h_{k}")
ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + s \\, \\left(\\sum_{k=0}^{q} A_{k,j} \\, b_{k}\\right)")
107 G ▹g 108▷ "b[k] | r" ᵉe̲q< "sum i:=0..7 | B[i,k] * g[i]" ▹b 109▷ "dA[k,j] | r 8" ᵉe̲q< "s * b[k] * X[j]" ▹dA 110▷ "dB[i,k] | 8 r" ᵉe̲q< "s * g[i] * h[k]" ▹dB 111▷ "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + s * (sum k:=0..q | A[k,j] * b[k])""# 112⍝│ The same layer written by the library's macro from a spec (lo:a_dapter<: 113⍝│ its forward and backward functions, xetal expand shows them); the 114⍝│ pipeline's dx is the macro's, within 1e-12: 1. 115"u:f_orward u:b_ackward" ˡᵒa̲dapter< "W A B s | r r | train A B"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A B s | r r | train A B\"" } (r = f̲irst s̲hape A) ∧ r = 2 s̲elect s̲hape B
ᵘf̲orward ← { g52:x → (W '+ '× i̲nner g52:x) + s × B '+ '× i̲nner A '+ '× i̲nner g52:x }
ᵘb̲ackward ← { (g52:x, g51:g) →
  g50:b ← (o̲\ B) '+ '× i̲nner g51:g
  (s × g50:b '+ '× i̲nner o̲\ g52:x, s × g51:g '+ '× i̲nner o̲\ A '+ '× i̲nner g52:x, ((o̲\ W) '+ '× i̲nner g51:g) + s × (o̲\ A) '+ '× i̲nner g50:b)
}
116(_, _, dx) ← ᵘb̲ackward (8 1 r̲eshape X, 8 1 r̲eshape G) 1170.000000000001 > 'm̲ax r̲/ a̲bs back − r̲avel dx 118 119⍝━━ Training: LoRA beside LoRA-FA 120 121⍝│ Both start as LoRA does, B at 0, from the same A (seeded); LoRA 122⍝│ trains A and B, LoRA-FA B alone. The loss after every 10 steps. 123a0 ← (W, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s) 124Y ← (W + T) '+ '× i̲nner I 125ᵘl̲oss ← { a → a ˡᵒl̲oss (I, Y) } 126(q1, L1) ← chunks '{ (z, L) → 127 z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z 128 (a, _, _) ← z2 129 (z2, L c̲at ᵘl̲oss a) 130} p̲ower (ˡᵒa̲damStart a0, ᵘl̲oss a0) 131(q2, L2) ← chunks '{ (z, L) → 132 z2 ← every '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower z 133 (a, _, _) ← z2 134 (z2, L c̲at ᵘl̲oss a) 135} p̲ower (ˡᵒa̲damStartFA a0, ᵘl̲oss a0) 136L1 137L2 138⍝│ The best any update of rank k can do (Eckart-Young): half the sum of 139⍝│ T's squared singular values past the k-th. They are the eigenvalues 140⍝│ of T' T, found by power iteration, each taken out once found. 141u0 ← 1.0 + 0.1 × f̲loat r̲ange 8 142ᵘb̲est ← { D → 143 (_, ev) ← 8 '{ (M, e) → 144 v ← 300 '{ w → m ← M '+ '× i̲nner w 145 m ÷ 0.000000000000000001 + s̲qrt '+ r̲/ m × m } p̲ower u0 146 lam ← '+ r̲/ v × M '+ '× i̲nner v 147 (M − lam × v '× t̲able v, e c̲at lam) 148 } p̲ower ((o̲\ D) '+ '× i̲nner D, 0 r̲eshape 0.0) 149 0.0 m̲ax 0.5 × '{ k → '+ r̲/ k d̲rop ev } e̲ach r̲ange 8 150} 151best ← ᵘb̲est T 152best 153⍝│ What each learned: LoRA's update, LoRA-FA's. 154((_, A1, B1, s1), _, _) ← q1 155((_, A2, B2, s2), _, _) ← q2 156learned ← ˡᵒd̲elta (A1, B1, s1) 157learnedFA ← ˡᵒd̲elta (A2, B2, s2) 158 159⍝━━ W in 4 bits 160 161⍝│ QLoRA keeps the frozen W in 4 bits and computes with it dequantized. 162⍝│ W in blocks of qk, each with its scale (its largest size): uniform 4 163⍝│ bits (rounded to -7 .. 7 steps of the scale / 7) and NF4 (the 164⍝│ nearest of 16 levels at the normal's quantiles). Their RMSE against 165⍝│ W, then against a normal 8 x 8 matrix Z (Box-Muller on the ripple): 166⍝│ W's numbers are a sine, piled up near its ends, so uniform does 167⍝│ better; on normal numbers NF4 does. 168Wu ← (4, qk) qz:u̲niform W 169Wn ← qk qz:n̲f4 W 170k2 ← f̲loat r̲ange 64 171Z ← 8 8 r̲eshape (s̲qrt -2.0 × l̲og 0.0001 m̲ax 0.5 + 0.5 × s̲in 78.233 × k2 × k2) × c̲os 6.283185307179586 × 0.5 + 0.5 × s̲in 37.719 × k2 × k2 172ᵘr̲mse ← { a b → s̲qrt ('+ r̲/ r̲avel (a − b) × a − b) ÷ f̲loat t̲ally r̲avel a } 173(Wu ᵘr̲mse W) c̲at (Wn ᵘr̲mse W) c̲at (((4, qk) qz:u̲niform Z) ᵘr̲mse Z) c̲at (qk qz:n̲f4 Z) ᵘr̲mse Z 174⍝│ The adapted layer on each base (QLoRA's forward pass: the adapter 175⍝│ kept in full precision): its output's largest change from y. 176('m̲ax r̲/ a̲bs y − (Wu, A, B, s) ˡᵒf̲orward X) c̲at 'm̲ax r̲/ a̲bs y − (Wn, A, B, s) ˡᵒf̲orward X 177 178⍝━━ QLoRA: the adapter on the 4-bit base 179 180⍝│ The base stored in NF4 (blocks of qk) and trained on as LoRA is, the 181⍝│ same start and steps, the base dequantized (W~, Wn above): its loss 182⍝│ every 10 steps. W~ is W plus the 4-bit error, so what the adapter 183⍝│ has to learn is T less it; the best rank r can do for that. 184Q ← qk ˡᵒq̲base W 185(q3, L3) ← chunks '{ (z, L) → 186 z2 ← every '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower z 187 (a, _, _) ← z2 188 (z2, L c̲at ᵘl̲oss a) 189} p̲ower (ˡᵒa̲damStart (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s), ᵘl̲oss (ˡᵒd̲equant Q, 1 ˡᵒs̲eeded r c̲at 8, (8 c̲at r) r̲eshape 0.0, s)) 190L3 191r s̲elect ᵘb̲est T − Wn − W 192⍝│ The merge into a 4-bit base: W~ + s B A quantized again. Its largest 193⍝│ miss of W~ + s B A, and of the adapter's output. 194((_, A3, B3, s3), _, _) ← q3 195learnedQ ← ˡᵒd̲elta (A3, B3, s3) 196Mq ← ˡᵒd̲equant qk ˡᵒm̲ergeQ (Q, A3, B3, s3) 197('m̲ax r̲/ r̲avel a̲bs Mq − Wn + learnedQ) c̲at 'm̲ax r̲/ r̲avel a̲bs (Mq '+ '× i̲nner X) − (Q, A3, B3, s3) ˡᵒf̲orwardQ X 198⍝│ Bits a weight: 4-bit codes with a 32-bit scale per block, then the 199⍝│ scales double-quantized (8 bits, blocks of 256); a 4096 x 14336 200⍝│ projection's MiB in FP32, FP16, NF4 and NF4 double-quantized. 201bits ← (qk, 256) qz:b̲its 4 202bits 203(f̲loat 4096 × 14336) × (32.0 16.0 c̲at bits) ÷ 8.0 × 1048576.0 204⍝━━ VeRA: shared random projections, two vectors 205 206⍝│ VeRA (Kopiczko, Blankevoort and Asano, 2024) freezes A and B too: 207⍝│ A0 (r x 8) and B0 (8 x r) are random, regenerated from a seed and 208⍝│ shared by every layer of the shape, and it trains two vectors, d (r 209⍝│ numbers) scaling A0 x and b (8) scaling B0's output: y = W x + b 210⍝│ (B0 (d (A0 x))). Forward and back as pipelines, at made-up vectors 211⍝│ dv and bv (a real adapter starts with b at 0). 212(_, A0, B0, _, _) ← 1 ˡᵒv̲era (W, r) 213dv ← 0.5 + 0.3 × c̲os 0.8 × f̲loat r̲ange r 214bv ← 0.6 × s̲in 1.7 × f̲loat r̲ange 8 215yv ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  x ← X
  h ← "h[k] | r" ᵉe̲q< "sum j:=0..7 | A0[k,j] * x[j]"
  u ← "u[k] | r" ᵉe̲q< "dv[k] * h[k]"
  v ← "v[i] | 8" ᵉe̲q< "sum k:=0..q | B0[i,k] * u[k]"
  g53:stage ← "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]"
  g53:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g56:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g55:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g54:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g57:grid ← r
  g58:g̲rid ← { k → g57:grid r̲eshape (o̲ffsets k s̲elect g57:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g57:grid) r̲eshape 0 }
  g57:grid r̲eshape (({ @ →
  g59:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g60:g̲rid ← { k → g59:grid r̲eshape (o̲ffsets k s̲elect g59:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g59:grid) r̲eshape 0 }
  '+ r̲/₂ g59:grid r̲eshape ((
    ((g54:r̲ank (A0, "A0", 2, "A0[k,j]")) g56:a̲t ((
      ((A0, "A0", 2, "j") g55:c̲heck (0 + g60:g̲rid 2)) + (2 s̲elect s̲hape A0) × 
      ((A0, "A0", 1, "k") g55:c̲heck (0 + g60:g̲rid 1))))) × 
    ((g54:r̲ank (x, "x", 1, "x[j]")) g56:a̲t (
      ((x, "x", 1, "j") g55:c̲heck (0 + g60:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g63:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g62:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g61:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g64:grid ← r
  g65:g̲rid ← { k → g64:grid r̲eshape (o̲ffsets k s̲elect g64:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g64:grid) r̲eshape 0 }
  g64:grid r̲eshape ((
    ((g61:r̲ank (dv, "dv", 1, "dv[k]")) g63:a̲t (
      ((dv, "dv", 1, "k") g62:c̲heck (0 + g65:g̲rid 1)))) × 
    ((g61:r̲ank (h, "h", 1, "h[k]")) g63:a̲t (
      ((h, "h", 1, "k") g62:c̲heck (0 + g65:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g68:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g67:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g66:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g69:grid ← 8
  g70:g̲rid ← { k → g69:grid r̲eshape (o̲ffsets k s̲elect g69:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g69:grid) r̲eshape 0 }
  g69:grid r̲eshape (({ @ →
  g71:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g72:g̲rid ← { k → g71:grid r̲eshape (o̲ffsets k s̲elect g71:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g71:grid) r̲eshape 0 }
  '+ r̲/₂ g71:grid r̲eshape ((
    ((g66:r̲ank (B0, "B0", 2, "B0[i,k]")) g68:a̲t ((
      ((B0, "B0", 2, "k") g67:c̲heck (0 + g72:g̲rid 2)) + (2 s̲elect s̲hape B0) × 
      ((B0, "B0", 1, "i") g67:c̲heck (0 + g72:g̲rid 1))))) × 
    ((g66:r̲ank (u, "u", 1, "u[k]")) g68:a̲t (
      ((u, "u", 1, "k") g67:c̲heck (0 + g72:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g75:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g74:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g73:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g76:grid ← 8
  g77:g̲rid ← { k → g76:grid r̲eshape (o̲ffsets k s̲elect g76:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g76:grid) r̲eshape 0 }
  g76:grid r̲eshape ((({ @ →
  g78:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g79:g̲rid ← { k → g78:grid r̲eshape (o̲ffsets k s̲elect g78:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g78:grid) r̲eshape 0 }
  '+ r̲/₂ g78:grid r̲eshape ((
    ((g73:r̲ank (W, "W", 2, "W[i,j]")) g75:a̲t ((
      ((W, "W", 2, "j") g74:c̲heck (0 + g79:g̲rid 2)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g74:c̲heck (0 + g79:g̲rid 1))))) × 
    ((g73:r̲ank (x, "x", 1, "x[j]")) g75:a̲t (
      ((x, "x", 1, "j") g74:c̲heck (0 + g79:g̲rid 2))))))
} @) + (
    ((g73:r̲ank (bv, "bv", 1, "bv[i]")) g75:a̲t (
      ((bv, "bv", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))) × 
    ((g73:r̲ank (v, "v", 1, "v[i]")) g75:a̲t (
      ((v, "v", 1, "i") g74:c̲heck (0 + g77:g̲rid 1)))))))
} @))
216 X ▹x 217▷ "h[k] | r" ᵉe̲q< "sum j:=0..7 | A0[k,j] * x[j]" ▹h 218▷ "u[k] | r" ᵉe̲q< "dv[k] * h[k]" ▹u 219▷ "v[i] | 8" ᵉe̲q< "sum k:=0..q | B0[i,k] * u[k]" ▹v 220▷ "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]""# 221⍝│ The same, drawn as math. 222drawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"X"# c̲at r#" \qquad \triangleright \, x \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("h[k] | r" ᵉt̲ex< "sum j:=0..7 | A0[k,j] * x[j]") c̲at r#" \qquad \triangleright \, h \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("u[k] | r" ᵉt̲ex< "dv[k] * h[k]") c̲at r#" \qquad \triangleright \, u \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("v[i] | 8" ᵉt̲ex< "sum k:=0..q | B0[i,k] * u[k]") c̲at r#" \qquad \triangleright \, v \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("y[i] | 8" ᵉt̲ex< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("h_{k} = \\sum_{j=0}^{7} A0_{k,j} \\, x_{j}")
ᵉt̲ex< expands to
("u_{k} = dv_{k} \\, h_{k}")
ᵉt̲ex< expands to
("v_{i} = \\sum_{k=0}^{q} B0_{i,k} \\, u_{k}")
ᵉt̲ex< expands to
("y_{i} = \\left(\\sum_{j=0}^{7} W_{i,j} \\, x_{j}\\right) + bv_{i} \\, v_{i}")
223 X ▹x 224▷ "h[k] | r" ᵉe̲q< "sum j:=0..7 | A0[k,j] * x[j]" ▹h 225▷ "u[k] | r" ᵉe̲q< "dv[k] * h[k]" ▹u 226▷ "v[i] | 8" ᵉe̲q< "sum k:=0..q | B0[i,k] * u[k]" ▹v 227▷ "y[i] | 8" ᵉe̲q< "(sum j:=0..7 | W[i,j] * x[j]) + bv[i] * v[i]""# 228yv 229⍝│ Back, for the gradient g at the output: c = B0' (b g) once (r 230⍝│ numbers), then dd = c (A0 x) and db = g (B0 (d (A0 x))), item by 231⍝│ item, and dx through W and through the adapter. The projections are 232⍝│ in every gradient and get none. 233hv ← A0 '+ '× i̲nner X 234vv ← B0 '+ '× i̲nner dv × hv 235Gv ← yv − (W + T) '+ '× i̲nner X 236backV ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g ← Gv
  c ← "c[k] | r" ᵉe̲q< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]"
  dd ← "dd[k] | r" ᵉe̲q< "c[k] * hv[k]"
  db ← "db[i] | 8" ᵉe̲q< "g[i] * vv[i]"
  g80:stage ← "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])"
  g80:stage
} @)
ᵉe̲q< expands to
(({ @ →
  g83:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g82:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g81:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g84:grid ← r
  g85:g̲rid ← { k → g84:grid r̲eshape (o̲ffsets k s̲elect g84:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g84:grid) r̲eshape 0 }
  g84:grid r̲eshape (({ @ →
  g86:grid ← r c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g87:g̲rid ← { k → g86:grid r̲eshape (o̲ffsets k s̲elect g86:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g86:grid) r̲eshape 0 }
  '+ r̲/₂ g86:grid r̲eshape (((
    ((g81:r̲ank (B0, "B0", 2, "B0[i,k]")) g83:a̲t ((
      ((B0, "B0", 2, "k") g82:c̲heck (0 + g87:g̲rid 1)) + (2 s̲elect s̲hape B0) × 
      ((B0, "B0", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) × 
    ((g81:r̲ank (bv, "bv", 1, "bv[i]")) g83:a̲t (
      ((bv, "bv", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))) × 
    ((g81:r̲ank (g, "g", 1, "g[i]")) g83:a̲t (
      ((g, "g", 1, "i") g82:c̲heck (0 + g87:g̲rid 2))))))
} @))
} @))
ᵉe̲q< expands to
(({ @ →
  g90:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g89:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g88:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g91:grid ← r
  g92:g̲rid ← { k → g91:grid r̲eshape (o̲ffsets k s̲elect g91:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g91:grid) r̲eshape 0 }
  g91:grid r̲eshape ((
    ((g88:r̲ank (c, "c", 1, "c[k]")) g90:a̲t (
      ((c, "c", 1, "k") g89:c̲heck (0 + g92:g̲rid 1)))) × 
    ((g88:r̲ank (hv, "hv", 1, "hv[k]")) g90:a̲t (
      ((hv, "hv", 1, "k") g89:c̲heck (0 + g92:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g95:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g94:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g93:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g96:grid ← 8
  g97:g̲rid ← { k → g96:grid r̲eshape (o̲ffsets k s̲elect g96:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g96:grid) r̲eshape 0 }
  g96:grid r̲eshape ((
    ((g93:r̲ank (g, "g", 1, "g[i]")) g95:a̲t (
      ((g, "g", 1, "i") g94:c̲heck (0 + g97:g̲rid 1)))) × 
    ((g93:r̲ank (vv, "vv", 1, "vv[i]")) g95:a̲t (
      ((vv, "vv", 1, "i") g94:c̲heck (0 + g97:g̲rid 1))))))
} @))
ᵉe̲q< expands to
(({ @ →
  g100:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
  g99:c̲heck ← { (A, name, k, from) e →
    n ← k s̲elect s̲hape A
    bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
    0 = t̲ally bad ? e
    "index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
  }
  g98:r̲ank ← { (A, name, q, ref) →
    (t̲ally s̲hape A) = q ? A
    "rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
  }
  g101:grid ← 8
  g102:g̲rid ← { k → g101:grid r̲eshape (o̲ffsets k s̲elect g101:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g101:grid) r̲eshape 0 }
  g101:grid r̲eshape ((({ @ →
  g103:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..7" } (1 + 7 − 0))
  g104:g̲rid ← { k → g103:grid r̲eshape (o̲ffsets k s̲elect g103:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g103:grid) r̲eshape 0 }
  '+ r̲/₂ g103:grid r̲eshape ((
    ((g98:r̲ank (W, "W", 2, "W[i,j]")) g100:a̲t ((
      ((W, "W", 2, "j") g99:c̲heck (0 + g104:g̲rid 1)) + (2 s̲elect s̲hape W) × 
      ((W, "W", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))) × 
    ((g98:r̲ank (g, "g", 1, "g[i]")) g100:a̲t (
      ((g, "g", 1, "i") g99:c̲heck (0 + g104:g̲rid 2))))))
} @) + ({ @ →
  g105:grid ← 8 c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..q" } (1 + q − 0))
  g106:g̲rid ← { k → g105:grid r̲eshape (o̲ffsets k s̲elect g105:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g105:grid) r̲eshape 0 }
  '+ r̲/₂ g105:grid r̲eshape (((
    ((g98:r̲ank (A0, "A0", 2, "A0[k,j]")) g100:a̲t ((
      ((A0, "A0", 2, "j") g99:c̲heck (0 + g106:g̲rid 1)) + (2 s̲elect s̲hape A0) × 
      ((A0, "A0", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) × 
    ((g98:r̲ank (dv, "dv", 1, "dv[k]")) g100:a̲t (
      ((dv, "dv", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))) × 
    ((g98:r̲ank (c, "c", 1, "c[k]")) g100:a̲t (
      ((c, "c", 1, "k") g99:c̲heck (0 + g106:g̲rid 2))))))
} @)))
} @))
237 Gv ▹g 238▷ "c[k] | r" ᵉe̲q< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]" ▹c 239▷ "dd[k] | r" ᵉe̲q< "c[k] * hv[k]" ▹dd 240▷ "db[i] | 8" ᵉe̲q< "g[i] * vv[i]" ▹db 241▷ "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])""# 242⍝│ The same, drawn as math. 243backDrawnV ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"Gv"# c̲at r#" \qquad \triangleright \, g \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("c[k] | r" ᵉt̲ex< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]") c̲at r#" \qquad \triangleright \, c \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dd[k] | r" ᵉt̲ex< "c[k] * hv[k]") c̲at r#" \qquad \triangleright \, dd \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("db[i] | 8" ᵉt̲ex< "g[i] * vv[i]") c̲at r#" \qquad \triangleright \, db \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("dx[j] | 8" ᵉt̲ex< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("c_{k} = \\sum_{i=0}^{7} B0_{i,k} \\, bv_{i} \\, g_{i}")
ᵉt̲ex< expands to
("dd_{k} = c_{k} \\, hv_{k}")
ᵉt̲ex< expands to
("db_{i} = g_{i} \\, vv_{i}")
ᵉt̲ex< expands to
("dx_{j} = \\left(\\sum_{i=0}^{7} W_{i,j} \\, g_{i}\\right) + \\left(\\sum_{k=0}^{q} A0_{k,j} \\, dv_{k} \\, c_{k}\\right)")
244 Gv ▹g 245▷ "c[k] | r" ᵉe̲q< "sum i:=0..7 | B0[i,k] * bv[i] * g[i]" ▹c 246▷ "dd[k] | r" ᵉe̲q< "c[k] * hv[k]" ▹dd 247▷ "db[i] | 8" ᵉe̲q< "g[i] * vv[i]" ▹db 248▷ "dx[j] | 8" ᵉe̲q< "(sum i:=0..7 | W[i,j] * g[i]) + (sum k:=0..q | A0[k,j] * dv[k] * c[k])""# 249⍝│ The macro's VeRA layer from its spec: the pipelines' y and dx are 250⍝│ its forward and backward functions', within 1e-12: 1 1. 251"u:f_orwardV u:b_ackwardV" ˡᵒa̲dapter< "W A0 B0 dv bv | r r | train dv bv"
ˡᵒa̲dapter< expands to
_ ← { v → v ? 0◆ ⎕P̲ANIC "lo:a_dapter<: A must be r x n and B m x r for \"W A0 B0 dv bv | r r | train dv bv\"" } (r = f̲irst s̲hape A0) ∧ r = 2 s̲elect s̲hape B0
ᵘf̲orwardV ← { g109:x → (W '+ '× i̲nner g109:x) + (B0 × bv '× t̲able dv) '+ '× i̲nner A0 '+ '× i̲nner g109:x }
ᵘb̲ackwardV ← { (g109:x, g108:g) →
  g107:b ← B0 × g108:g '+ '× i̲nner o̲\ A0 '+ '× i̲nner g109:x
  (bv '+ '× i̲nner g107:b, g107:b '+ '× i̲nner dv, ((o̲\ W) '+ '× i̲nner g108:g) + (o̲\ A0) '+ '× i̲nner (o̲\ B0 × bv '× t̲able dv) '+ '× i̲nner g108:g)
}
252(_, _, dxv) ← ᵘb̲ackwardV (8 1 r̲eshape X, 8 1 r̲eshape Gv) 2530.000000000001 > ('m̲ax r̲/ a̲bs yv − ᵘf̲orwardV X) c̲at 'm̲ax r̲/ a̲bs backV − r̲avel dxv 254⍝│ VeRA trained beside LoRA, from its start (b at 0, d at 0.1), on the 255⍝│ same known update and steps: its loss every 10 steps. 256V ← 1 ˡᵒv̲era (W, r) 257(q4, L4) ← chunks '{ (z, L) → 258 z2 ← every '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower z 259 (a, _, _) ← z2 260 (z2, L c̲at a ˡᵒl̲ossV (I, Y)) 261} p̲ower (ˡᵒa̲damStartV V, V ˡᵒl̲ossV (I, Y)) 262L4 263((_, _, _, d4, b4), _, _) ← q4 264learnedV ← ˡᵒd̲eltaV (A0, B0, d4, b4) 265⍝│ What it trains against LoRA, on this layer: 8 + r and 16 r numbers; 266⍝│ its projections, 16 r numbers, are shared by every 8 x 8 layer (or 267⍝│ regenerated from the seed). On the large model: one expert's VeRA 268⍝│ numbers, a layer of experts', and the projections they share, two 269⍝│ pairs, 2 r (d + f). 270(r ˡᵒp̲aramsV 8 8) c̲at r ˡᵒp̲arams 8 8 271(r ˡᵒe̲xpertV d c̲at f) c̲at (experts × r ˡᵒe̲xpertV d c̲at f) c̲at 2 × r × d + f 272⍝ -- end of the explorer at rank r 273 274⍝━━ The rank sweep 275 276⍝│ LoRA and LoRA-FA trained at each rank 1 .. 8 (300 steps each) from 277⍝│ the same start: LoRA reaches the best above at every rank; LoRA-FA, 278⍝│ its update held in A's row space, does not; nor does VeRA, which 279⍝│ trains 8 + k numbers. 280ᵘs̲tart ← { k → (W, 1 ˡᵒs̲eeded k c̲at 8, (8 c̲at k) r̲eshape 0.0, alpha ˡᵒs̲caling k) } 281ᵘl̲ora ← { k → 282 (a, _, _) ← 300 '{ e → lr ˡᵒa̲dam (e, (I, Y)) } p̲ower ˡᵒa̲damStart ᵘs̲tart k 283 ᵘl̲oss a 284} 285ᵘv̲era ← { k → 286 (a, _, _) ← 300 '{ e → lr ˡᵒa̲damV (e, (I, Y)) } p̲ower ˡᵒa̲damStartV 1 ˡᵒv̲era (W, k) 287 a ˡᵒl̲ossV (I, Y) 288} 289ᵘf̲a ← { k → 290 (a, _, _) ← 300 '{ e → lr ˡᵒa̲damFA (e, (I, Y)) } p̲ower ˡᵒa̲damStartFA ᵘs̲tart k 291 ᵘl̲oss a 292} 293'ᵘl̲ora e̲ach r̲ange 8 294'ᵘf̲a e̲ach r̲ange 8 295'ᵘv̲era e̲ach r̲ange 8