lora-hello.xtl

1⍝!/usr/bin/env xetal
2⍝│ LoRA hello: what does a small adapter add to a layer whose weights
3⍝│ do not change? The question behind it: how can a model learn
4⍝│ something new without every one of its numbers being retrained (the
5⍝│ cost) or overwritten (the forgetting)? LoRA's answer is to freeze
6⍝│ the weights and train a small correction beside them. Here is one
7⍝│ layer of that, shrunk to 4 rows and 4 columns so every number can be
8⍝│ read: a frozen table W and an adapter of rank 1, with X_eTaL-ML's
9⍝│ LoRA library. Nothing is trained here (the adapter's numbers are
10⍝│ made up): this shows what an adapter is, where it sits, what it adds
11⍝│ and what it costs. The layer is y = W x + s B (A x): W stays as it
12⍝│ is, and only A (1 row of 4) and B (1 column of 4) would be trained.
13
14"lo:" u̲se< "LoRA"
15
16⍝━━ The layer
17
18⍝│ The frozen table W (16 made-up numbers, from a sine so they need
19⍝│ not be typed), the adapter's two thin tables A and B, one input x,
20⍝│ and the strength s = alpha / r (alpha is 2, the rank r is 1).
21W ← 4 4 r̲eshape s̲in 0.7 × f̲loat r̲ange 16
22A ← 1 4 r̲eshape 0.5 -0.5 1.0 0.25
23B ← 4 1 r̲eshape 1.0 0.0 -1.0 0.5
24x ← 1.0 2.0 3.0 4.0
25s ← 2.0 ˡᵒs̲caling 1
26⍝ The numbers of the layer, for the page to show: W row by row, the
27⍝ one row of A, the one column of B.
28r̲avel W
29r̲avel A
30r̲avel B
31
32⍝━━ What the adapter adds
33
34⍝│ The squeeze: h = A x, the input brought down to one number (the
35⍝│ rank is 1). Printed after the strength s.
36h ← A '+ '× i̲nner x
37s c̲at h
38⍝ The squeeze written out: each number of A times the matching number
39⍝ of x (h is their sum). Then that one number times the strength.
40x × r̲avel A
41s × h
42
43⍝│ What the frozen layer gives, W x; what the adapted layer gives,
44⍝│ W x + s B (A x); and the correction the adapter adds, s B (A x).
45⍝│ Then a check that the two outputs differ by exactly the correction:
46⍝│ 1 means yes.
47base ← W '+ '× i̲nner x
48y ← (W, A, B, s) ˡᵒf̲orward x
49correction ← (ˡᵒd̲elta (A, B, s)) '+ '× i̲nner x
50⍝ The first row of W times x written out: its sum is the first number
51⍝ of base.
52x × 4 t̲ake r̲avel W
53base
54y
55correction
560.000000000001 > 'm̲ax r̲/ a̲bs correction − y − base
57
58⍝━━ The correction as one table, and the merge
59
60⍝│ Written out as a table, the correction is s B A: 4 rows and 4
61⍝│ columns, every row a multiple of A, so its rank is 1. Added to W it
62⍝│ gives one table, the merged one, whose output is the same as W and
63⍝│ the adapter together: 1 means yes.
64table ← ˡᵒd̲elta (A, B, s)
65merged ← ˡᵒm̲erge (W, A, B, s)
66r̲avel table
67ˡᵒr̲ank table
680.000000000001 > 'm̲ax r̲/ a̲bs y − merged '+ '× i̲nner x
69
70⍝━━ What it costs
71
72⍝│ The numbers an adapter of rank 1 trains on a table of 4 rows and 4
73⍝│ columns, r (rows + columns), beside the 16 of the table itself.
74counts ← (1 ˡᵒp̲arams 4 4) c̲at 16
75counts
76
77⍝━━ On a layer of a real model
78
79⍝│ The same count where it matters: one table of 4096 rows and 4096
80⍝│ columns (a size found in a 7-billion-parameter model) at rank 16:
81⍝│ the numbers the adapter trains, the numbers of the table, and the
82⍝│ first as a percentage of the second.
83big ← (16 ˡᵒp̲arams 4096 4096) c̲at 4096 × 4096
84big
85100.0 × (f̲loat 16 ˡᵒp̲arams 4096 4096) ÷ f̲loat 4096 × 4096