programdemos/lora-hello/lora-hello.xtl
LoRA hello: what does a small adapter add to a layer whose weights do not change? The question behind it: how can a model learn something new without every one of its numbers being retrained (the cost) or overwritten (the forgetting)? LoRA's answer is to freeze the weights and train a small correction beside them. Here is one layer of that, shrunk to 4 rows and 4 columns so every number can be read: a frozen table W and an adapter of rank 1, with X_eTaL-ML's LoRA library. Nothing is trained here (the adapter's numbers are made up): this shows what an adapter is, where it sits, what it adds and what it costs. The layer is y = W x + s B (A x): W stays as it is, and only A (1 row of 4) and B (1 column of 4) would be trained.
The layer
W : Float
The frozen table W (16 made-up numbers, from a sine so they need not be typed), the adapter's two thin tables A and B, one input x, and the strength s = alpha / r (alpha is 2, the rank r is 1).
W ← 4 4 r̲eshape s̲in 0.7 × f̲loat r̲ange 16
x : Float
x ← 1.0 2.0 3.0 4.0
s : Float
s ← 2.0 ˡᵒs̲caling 1
What the adapter adds
h : Float
The squeeze: h = A x, the input brought down to one number (the rank is 1). Printed after the strength s.
h ← A '+ '× i̲nner x
base : Float
What the frozen layer gives, W x; what the adapted layer gives, W x + s B (A x); and the correction the adapter adds, s B (A x). Then a check that the two outputs differ by exactly the correction: 1 means yes.
base ← W '+ '× i̲nner x
The correction as one table, and the merge
table : Float
Written out as a table, the correction is s B A: 4 rows and 4 columns, every row a multiple of A, so its rank is 1. Added to W it gives one table, the merged one, whose output is the same as W and the adapter together: 1 means yes.
table ← ˡᵒd̲elta (A, B, s)
What it costs
counts : Int
The numbers an adapter of rank 1 trains on a table of 4 rows and 4 columns, r (rows + columns), beside the 16 of the table itself.
counts ← (1 ˡᵒp̲arams 4 4) c̲at 16
On a layer of a real model
big : Int
The same count where it matters: one table of 4096 rows and 4096 columns (a size found in a 7-billion-parameter model) at rank 16: the numbers the adapter trains, the numbers of the table, and the first as a percentage of the second.
big ← (16 ˡᵒp̲arams 4096 4096) c̲at 4096 × 4096