programlibs/LoRA/demos/sizes.xtl
One projection adapted, then what adapters cost on a large MoE. Run with "just demo LoRA sizes".
An 8 x 8 projection at rank 3
i : Float
A frozen W, A 3 x 8, B 8 x 3 at alpha 6 (s = 2), and one input, by a fixed ripple (no chance).
i ← f̲loat r̲ange 64
a : (Float, Float, Float, Float)
a ← (W, 3 8 r̲eshape 0.2 × c̲os 0.3 × 24 t̲ake i × i, 8 3 r̲eshape 0.1 × s̲in 0.5 × 24 t̲ake i × i, 6.0 ˡᵒs̲caling 3)
y : Float
The adapted output, the factors apart, then the same by the merged matrix: they agree within 1e-12 (1).
y ← a ˡᵒf̲orward x
A SwiGLU expert, d 4096, f 14336
r : Int
For ranks 8, 16, 32 and 64 (a row each): the numbers one expert's adapters train, and the MiB they take in BF16 for one expert and for 64.
r ← 8 16 32 64
Used in: n, libs/LoRA/demos/sizes.xtl:33