sourcelibs/LoRA/demos/sizes.xtl
1⍝!/usr/bin/env xetal
2⍝│ One projection adapted, then what adapters cost on a large MoE. Run
3⍝│ with "just demo LoRA sizes".
4
5"lo:" u̲se< "LoRA"
6
7⍝━━ An 8 x 8 projection at rank 3
8
9⍝│ A frozen W, A 3 x 8, B 8 x 3 at alpha 6 (s = 2), and one input, by a
10⍝│ fixed ripple (no chance).
11i ← f̲loat r̲ange 64
12W ← 8 8 r̲eshape s̲in 0.7 × i × i
13a ← (W, 3 8 r̲eshape 0.2 × c̲os 0.3 × 24 t̲ake i × i, 8 3 r̲eshape 0.1 × s̲in 0.5 × 24 t̲ake i × i, 6.0 ˡᵒs̲caling 3)
14x ← c̲os 0.4 × 8 t̲ake i
15⍝│ The adapted output, the factors apart, then the same by the merged
16⍝│ matrix: they agree within 1e-12 (1).
17y ← a ˡᵒf̲orward x
18y
190.000000000001 > 'm̲ax r̲/ a̲bs y − (ˡᵒm̲erge a) '+ '× i̲nner x
20⍝│ The numbers trained against the matrix's, and the update's rank.
21(3 ˡᵒp̲arams 8 8) c̲at 64
22(_, A, B, s) ← a
23ˡᵒr̲ank ˡᵒd̲elta (A, B, s)
24
25⍝━━ A SwiGLU expert, d 4096, f 14336
26
27⍝│ For ranks 8, 16, 32 and 64 (a row each): the numbers one expert's adapters train,
28⍝│ and the MiB they take in BF16 for one expert and for 64.
29r ← 8 16 32 64
30ᵘe̲xpert ← { k → k ˡᵒe̲xpert 4096 14336 }
31n ← 'ᵘe̲xpert e̲ach r
32mib ← (16 ˡᵒb̲ytes n) ÷ 1048576.0
33o̲\ 4 4 r̲eshape (f̲loat r) c̲at (f̲loat n) c̲at mib c̲at 64.0 × mib