sourcelibs/LoRA/demos/sizes.xtl

1⍝!/usr/bin/env xetal 2⍝│ One projection adapted, then what adapters cost on a large MoE. Run 3⍝│ with "just demo LoRA sizes". 4 5"lo:" u̲se< "LoRA" 6 7⍝━━ An 8 x 8 projection at rank 3 8 9⍝│ A frozen W, A 3 x 8, B 8 x 3 at alpha 6 (s = 2), and one input, by a 10⍝│ fixed ripple (no chance). 11i ← f̲loat r̲ange 64 12W ← 8 8 r̲eshape s̲in 0.7 × i × i 13a ← (W, 3 8 r̲eshape 0.2 × c̲os 0.3 × 24 t̲ake i × i, 8 3 r̲eshape 0.1 × s̲in 0.5 × 24 t̲ake i × i, 6.0 ˡᵒs̲caling 3) 14x ← c̲os 0.4 × 8 t̲ake i 15⍝│ The adapted output, the factors apart, then the same by the merged 16⍝│ matrix: they agree within 1e-12 (1). 17y ← a ˡᵒf̲orward x 18y 190.000000000001 > 'm̲ax r̲/ a̲bs y − (ˡᵒm̲erge a) '+ '× i̲nner x 20⍝│ The numbers trained against the matrix's, and the update's rank. 21(3 ˡᵒp̲arams 8 8) c̲at 64 22(_, A, B, s) ← a 23ˡᵒr̲ank ˡᵒd̲elta (A, B, s) 24 25⍝━━ A SwiGLU expert, d 4096, f 14336 26 27⍝│ For ranks 8, 16, 32 and 64 (a row each): the numbers one expert's adapters train, 28⍝│ and the MiB they take in BF16 for one expert and for 64. 29r ← 8 16 32 64 30ᵘe̲xpert ← { k → k ˡᵒe̲xpert 4096 14336 } 31n ← 'ᵘe̲xpert e̲ach r 32mib ← (16 ˡᵒb̲ytes n) ÷ 1048576.0 33o̲\ 4 4 r̲eshape (f̲loat r) c̲at (f̲loat n) c̲at mib c̲at 64.0 × mib