programlibs/LoRA/demos/sizes.xtl

One projection adapted, then what adapters cost on a large MoE. Run with "just demo LoRA sizes".

source · imports lo: libs/LoRA/src/LoRA.xtl libs/LoRA/src/LoRA.xtlm

An 8 x 8 projection at rank 3

i : Float

value · line 11

A frozen W, A 3 x 8, B 8 x 3 at alpha 6 (s = 2), and one input, by a fixed ripple (no chance).

i ← f̲loat r̲ange 64
Used in: W, a, x

W : Float

value · line 12
W ← 8 8 r̲eshape s̲in 0.7 × i × i
Used in: a

a : (Float, Float, Float, Float)

value · line 13
a ← (W, 3 8 r̲eshape 0.2 × c̲os 0.3 × 24 t̲ake i × i, 8 3 r̲eshape 0.1 × s̲in 0.5 × 24 t̲ake i × i, 6.0 ˡᵒs̲caling 3)

x : Float

value · line 14
x ← c̲os 0.4 × 8 t̲ake i

y : Float

value · line 17

The adapted output, the factors apart, then the same by the merged matrix: they agree within 1e-12 (1).

y ← a ˡᵒf̲orward x

A : Float

value · line 22
(_, A, B, s) ← a

B : Float

value · line 22
(_, A, B, s) ← a

s : Float

value · line 22
(_, A, B, s) ← a

A SwiGLU expert, d 4096, f 14336

r : Int

value · line 29

For ranks 8, 16, 32 and 64 (a row each): the numbers one expert's adapters train, and the MiB they take in BF16 for one expert and for 64.

r ← 8 16 32 64

ᵘe̲xpert : Num a => a -> a

function · line 30
ᵘe̲xpert ← { k → k ˡᵒe̲xpert 4096 14336 }
Used in: n

n : Int

value · line 31
n ← 'ᵘe̲xpert e̲ach r

mib : Float

value · line 32
mib ← (16 ˡᵒb̲ytes n) ÷ 1048576.0