librarylibs/Norm/src/Norm.xtl
Norm: normalizing each row: layer norm and RMS norm, with a gain and a bias
Import it with an alias of your choice: "nm:" u_se< "Norm". Put libs/Norm/src on XETAL_PATH ("just path"); the reference is libs/Norm/docs. Names with l: are exported; h: names are private to this file.
Rows are examples (or tokens); each is normalized along its own numbers. A gain and a bias have one number per column and travel as a pair, (g, b); eps (0.00001) keeps a constant row from dividing by 0, so results are a hair from the exact ones (-0.99999 for -1).
Helpers (private)
ʰs̲pread : a -> b -> a
A value per row spread across that row's columns.
ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }
Normalizing
ˡs̲tandard : Float -> Float
s̲tandard m: each row less its mean, over its standard deviation
(population): mean 0 and variance 1 a row.
"nm:" u̲se< "Norm"
ⁿᵐs̲tandard 1 4 r̲eshape 1.0 2.0 3.0 6.0 -1.0690434404458735 -0.5345217202229368 0.0 1.6035651606688102
ˡs̲tandard ← { m → n ← f̲loat 2 s̲elect s̲hape m c ← m − (('+ r̲/₂ m) ÷ n) ʰs̲pread m c ÷ ((eps + ('+ r̲/₂ c × c) ÷ n) ^ 0.5) ʰs̲pread m }
ˡl̲ayer : (Float, Float) -> Float -> Float
(g, b) l̲ayer m: layer norm: each row standardized, then times the
gain g plus the bias b (a number per column).
"nm:" u̲se< "Norm"
(2.0 2.0, 1.0 1.0) ⁿᵐl̲ayer 1 2 r̲eshape 3.0 5.0 -0.9999900000749995 2.9999900000749995
ˡl̲ayer ← { (g, b) m → (((s̲hape m) r̲eshape g) × ˡs̲tandard m) + (s̲hape m) r̲eshape b }
ˡr̲ms : Float -> Float -> Float
g r̲ms m: RMS norm: each row over the square root of its mean
square, times the gain g (no centering, no bias).
"nm:" u̲se< "Norm"
1.0 1.0 ⁿᵐr̲ms 1 2 r̲eshape 3.0 4.0 0.8485277980128058 1.1313703973504077
ˡr̲ms ← { g m → n ← f̲loat 2 s̲elect s̲hape m ((s̲hape m) r̲eshape g) × m ÷ ((eps + ('+ r̲/₂ m × m) ÷ n) ^ 0.5) ʰs̲pread m }