sourcelibs/Norm/src/Norm.xtl

1⍝│ Norm: normalizing each row: layer norm and RMS norm, with a gain and a bias 2⍝│ 3⍝│ Import it with an alias of your choice: "nm:" u_se< "Norm". 4⍝│ Put libs/Norm/src on XETAL_PATH ("just path"); the reference is libs/Norm/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ Rows are examples (or tokens); each is normalized along its own 8⍝│ numbers. A gain and a bias have one number per column and travel as 9⍝│ a pair, (g, b); eps (0.00001) keeps a constant row from dividing by 10⍝│ 0, so results are a hair from the exact ones (-0.99999 for -1). 11 12eps ← 0.00001 13 14⍝━━ Helpers (private) 15 16⍝│ A value per row spread across that row's columns. 17ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m } 18 19⍝━━ Normalizing 20 21⍝│ s̲tandard m: each row less its mean, over its standard deviation 22⍝│ (population): mean 0 and variance 1 a row. 23⍝│ >> "nm:" u_se< "Norm" 24⍝│ >> nm:s_tandard 1 4 r_eshape 1.0 2.0 3.0 6.0 25⍝│ -1.0690434404458735 -0.5345217202229368 0.0 1.6035651606688102 26ˡs̲tandard ← { m → 27 n ← f̲loat 2 s̲elect s̲hape m 28 c ← m − (('+ r̲/₂ m) ÷ n) ʰs̲pread m 29 c ÷ ((eps + ('+ r̲/₂ c × c) ÷ n) ^ 0.5) ʰs̲pread m 30} 31 32⍝│ (g, b) l̲ayer m: layer norm: each row standardized, then times the 33⍝│ gain g plus the bias b (a number per column). 34⍝│ >> "nm:" u_se< "Norm" 35⍝│ >> (2.0 2.0, 1.0 1.0) nm:l_ayer 1 2 r_eshape 3.0 5.0 36⍝│ -0.9999900000749995 2.9999900000749995 37ˡl̲ayer ← { (g, b) m → (((s̲hape m) r̲eshape g) × ˡs̲tandard m) + (s̲hape m) r̲eshape b } 38 39⍝│ g r̲ms m: RMS norm: each row over the square root of its mean 40⍝│ square, times the gain g (no centering, no bias). 41⍝│ >> "nm:" u_se< "Norm" 42⍝│ >> 1.0 1.0 nm:r_ms 1 2 r_eshape 3.0 4.0 43⍝│ 0.8485277980128058 1.1313703973504077 44ˡr̲ms ← { g m → 45 n ← f̲loat 2 s̲elect s̲hape m 46 ((s̲hape m) r̲eshape g) × m ÷ ((eps + ('+ r̲/₂ m × m) ÷ n) ^ 0.5) ʰs̲pread m 47}