sourcelibs/Norm/src/Norm.xtl
1⍝│ Norm: normalizing each row: layer norm and RMS norm, with a gain and a bias
2⍝│
3⍝│ Import it with an alias of your choice: "nm:" u_se< "Norm".
4⍝│ Put libs/Norm/src on XETAL_PATH ("just path"); the reference is libs/Norm/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ Rows are examples (or tokens); each is normalized along its own
8⍝│ numbers. A gain and a bias have one number per column and travel as
9⍝│ a pair, (g, b); eps (0.00001) keeps a constant row from dividing by
10⍝│ 0, so results are a hair from the exact ones (-0.99999 for -1).
11
12eps ← 0.00001
13
14⍝━━ Helpers (private)
15
16⍝│ A value per row spread across that row's columns.
17ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }
18
19⍝━━ Normalizing
20
21⍝│ s̲tandard m: each row less its mean, over its standard deviation
22⍝│ (population): mean 0 and variance 1 a row.
23⍝│ >> "nm:" u_se< "Norm"
24⍝│ >> nm:s_tandard 1 4 r_eshape 1.0 2.0 3.0 6.0
25⍝│ -1.0690434404458735 -0.5345217202229368 0.0 1.6035651606688102
26ˡs̲tandard ← { m →
27 n ← f̲loat 2 s̲elect s̲hape m
28 c ← m − (('+ r̲/₂ m) ÷ n) ʰs̲pread m
29 c ÷ ((eps + ('+ r̲/₂ c × c) ÷ n) ^ 0.5) ʰs̲pread m
30}
31
32⍝│ (g, b) l̲ayer m: layer norm: each row standardized, then times the
33⍝│ gain g plus the bias b (a number per column).
34⍝│ >> "nm:" u_se< "Norm"
35⍝│ >> (2.0 2.0, 1.0 1.0) nm:l_ayer 1 2 r_eshape 3.0 5.0
36⍝│ -0.9999900000749995 2.9999900000749995
37ˡl̲ayer ← { (g, b) m → (((s̲hape m) r̲eshape g) × ˡs̲tandard m) + (s̲hape m) r̲eshape b }
38
39⍝│ g r̲ms m: RMS norm: each row over the square root of its mean
40⍝│ square, times the gain g (no centering, no bias).
41⍝│ >> "nm:" u_se< "Norm"
42⍝│ >> 1.0 1.0 nm:r_ms 1 2 r_eshape 3.0 4.0
43⍝│ 0.8485277980128058 1.1313703973504077
44ˡr̲ms ← { g m →
45 n ← f̲loat 2 s̲elect s̲hape m
46 ((s̲hape m) r̲eshape g) × m ÷ ((eps + ('+ r̲/₂ m × m) ÷ n) ^ 0.5) ʰs̲pread m
47}