librarylibs/Norm/src/Norm.xtl

Norm: normalizing each row: layer norm and RMS norm, with a gain and a bias

Import it with an alias of your choice: "nm:" u_se< "Norm". Put libs/Norm/src on XETAL_PATH ("just path"); the reference is libs/Norm/docs. Names with l: are exported; h: names are private to this file.

Rows are examples (or tokens); each is normalized along its own numbers. A gain and a bias have one number per column and travel as a pair, (g, b); eps (0.00001) keeps a constant row from dividing by 0, so results are a hair from the exact ones (-0.99999 for -1).

source

eps : Float

value (private) · line 12
eps ← 0.00001

Helpers (private)

ʰs̲pread : a -> b -> a

function (private) · line 17

A value per row spread across that row's columns.

ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }

Normalizing

ˡs̲tandard : Float -> Float

function · line 26

s̲tandard m: each row less its mean, over its standard deviation (population): mean 0 and variance 1 a row.

      "nm:" u̲se< "Norm"
      ⁿᵐs̲tandard 1 4 r̲eshape 1.0 2.0 3.0 6.0
-1.0690434404458735 -0.5345217202229368 0.0 1.6035651606688102
ˡs̲tandard ← { m →
  n ← f̲loat 2 s̲elect s̲hape m
  c ← m − (('+ r̲/₂ m) ÷ n) ʰs̲pread m
  c ÷ ((eps + ('+ r̲/₂ c × c) ÷ n) ^ 0.5) ʰs̲pread m
}
Used in: ˡl̲ayer, norm

ˡl̲ayer : (Float, Float) -> Float -> Float

function · line 37

(g, b) l̲ayer m: layer norm: each row standardized, then times the gain g plus the bias b (a number per column).

      "nm:" u̲se< "Norm"
      (2.0 2.0, 1.0 1.0) ⁿᵐl̲ayer 1 2 r̲eshape 3.0 5.0
-0.9999900000749995 2.9999900000749995
ˡl̲ayer ← { (g, b) m → (((s̲hape m) r̲eshape g) × ˡs̲tandard m) + (s̲hape m) r̲eshape b }

ˡr̲ms : Float -> Float -> Float

function · line 44

g r̲ms m: RMS norm: each row over the square root of its mean square, times the gain g (no centering, no bias).

      "nm:" u̲se< "Norm"
      1.0 1.0 ⁿᵐr̲ms 1 2 r̲eshape 3.0 4.0
0.8485277980128058 1.1313703973504077
ˡr̲ms ← { g m →
  n ← f̲loat 2 s̲elect s̲hape m
  ((s̲hape m) r̲eshape g) × m ÷ ((eps + ('+ r̲/₂ m × m) ÷ n) ^ 0.5) ʰs̲pread m
}