sourcelibs/Norm/demos/rows.xtl

1⍝!/usr/bin/env xetal 2⍝│ Two rows on very different scales, made alike by layer norm and by 3⍝│ RMS norm. Run with "just demo-lib Norm rows". 4 5"nm:" u̲se< "Norm" 6 7m ← 2 4 r̲eshape 0.01 0.02 0.03 0.06 100.0 200.0 300.0 600.0 8ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 } 9⍝│ Layer norm: nearly the same row twice (a row's scale does not 10⍝│ matter), but not quite: the small row's variance (0.00035) is not 11⍝│ large next to eps (0.00001), which keeps a constant row from dividing 12⍝│ by 0; the large row's is. 13ᵘr̲ound (4 r̲eshape 1.0, 4 r̲eshape 0.0) ⁿᵐl̲ayer m 14⍝│ RMS norm: likewise, and not centered. 15ᵘr̲ound (4 r̲eshape 1.0) ⁿᵐr̲ms m