sourcelibs/Integrals/demos/gelu.xtl
1⍝!/usr/bin/env xetal
2⍝ Integrals: two continuous equations of machine learning by the
3⍝ macro. GELU, the activation of BERT and GPT, is x times the Gaussian
4⍝ CDF, x Phi(x) (Hendrycks and Gimpel, 2016), Phi(x) = 1/2 + the
5⍝ integral from 0 to x of the density exp(-u^2 / 2) / sqrt(2 pi): no
6⍝ improper bound needed. The tanh form most frameworks compute is an
7⍝ approximation; the integral says by how much. Then the KL divergence
8⍝ of two Gaussians, the integral of p log (p / q) over a truncation,
9⍝ beside its closed form. Run with "just demo Integrals".
10"integ:" u̲se< "Integrals"
11"act:" u̲se< "Activations"
12⍝ The standard Gaussian density.
13ᵘp̲hi ← { u → (e̲xp n̲eg (u × u) ÷ 2.0) ÷ (2.0 × p̲i @) ^ 0.5 }
14⍝ GELU at one x: x (1/2 + integral of the density from 0 to x), simpson on 32 steps.
15ᵘg̲elu ← { x → x × 0.5 + @ ⁱⁿᵗᵉᵍi̲ntegral< "integral u:=0.0..x by simpson 32 | u:p_hi u" }
16⍝ The tanh approximation (Activations' g_elu, written out): x/2 (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))).
17ᵘa̲pprox ← { x → 0.5 × x × 1.0 + ᵃᶜᵗt̲anh ((2.0 ÷ p̲i @) ^ 0.5) × x + 0.044715 × x ^ 3 }
18xs ← -3.0 -2.0 -1.0 -0.5 0.0 0.5 1.0 2.0 3.0
19g ← 'ᵘg̲elu e̲ach xs
20(f̲loat f̲loor 0.5 + 10000.0 × g) ÷ 10000.0
21⍝ The approximation's error at those points, and its largest over -4..4.
22(f̲loat f̲loor 0.5 + 1000000.0 × g − ᵘa̲pprox xs) ÷ 1000000.0
23fine ← -4.0 + 0.05 × f̲loat o̲ffsets 161
24(f̲loat f̲loor 0.5 + 1000000.0 × 'm̲ax r̲/ a̲bs ('ᵘg̲elu e̲ach fine) − ᵘa̲pprox fine) ÷ 1000000.0
25⍝ What the paper writes.
26@ ⁱⁿᵗᵉᵍt̲ex< "integral u:=0.0..x | u:p_hi u"
27⍝ KL(N(0, 1) || N(1, 2)): the integral of p log (p / q) over -10..10
28⍝ (the tails beyond are below 1e-20), beside the closed form
29⍝ log(s2 / s1) + (s1^2 + (m1 - m2)^2) / (2 s2^2) - 1/2.
30ᵘn̲ormal ← { (m, s) x → (e̲xp n̲eg ((x − m) ^ 2) ÷ 2.0 × s × s) ÷ s × (2.0 × p̲i @) ^ 0.5 }
31kl ← @ ⁱⁿᵗᵉᵍi̲ntegral< "integral x:=-10.0..10.0 by simpson 400 | ((0.0, 1.0) u:n_ormal x) * l_og ((0.0, 1.0) u:n_ormal x) / (1.0, 2.0) u:n_ormal x"
32closed ← (l̲og 2.0) + ((1.0 + 1.0) ÷ 2.0 × 4.0) − 0.5
33(f̲loat f̲loor 0.5 + 1000000.0 × kl c̲at closed) ÷ 1000000.0
34@ ⁱⁿᵗᵉᵍt̲ex< "integral x:=-10.0..10.0 | p(x) * l_og p(x) / q(x)"
ⁱⁿᵗᵉᵍi̲ntegral< expands to
({ @ →
g1:a ← 0.0
g2:b ← x
g3:h ← (g2:b − g1:a) ÷ 32.0
u ← g1:a + g3:h × f̲loat o̲ffsets 33
g4:w ← (1.0 c̲at (31 r̲eshape 4.0 2.0) c̲at 1.0) ÷ 3.0
g3:h × '+ r̲/ g4:w × ᵘp̲hi u
} @)ⁱⁿᵗᵉᵍt̲ex< expands to
"\\int_{0.0}^{x} u:p_hi u \\, du"ⁱⁿᵗᵉᵍi̲ntegral< expands to
({ @ →
g5:a ← -10.0
g6:b ← 10.0
g7:h ← (g6:b − g5:a) ÷ 400.0
x ← g5:a + g7:h × f̲loat o̲ffsets 401
g8:w ← (1.0 c̲at (399 r̲eshape 4.0 2.0) c̲at 1.0) ÷ 3.0
g7:h × '+ r̲/ g8:w × ((0.0, 1.0) ᵘn̲ormal x) × l̲og ((0.0, 1.0) ᵘn̲ormal x) ÷ (1.0, 2.0) ᵘn̲ormal x
} @)ⁱⁿᵗᵉᵍt̲ex< expands to
"\\int_{-10.0}^{10.0} p(x) * l_og p(x) / q(x) \\, dx"