librarylibs/Activations/src/Activations.xtl
Activations: activation functions, a derivative for each, and the gated layers (GLU, ReGLU, GEGLU, SwiGLU)
Import it with an alias of your choice: "act:" u_se< "Activations". Put libs/Activations/src on XETAL_PATH ("just path"); the reference is libs/Activations/docs. Names with l: are exported; h: names are private to this file.
Every activation works item by item on a Float array of any shape,
so a batch (one example per row) goes through in one call. A
setting comes first (0.1 ᵃᶜᵗl̲eaky x, 1.5 ᵃᶜᵗs̲wish x); PReLU's
slopes are one per column (the last axis), as a bias is. Each
function f has its derivative d_f written in terms of f's input
(ᵃᶜᵗd̲gelu z, not of its output): X_eTaL has no automatic
differentiation, and the smooth activations cannot be written at
their output as ReLU, tanh and sigmoid can. A gated layer takes its
two weight arrays as a tuple, (W, V) ᵃᶜᵗs̲wiglu x: the activation
on x W, times the linear x V (Shazeer 2020's order).
tanh and erf are X_eTaL's builtins, t̲anh and e̲rf (libm's, at
full accuracy: docs/xetal-asks.md M19, landed); ᵃᶜᵗt̲anh is the
builtin under this library's name, beside its other activations.
Helpers (private)
ʰg̲eluTanh : Float -> Float
The inner tanh of GELU's approximation, at the point u(x) = sqrt(2/pi) (x + 0.044715 x^3).
ʰg̲eluTanh ← { x → ˡt̲anh 0.7978845608028654 × x + 0.044715 × x × x × x }
Activations
ˡl̲eaky : Float -> Float -> Float
a l̲eaky x: leaky ReLU, x where positive and a times x elsewhere.
"act:" u̲se< "Activations"
0.1 ᵃᶜᵗl̲eaky -2.0 3.0 -0.2 3.0
ˡl̲eaky ← { a x → (x m̲ax 0.0) + a × x m̲in 0.0 }
ˡp̲relu : Float -> Float -> Float
a p̲relu x: PReLU (He et al. 2015), a leaky ReLU whose slope a is
one number per column (a's length is x's last extent; a scalar is
l_eaky). The slopes are learned; the library only applies them.
"act:" u̲se< "Activations"
0.0 0.5 ᵃᶜᵗp̲relu 2 2 r̲eshape -2.0 -2.0 1.0 -4.0 0.0 -1.0 1.0 -2.0
ˡp̲relu ← { a x → (x m̲ax 0.0) + ((s̲hape x) r̲eshape a) × x m̲in 0.0 }
ˡe̲lu : Float -> Float -> Float
a e̲lu x: ELU (Clevert et al. 2016), x where positive and
a (e^x - 1) elsewhere: smooth at 0 when a is 1, bounded below by -a.
"act:" u̲se< "Activations"
1.0 ᵃᶜᵗe̲lu -1.0 2.0 -0.6321205588285577 2.0
ˡe̲lu ← { a x → (x m̲ax 0.0) + a × (e̲xp x m̲in 0.0) − 1.0 }
ˡs̲elu : Float -> Float
s̲elu x: SELU (Klambauer et al. 2017), lambda times an ELU with
alpha, the two constants that keep a standard-normal input's mean 0
and variance 1 through a layer (a self-normalizing network needs no
norm layer).
"act:" u̲se< "Activations"
ᵃᶜᵗs̲elu 0.0 1.0 0.0 1.0507009873554805
ˡs̲elu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡe̲lu x }
ˡs̲igmoid : Num a => a -> Float
s̲igmoid x: 1 / (1 + e^-x), item by item.
"act:" u̲se< "Activations"
ᵃᶜᵗs̲igmoid 0.0 2.0 0.5 0.8807970779778823
ˡs̲igmoid ← { x → 1.0 ÷ 1.0 + e̲xp n̲eg x }
ˡt̲anh : Float -> Float
t̲anh x: the hyperbolic tangent, X_eTaL's builtin (exact near 0,
1 without overflow far out).
"act:" u̲se< "Activations"
ᵃᶜᵗt̲anh 0.0 1.0 0.0 0.7615941559557649
ˡt̲anh ← { x → t̲anh x }
ˡs̲oftplus : Float -> Float
s̲oftplus x: log(1 + e^x), a smooth ReLU, in the form that does
not overflow: max(x, 0) + log(1 + e^-|x|).
"act:" u̲se< "Activations"
ᵃᶜᵗs̲oftplus 0.0 1000.0 0.6931471805599453 1000.0
ˡs̲oftplus ← { x → (x m̲ax 0.0) + l̲og 1.0 + e̲xp n̲eg a̲bs x }
ˡs̲ilu : Float -> Float
s̲ilu x: SiLU, x sigmoid(x) (Hendrycks and Gimpel 2016; Swish with
beta 1, Ramachandran et al. 2017): smooth, dips to -0.278 below 0.
"act:" u̲se< "Activations"
ᵃᶜᵗs̲ilu -1.0 1.0 -0.2689414213699951 0.7310585786300049
ˡs̲ilu ← { x → x × ˡs̲igmoid x }
ˡs̲wish : Float -> Float -> Float
b s̲wish x: Swish, x sigmoid(b x); b = 1 is s_ilu, a large b tends
to ReLU, b = 1.702 is close to GELU.
"act:" u̲se< "Activations"
1.702 ᵃᶜᵗs̲wish 1.0 0.8457957659328212
ˡs̲wish ← { b x → x × ˡs̲igmoid b × x }
ˡg̲elu : Float -> Float
g̲elu x: GELU (Hendrycks and Gimpel 2016) in the tanh form the
frameworks compute, x/2 (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))),
within 4.73e-4 of the exact g_eluErf.
"act:" u̲se< "Activations"
ᵃᶜᵗg̲elu -1.0 1.0 -0.1588080093917233 0.8411919906082768
ˡg̲elu ← { x → 0.5 × x × 1.0 + ʰg̲eluTanh x }
ˡg̲eluErf : Float -> Float
g̲eluErf x: the exact GELU, x Phi(x) = x/2 (1 + erf(x / sqrt 2)),
Phi the normal CDF (erf X_eTaL's builtin).
"act:" u̲se< "Activations"
ᵃᶜᵗg̲eluErf 1.0 0.8413447460685429
ˡg̲eluErf ← { x → 0.5 × x × 1.0 + e̲rf x ÷ 1.4142135623730951 }
ˡm̲ish : Float -> Float
m̲ish x: Mish (Misra 2019), x tanh(softplus(x)): smooth, dips to
-0.309 below 0, finite at any x.
"act:" u̲se< "Activations"
ᵃᶜᵗm̲ish -1.0 1.0 -0.30340146137410895 0.8650983882673103
ˡm̲ish ← { x → x × ˡt̲anh ˡs̲oftplus x }
ˡh̲ardSigmoid : Float -> Float
h̲ardSigmoid x: min(max(x + 3, 0), 6) / 6 (Howard et al. 2019,
MobileNetV3): a sigmoid of three straight lines, no e_xp.
"act:" u̲se< "Activations"
ᵃᶜᵗh̲ardSigmoid -4.0 0.0 1.5 0.0 0.5 0.75
ˡh̲ardSigmoid ← { x → (6.0 m̲in 0.0 m̲ax x + 3.0) ÷ 6.0 }
ˡh̲ardSwish : Float -> Float
h̲ardSwish x: x h_ardSigmoid(x), the same paper's SiLU of straight
lines, within 0.15 of s_ilu everywhere.
"act:" u̲se< "Activations"
ᵃᶜᵗh̲ardSwish -4.0 1.0 4.0 -0.0 0.6666666666666666 4.0
ˡh̲ardSwish ← { x → x × ˡh̲ardSigmoid x }
The gated layers
ˡg̲lu : (Float, Float) -> Float -> Float
(W, V) g̲lu x: GLU (Dauphin et al. 2017), sigmoid(x W) * (x V):
the activation on one projection of each row of x, times the other
(a gate), item by item. x is a batch (one row each), W and V both
inputs by hidden.
"act:" u̲se< "Activations"
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᶜᵗg̲lu 1 2 r̲eshape 0.0 1.0 0.0 1.4621171572600098
ˡg̲lu ← { (W, V) x → (ˡs̲igmoid x '+ '× i̲nner W) × x '+ '× i̲nner V }
ˡr̲eglu : (Float, Float) -> Float -> Float
(W, V) r̲eglu x: relu(x W) * (x V) (Shazeer 2020).
ˡr̲eglu ← { (W, V) x → (ˡr̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
ˡg̲eglu : (Float, Float) -> Float -> Float
(W, V) g̲eglu x: gelu(x W) * (x V) (Shazeer 2020).
ˡg̲eglu ← { (W, V) x → (ˡg̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
ˡs̲wiglu : (Float, Float) -> Float -> Float
(W, V) s̲wiglu x: silu(x W) * (x V) (Shazeer 2020; LLaMA's
feed-forward layer, with a hidden width of 2/3 the usual to keep
the parameter count).
"act:" u̲se< "Activations"
(2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᶜᵗs̲wiglu 1 2 r̲eshape 0.0 1.0 0.0 1.4621171572600098
ˡs̲wiglu ← { (W, V) x → (ˡs̲ilu x '+ '× i̲nner W) × x '+ '× i̲nner V }
Derivatives, in terms of each function's input
ˡd̲relu : Float -> Float
d̲relu x: 1 where x is positive, else 0.
"act:" u̲se< "Activations"
ᵃᶜᵗd̲relu -1.5 0.0 2.0 0.0 0.0 1.0
ˡd̲relu ← { x → f̲loat x > 0.0 }
ˡd̲leaky : Float -> Float -> Float
a d̲leaky x: 1 where x is positive, else a.
ˡd̲leaky ← { a x → 1.0 + (a − 1.0) × f̲loat x ≤ 0.0 }
ˡd̲prelu : Float -> Float -> Float
a d̲prelu x: 1 where x is positive, else the column's slope.
ˡd̲prelu ← { a x → 1.0 + (((s̲hape x) r̲eshape a) − 1.0) × f̲loat x ≤ 0.0 }
ˡd̲elu : Float -> Float -> Float
a d̲elu x: 1 where x is positive, else a e^x.
ˡd̲elu ← { a x → (f̲loat x > 0.0) + a × (f̲loat x ≤ 0.0) × e̲xp x m̲in 0.0 }
ˡd̲selu : Float -> Float
d̲selu x: lambda where x is positive, else lambda alpha e^x.
ˡd̲selu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡd̲elu x }
ˡd̲sigmoid : Num a => a -> Float
d̲sigmoid x: s (1 - s) for s = sigmoid(x): at most 1/4, at 0.
"act:" u̲se< "Activations"
ᵃᶜᵗd̲sigmoid 0.0 0.25
ˡd̲sigmoid ← { x → s ← ˡs̲igmoid x◆ s × 1.0 − s }
ˡd̲tanh : Float -> Float
d̲tanh x: 1 - tanh(x)^2.
ˡd̲tanh ← { x → t ← ˡt̲anh x◆ 1.0 − t × t }
ˡd̲silu : Float -> Float
d̲silu x: s (1 + x (1 - s)) for s = sigmoid(x).
ˡd̲silu ← { x → s ← ˡs̲igmoid x◆ s × 1.0 + x × 1.0 − s }
ˡd̲swish : Float -> Float -> Float
b d̲swish x: s + b x s (1 - s) for s = sigmoid(b x).
ˡd̲swish ← { b x → s ← ˡs̲igmoid b × x◆ s + b × x × s × 1.0 − s }
ˡd̲gelu : Float -> Float
d̲gelu x: the tanh form's derivative, (1 + t)/2 + x/2 (1 - t^2) sqrt(2/pi) (1 + 3 * 0.044715 x^2), t the inner tanh.
"act:" u̲se< "Activations"
ᵃᶜᵗd̲gelu 0.0 0.5
ˡd̲gelu ← { x → t ← ʰg̲eluTanh x◆ (0.5 × 1.0 + t) + 0.5 × x × (1.0 − t × t) × 0.7978845608028654 × 1.0 + 0.134145 × x × x }
ˡd̲geluErf : Float -> Float
d̲geluErf x: Phi(x) + x phi(x), the normal CDF and density.
ˡd̲geluErf ← { x → (0.5 × 1.0 + e̲rf x ÷ 1.4142135623730951) + x × 0.3989422804014327 × e̲xp n̲eg x × x ÷ 2.0 }
ˡd̲mish : Float -> Float
d̲mish x: t + x (1 - t^2) sigmoid(x) for t = tanh(softplus(x)).
ˡd̲mish ← { x → t ← ˡt̲anh ˡs̲oftplus x◆ t + x × (1.0 − t × t) × ˡs̲igmoid x }
ˡd̲hardSigmoid : Float -> Float
d̲hardSigmoid x: 1/6 where |x| < 3, else 0.
ˡd̲hardSigmoid ← { x → (f̲loat (a̲bs x) < 3.0) ÷ 6.0 }