librarylibs/Activations/src/Activations.xtl

Activations: activation functions, a derivative for each, and the gated layers (GLU, ReGLU, GEGLU, SwiGLU)

Import it with an alias of your choice: "act:" u_se< "Activations". Put libs/Activations/src on XETAL_PATH ("just path"); the reference is libs/Activations/docs. Names with l: are exported; h: names are private to this file.

Every activation works item by item on a Float array of any shape, so a batch (one example per row) goes through in one call. A setting comes first (0.1 ᵃᶜᵗl̲eaky x, 1.5 ᵃᶜᵗs̲wish x); PReLU's slopes are one per column (the last axis), as a bias is. Each function f has its derivative d_f written in terms of f's input (ᵃᶜᵗd̲gelu z, not of its output): X_eTaL has no automatic differentiation, and the smooth activations cannot be written at their output as ReLU, tanh and sigmoid can. A gated layer takes its two weight arrays as a tuple, (W, V) ᵃᶜᵗs̲wiglu x: the activation on x W, times the linear x V (Shazeer 2020's order).

tanh and erf are X_eTaL's builtins, t̲anh and e̲rf (libm's, at full accuracy: docs/xetal-asks.md M19, landed); ᵃᶜᵗt̲anh is the builtin under this library's name, beside its other activations.

source

Helpers (private)

ʰg̲eluTanh : Float -> Float

function (private) · line 25

The inner tanh of GELU's approximation, at the point u(x) = sqrt(2/pi) (x + 0.044715 x^3).

ʰg̲eluTanh ← { x → ˡt̲anh 0.7978845608028654 × x + 0.044715 × x × x × x }

Activations

ˡr̲elu : Float -> Float

function · line 33

r̲elu x: max(x, 0), item by item.

      "act:" u̲se< "Activations"
      ᵃᶜᵗr̲elu -1.5 0.0 2.0
0.0 0.0 2.0
ˡr̲elu ← { x → x m̲ax 0.0 }

ˡl̲eaky : Float -> Float -> Float

function · line 39

a l̲eaky x: leaky ReLU, x where positive and a times x elsewhere.

      "act:" u̲se< "Activations"
      0.1 ᵃᶜᵗl̲eaky -2.0 3.0
-0.2 3.0
ˡl̲eaky ← { a x → (x m̲ax 0.0) + a × x m̲in 0.0 }
Used in: values, rect

ˡp̲relu : Float -> Float -> Float

function · line 48

a p̲relu x: PReLU (He et al. 2015), a leaky ReLU whose slope a is one number per column (a's length is x's last extent; a scalar is l_eaky). The slopes are learned; the library only applies them.

      "act:" u̲se< "Activations"
      0.0 0.5 ᵃᶜᵗp̲relu 2 2 r̲eshape -2.0 -2.0 1.0 -4.0
0.0 -1.0
1.0 -2.0
ˡp̲relu ← { a x → (x m̲ax 0.0) + ((s̲hape x) r̲eshape a) × x m̲in 0.0 }
Used in: values

ˡe̲lu : Float -> Float -> Float

function · line 55

a e̲lu x: ELU (Clevert et al. 2016), x where positive and a (e^x - 1) elsewhere: smooth at 0 when a is 1, bounded below by -a.

      "act:" u̲se< "Activations"
      1.0 ᵃᶜᵗe̲lu -1.0 2.0
-0.6321205588285577 2.0
ˡe̲lu ← { a x → (x m̲ax 0.0) + a × (e̲xp x m̲in 0.0) − 1.0 }

ˡs̲elu : Float -> Float

function · line 64

s̲elu x: SELU (Klambauer et al. 2017), lambda times an ELU with alpha, the two constants that keep a standard-normal input's mean 0 and variance 1 through a layer (a self-normalizing network needs no norm layer).

      "act:" u̲se< "Activations"
      ᵃᶜᵗs̲elu 0.0 1.0
0.0 1.0507009873554805
ˡs̲elu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡe̲lu x }
Used in: values, rect

ˡs̲igmoid : Num a => a -> Float

function · line 70

s̲igmoid x: 1 / (1 + e^-x), item by item.

      "act:" u̲se< "Activations"
      ᵃᶜᵗs̲igmoid 0.0 2.0
0.5 0.8807970779778823
ˡs̲igmoid ← { x → 1.0 ÷ 1.0 + e̲xp n̲eg x }

ˡt̲anh : Float -> Float

function · line 77

t̲anh x: the hyperbolic tangent, X_eTaL's builtin (exact near 0, 1 without overflow far out).

      "act:" u̲se< "Activations"
      ᵃᶜᵗt̲anh 0.0 1.0
0.0 0.7615941559557649
ˡt̲anh ← { x → t̲anh x }

ˡs̲oftplus : Float -> Float

function · line 84

s̲oftplus x: log(1 + e^x), a smooth ReLU, in the form that does not overflow: max(x, 0) + log(1 + e^-|x|).

      "act:" u̲se< "Activations"
      ᵃᶜᵗs̲oftplus 0.0 1000.0
0.6931471805599453 1000.0
ˡs̲oftplus ← { x → (x m̲ax 0.0) + l̲og 1.0 + e̲xp n̲eg a̲bs x }

ˡs̲ilu : Float -> Float

function · line 91

s̲ilu x: SiLU, x sigmoid(x) (Hendrycks and Gimpel 2016; Swish with beta 1, Ramachandran et al. 2017): smooth, dips to -0.278 below 0.

      "act:" u̲se< "Activations"
      ᵃᶜᵗs̲ilu -1.0 1.0
-0.2689414213699951 0.7310585786300049
ˡs̲ilu ← { x → x × ˡs̲igmoid x }

ˡs̲wish : Float -> Float -> Float

function · line 98

b s̲wish x: Swish, x sigmoid(b x); b = 1 is s_ilu, a large b tends to ReLU, b = 1.702 is close to GELU.

      "act:" u̲se< "Activations"
      1.702 ᵃᶜᵗs̲wish 1.0
0.8457957659328212
ˡs̲wish ← { b x → x × ˡs̲igmoid b × x }
Used in: values, smooth

ˡg̲elu : Float -> Float

function · line 106

g̲elu x: GELU (Hendrycks and Gimpel 2016) in the tanh form the frameworks compute, x/2 (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))), within 4.73e-4 of the exact g_eluErf.

      "act:" u̲se< "Activations"
      ᵃᶜᵗg̲elu -1.0 1.0
-0.1588080093917233 0.8411919906082768
ˡg̲elu ← { x → 0.5 × x × 1.0 + ʰg̲eluTanh x }

ˡg̲eluErf : Float -> Float

function · line 113

g̲eluErf x: the exact GELU, x Phi(x) = x/2 (1 + erf(x / sqrt 2)), Phi the normal CDF (erf X_eTaL's builtin).

      "act:" u̲se< "Activations"
      ᵃᶜᵗg̲eluErf 1.0
0.8413447460685429
ˡg̲eluErf ← { x → 0.5 × x × 1.0 + e̲rf x ÷ 1.4142135623730951 }
Used in: values, smooth

ˡm̲ish : Float -> Float

function · line 120

m̲ish x: Mish (Misra 2019), x tanh(softplus(x)): smooth, dips to -0.309 below 0, finite at any x.

      "act:" u̲se< "Activations"
      ᵃᶜᵗm̲ish -1.0 1.0
-0.30340146137410895 0.8650983882673103
ˡm̲ish ← { x → x × ˡt̲anh ˡs̲oftplus x }

ˡh̲ardSigmoid : Float -> Float

function · line 127

h̲ardSigmoid x: min(max(x + 3, 0), 6) / 6 (Howard et al. 2019, MobileNetV3): a sigmoid of three straight lines, no e_xp.

      "act:" u̲se< "Activations"
      ᵃᶜᵗh̲ardSigmoid -4.0 0.0 1.5
0.0 0.5 0.75
ˡh̲ardSigmoid ← { x → (6.0 m̲in 0.0 m̲ax x + 3.0) ÷ 6.0 }

ˡh̲ardSwish : Float -> Float

function · line 134

h̲ardSwish x: x h_ardSigmoid(x), the same paper's SiLU of straight lines, within 0.15 of s_ilu everywhere.

      "act:" u̲se< "Activations"
      ᵃᶜᵗh̲ardSwish -4.0 1.0 4.0
-0.0 0.6666666666666666 4.0
ˡh̲ardSwish ← { x → x × ˡh̲ardSigmoid x }
Used in: values, smooth

The gated layers

ˡg̲lu : (Float, Float) -> Float -> Float

function · line 145

(W, V) g̲lu x: GLU (Dauphin et al. 2017), sigmoid(x W) * (x V): the activation on one projection of each row of x, times the other (a gate), item by item. x is a batch (one row each), W and V both inputs by hidden.

      "act:" u̲se< "Activations"
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᶜᵗg̲lu 1 2 r̲eshape 0.0 1.0
0.0 1.4621171572600098
ˡg̲lu ← { (W, V) x → (ˡs̲igmoid x '+ '× i̲nner W) × x '+ '× i̲nner V }
Used in: ᵘg̲lu1

ˡr̲eglu : (Float, Float) -> Float -> Float

function · line 148

(W, V) r̲eglu x: relu(x W) * (x V) (Shazeer 2020).

ˡr̲eglu ← { (W, V) x → (ˡr̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
Used in: ᵘr̲eglu1

ˡg̲eglu : (Float, Float) -> Float -> Float

function · line 151

(W, V) g̲eglu x: gelu(x W) * (x V) (Shazeer 2020).

ˡg̲eglu ← { (W, V) x → (ˡg̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
Used in: ᵘg̲eglu1

ˡs̲wiglu : (Float, Float) -> Float -> Float

function · line 159

(W, V) s̲wiglu x: silu(x W) * (x V) (Shazeer 2020; LLaMA's feed-forward layer, with a hidden width of 2/3 the usual to keep the parameter count).

      "act:" u̲se< "Activations"
      (2 2 r̲eshape 1.0 0.0 0.0 1.0, 2 2 r̲eshape 2.0 0.0 0.0 2.0) ᵃᶜᵗs̲wiglu 1 2 r̲eshape 0.0 1.0
0.0 1.4621171572600098
ˡs̲wiglu ← { (W, V) x → (ˡs̲ilu x '+ '× i̲nner W) × x '+ '× i̲nner V }
Used in: ᵘs̲wiglu1

Derivatives, in terms of each function's input

ˡd̲relu : Float -> Float

function · line 167

d̲relu x: 1 where x is positive, else 0.

      "act:" u̲se< "Activations"
      ᵃᶜᵗd̲relu -1.5 0.0 2.0
0.0 0.0 1.0
ˡd̲relu ← { x → f̲loat x > 0.0 }
Used in: slopes, dRect

ˡd̲leaky : Float -> Float -> Float

function · line 170

a d̲leaky x: 1 where x is positive, else a.

ˡd̲leaky ← { a x → 1.0 + (a − 1.0) × f̲loat x ≤ 0.0 }
Used in: slopes, dRect

ˡd̲prelu : Float -> Float -> Float

function · line 173

a d̲prelu x: 1 where x is positive, else the column's slope.

ˡd̲prelu ← { a x → 1.0 + (((s̲hape x) r̲eshape a) − 1.0) × f̲loat x ≤ 0.0 }
Used in: slopes

ˡd̲elu : Float -> Float -> Float

function · line 176

a d̲elu x: 1 where x is positive, else a e^x.

ˡd̲elu ← { a x → (f̲loat x > 0.0) + a × (f̲loat x ≤ 0.0) × e̲xp x m̲in 0.0 }

ˡd̲selu : Float -> Float

function · line 179

d̲selu x: lambda where x is positive, else lambda alpha e^x.

ˡd̲selu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡd̲elu x }
Used in: slopes, dRect

ˡd̲sigmoid : Num a => a -> Float

function · line 185

d̲sigmoid x: s (1 - s) for s = sigmoid(x): at most 1/4, at 0.

      "act:" u̲se< "Activations"
      ᵃᶜᵗd̲sigmoid 0.0
0.25
ˡd̲sigmoid ← { x → s ← ˡs̲igmoid x◆ s × 1.0 − s }
Used in: slopes, dClassical

ˡd̲tanh : Float -> Float

function · line 188

d̲tanh x: 1 - tanh(x)^2.

ˡd̲tanh ← { x → t ← ˡt̲anh x◆ 1.0 − t × t }
Used in: slopes, dClassical

ˡd̲softplus : Num a => a -> Float

function · line 191

d̲softplus x: sigmoid(x).

ˡd̲softplus ← { x → ˡs̲igmoid x }
Used in: slopes, dSmooth

ˡd̲silu : Float -> Float

function · line 194

d̲silu x: s (1 + x (1 - s)) for s = sigmoid(x).

ˡd̲silu ← { x → s ← ˡs̲igmoid x◆ s × 1.0 + x × 1.0 − s }
Used in: slopes, dSmooth

ˡd̲swish : Float -> Float -> Float

function · line 197

b d̲swish x: s + b x s (1 - s) for s = sigmoid(b x).

ˡd̲swish ← { b x → s ← ˡs̲igmoid b × x◆ s + b × x × s × 1.0 − s }
Used in: slopes, dSmooth

ˡd̲gelu : Float -> Float

function · line 203

d̲gelu x: the tanh form's derivative, (1 + t)/2 + x/2 (1 - t^2) sqrt(2/pi) (1 + 3 * 0.044715 x^2), t the inner tanh.

      "act:" u̲se< "Activations"
      ᵃᶜᵗd̲gelu 0.0
0.5
ˡd̲gelu ← { x → t ← ʰg̲eluTanh x◆ (0.5 × 1.0 + t) + 0.5 × x × (1.0 − t × t) × 0.7978845608028654 × 1.0 + 0.134145 × x × x }
Used in: slopes, dSmooth

ˡd̲geluErf : Float -> Float

function · line 206

d̲geluErf x: Phi(x) + x phi(x), the normal CDF and density.

ˡd̲geluErf ← { x → (0.5 × 1.0 + e̲rf x ÷ 1.4142135623730951) + x × 0.3989422804014327 × e̲xp n̲eg x × x ÷ 2.0 }
Used in: slopes, dSmooth

ˡd̲mish : Float -> Float

function · line 209

d̲mish x: t + x (1 - t^2) sigmoid(x) for t = tanh(softplus(x)).

ˡd̲mish ← { x → t ← ˡt̲anh ˡs̲oftplus x◆ t + x × (1.0 − t × t) × ˡs̲igmoid x }
Used in: slopes, dSmooth

ˡd̲hardSigmoid : Float -> Float

function · line 212

d̲hardSigmoid x: 1/6 where |x| < 3, else 0.

ˡd̲hardSigmoid ← { x → (f̲loat (a̲bs x) < 3.0) ÷ 6.0 }
Used in: slopes, dClassical

ˡd̲hardSwish : Float -> Float

function · line 215

d̲hardSwish x: (2x + 3)/6 where |x| < 3; 0 below, 1 from 3 on.

ˡd̲hardSwish ← { x → (f̲loat x ≥ 3.0) + (f̲loat (a̲bs x) < 3.0) × ((2.0 × x) + 3.0) ÷ 6.0 }
Used in: slopes, dSmooth