programdemos/activations/activations.xtl

Activation functions, graphed: every function of this repository's Activations library and its derivative over x from -6 to 6, two charts a family (the functions, their slopes), drawn by X_eTaL-libraries' Plot with an x axis of real values (p:f_unctionChart!). The live page adds sliders for the parameters and an editor for an activation of your own.

source · imports act: libs/Activations/src/Activations.xtl; p: work/libs/Plot/src/Plot.xtl

Data

xs : Float

value · line 16

The x values: 241 from -6 to 6, a twentieth apart. The parameters: leaky ReLU's slope, ELU's alpha, Swish's beta.

xs ← 241 ᵖl̲inspace (-6.0, 6.0)

leak : Float

value · line 17
leak ← 0.1
Used in: rect, dRect

alpha : Float

value · line 18
alpha ← 1.0
Used in: rect, dRect

beta : Float

value · line 19
beta ← 1.5
Used in: smooth, dSmooth

ᵘc̲hart : Box Char -> Float -> Char

function · line 23

A family's chart: the title, the axes' names and a name per curve, then the curves, a row each over xs.

ᵘc̲hart ← { labels ys → labels ᵖf̲unctionChart! '{ i → 2 241 r̲eshape xs c̲at i s̲elect ys } m̲ap r̲ange t̲ally ys }
Used in: _

ᵘs̲lope : (Float -> Float) -> Float -> Float

function · line 26

A slope by central differences, for the gates, of which Activations gives no derivative.

ᵘs̲lope ← { f̲ x → 5000.0 × (f̲ x + 0.0001) − f̲ x − 0.0001 }
Used in: dGated

one : (Float, Float)

value · line 28

The gates as one-number layers, W = V = 1: x times its gate.

one ← (1 1 r̲eshape 1.0, 1 1 r̲eshape 1.0)

ᵘg̲lu1 : Float -> Float

function · line 29
ᵘg̲lu1 ← { x → r̲avel one ᵃᶜᵗg̲lu 241 1 r̲eshape x }
Used in: gated, dGated

ᵘr̲eglu1 : Float -> Float

function · line 30
ᵘr̲eglu1 ← { x → r̲avel one ᵃᶜᵗr̲eglu 241 1 r̲eshape x }
Used in: gated, dGated

ᵘg̲eglu1 : Float -> Float

function · line 31
ᵘg̲eglu1 ← { x → r̲avel one ᵃᶜᵗg̲eglu 241 1 r̲eshape x }
Used in: gated, dGated

ᵘs̲wiglu1 : Float -> Float

function · line 32
ᵘs̲wiglu1 ← { x → r̲avel one ᵃᶜᵗs̲wiglu 241 1 r̲eshape x }
Used in: gated, dGated

Classical

classical : Float

value · line 37

The squashing functions: sigmoid, tanh, hard sigmoid.

classical ← 3 241 r̲eshape (ᵃᶜᵗs̲igmoid xs) c̲at (ᵃᶜᵗt̲anh xs) c̲at ᵃᶜᵗh̲ardSigmoid xs

dClassical : Float

value · line 38
dClassical ← 3 241 r̲eshape (ᵃᶜᵗd̲sigmoid xs) c̲at (ᵃᶜᵗd̲tanh xs) c̲at ᵃᶜᵗd̲hardSigmoid xs

_ : Char

function · line 39
_ ← ("Classical" "x" "f(x)" "sigmoid" "tanh" "hard sigmoid") ᵘc̲hart classical

_ : Char

function · line 40
_ ← ("Slopes" "x" "f'(x)" "sigmoid" "tanh" "hard sigmoid") ᵘc̲hart dClassical

Rectifiers

rect : Float

value · line 45

ReLU, leaky ReLU (its slope leak), ELU (alpha), SELU.

rect ← 4 241 r̲eshape (ᵃᶜᵗr̲elu xs) c̲at (leak ᵃᶜᵗl̲eaky xs) c̲at (alpha ᵃᶜᵗe̲lu xs) c̲at ᵃᶜᵗs̲elu xs

dRect : Float

value · line 46
dRect ← 4 241 r̲eshape (ᵃᶜᵗd̲relu xs) c̲at (leak ᵃᶜᵗd̲leaky xs) c̲at (alpha ᵃᶜᵗd̲elu xs) c̲at ᵃᶜᵗd̲selu xs

_ : Char

function · line 47
_ ← ("Rectifiers" "x" "f(x)" "ReLU" "leaky ReLU" "ELU" "SELU") ᵘc̲hart rect

_ : Char

function · line 48
_ ← ("Slopes" "x" "f'(x)" "ReLU" "leaky ReLU" "ELU" "SELU") ᵘc̲hart dRect

Smooth

smooth : Float

value · line 54

Softplus, SiLU, Swish (beta), GELU (the tanh form and the exact one), Mish, hard swish.

smooth ← 7 241 r̲eshape (ᵃᶜᵗs̲oftplus xs) c̲at (ᵃᶜᵗs̲ilu xs) c̲at (beta ᵃᶜᵗs̲wish xs) c̲at (ᵃᶜᵗg̲elu xs) c̲at (ᵃᶜᵗg̲eluErf xs) c̲at (ᵃᶜᵗm̲ish xs) c̲at ᵃᶜᵗh̲ardSwish xs

dSmooth : Float

value · line 55
dSmooth ← 7 241 r̲eshape (ᵃᶜᵗd̲softplus xs) c̲at (ᵃᶜᵗd̲silu xs) c̲at (beta ᵃᶜᵗd̲swish xs) c̲at (ᵃᶜᵗd̲gelu xs) c̲at (ᵃᶜᵗd̲geluErf xs) c̲at (ᵃᶜᵗd̲mish xs) c̲at ᵃᶜᵗd̲hardSwish xs

_ : Char

function · line 56
_ ← ("Smooth" "x" "f(x)" "softplus" "SiLU" "Swish" "GELU" "GELU (erf)" "Mish" "hard swish") ᵘc̲hart smooth

_ : Char

function · line 57
_ ← ("Slopes" "x" "f'(x)" "softplus" "SiLU" "Swish" "GELU" "GELU (erf)" "Mish" "hard swish") ᵘc̲hart dSmooth

Gated

gated : Float

value · line 63

The gated layers on one number, W = V = 1: x times sigmoid, ReLU, GELU or SiLU of x; their slopes by central differences.

gated ← 4 241 r̲eshape (ᵘg̲lu1 xs) c̲at (ᵘr̲eglu1 xs) c̲at (ᵘg̲eglu1 xs) c̲at ᵘs̲wiglu1 xs

dGated : Float

value · line 64
dGated ← 4 241 r̲eshape ('ᵘg̲lu1 ᵘs̲lope xs) c̲at ('ᵘr̲eglu1 ᵘs̲lope xs) c̲at ('ᵘg̲eglu1 ᵘs̲lope xs) c̲at 'ᵘs̲wiglu1 ᵘs̲lope xs

_ : Char

function · line 65
_ ← ("Gated (W = V = 1)" "x" "f(x)" "GLU" "ReGLU" "GEGLU" "SwiGLU") ᵘc̲hart gated

_ : Char

function · line 66
_ ← ("Slopes (by differences)" "x" "f'(x)" "GLU" "ReGLU" "GEGLU" "SwiGLU") ᵘc̲hart dGated

The ranges

ᵘs̲pan : Num a => a -> a

function · line 72

Each family's least and greatest values over -6..6, a row a family (classical, rectifiers, smooth, gated); then its slopes'.

ᵘs̲pan ← { f → ('m̲in r̲/ r̲avel f) c̲at 'm̲ax r̲/ r̲avel f }