programdemos/activations/activations.xtl
Activation functions, graphed: every function of this repository's Activations library and its derivative over x from -6 to 6, two charts a family (the functions, their slopes), drawn by X_eTaL-libraries' Plot with an x axis of real values (p:f_unctionChart!). The live page adds sliders for the parameters and an editor for an activation of your own.
Data
xs : Float
The x values: 241 from -6 to 6, a twentieth apart. The parameters: leaky ReLU's slope, ELU's alpha, Swish's beta.
xs ← 241 ᵖl̲inspace (-6.0, 6.0)
ᵘc̲hart : Box Char -> Float -> Char
A family's chart: the title, the axes' names and a name per curve, then the curves, a row each over xs.
ᵘc̲hart ← { labels ys → labels ᵖf̲unctionChart! '{ i → 2 241 r̲eshape xs c̲at i s̲elect ys } m̲ap r̲ange t̲ally ys }
ᵘs̲lope : (Float -> Float) -> Float -> Float
A slope by central differences, for the gates, of which Activations gives no derivative.
ᵘs̲lope ← { f̲ x → 5000.0 × (f̲ x + 0.0001) − f̲ x − 0.0001 }
one : (Float, Float)
The gates as one-number layers, W = V = 1: x times its gate.
one ← (1 1 r̲eshape 1.0, 1 1 r̲eshape 1.0)
ᵘs̲wiglu1 : Float -> Float
ᵘs̲wiglu1 ← { x → r̲avel one ᵃᶜᵗs̲wiglu 241 1 r̲eshape x }
Classical
classical : Float
The squashing functions: sigmoid, tanh, hard sigmoid.
classical ← 3 241 r̲eshape (ᵃᶜᵗs̲igmoid xs) c̲at (ᵃᶜᵗt̲anh xs) c̲at ᵃᶜᵗh̲ardSigmoid xs
dClassical : Float
dClassical ← 3 241 r̲eshape (ᵃᶜᵗd̲sigmoid xs) c̲at (ᵃᶜᵗd̲tanh xs) c̲at ᵃᶜᵗd̲hardSigmoid xs
_ : Char
_ ← ("Slopes" "x" "f'(x)" "sigmoid" "tanh" "hard sigmoid") ᵘc̲hart dClassical
Rectifiers
rect : Float
ReLU, leaky ReLU (its slope leak), ELU (alpha), SELU.
rect ← 4 241 r̲eshape (ᵃᶜᵗr̲elu xs) c̲at (leak ᵃᶜᵗl̲eaky xs) c̲at (alpha ᵃᶜᵗe̲lu xs) c̲at ᵃᶜᵗs̲elu xs
dRect : Float
dRect ← 4 241 r̲eshape (ᵃᶜᵗd̲relu xs) c̲at (leak ᵃᶜᵗd̲leaky xs) c̲at (alpha ᵃᶜᵗd̲elu xs) c̲at ᵃᶜᵗd̲selu xs
Smooth
smooth : Float
Softplus, SiLU, Swish (beta), GELU (the tanh form and the exact one), Mish, hard swish.
smooth ← 7 241 r̲eshape (ᵃᶜᵗs̲oftplus xs) c̲at (ᵃᶜᵗs̲ilu xs) c̲at (beta ᵃᶜᵗs̲wish xs) c̲at (ᵃᶜᵗg̲elu xs) c̲at (ᵃᶜᵗg̲eluErf xs) c̲at (ᵃᶜᵗm̲ish xs) c̲at ᵃᶜᵗh̲ardSwish xs
dSmooth : Float
dSmooth ← 7 241 r̲eshape (ᵃᶜᵗd̲softplus xs) c̲at (ᵃᶜᵗd̲silu xs) c̲at (beta ᵃᶜᵗd̲swish xs) c̲at (ᵃᶜᵗd̲gelu xs) c̲at (ᵃᶜᵗd̲geluErf xs) c̲at (ᵃᶜᵗd̲mish xs) c̲at ᵃᶜᵗd̲hardSwish xs
_ : Char
_ ← ("Smooth" "x" "f(x)" "softplus" "SiLU" "Swish" "GELU" "GELU (erf)" "Mish" "hard swish") ᵘc̲hart smooth
_ : Char
_ ← ("Slopes" "x" "f'(x)" "softplus" "SiLU" "Swish" "GELU" "GELU (erf)" "Mish" "hard swish") ᵘc̲hart dSmooth
Gated
gated : Float
The gated layers on one number, W = V = 1: x times sigmoid, ReLU, GELU or SiLU of x; their slopes by central differences.
gated ← 4 241 r̲eshape (ᵘg̲lu1 xs) c̲at (ᵘr̲eglu1 xs) c̲at (ᵘg̲eglu1 xs) c̲at ᵘs̲wiglu1 xs
dGated : Float
dGated ← 4 241 r̲eshape ('ᵘg̲lu1 ᵘs̲lope xs) c̲at ('ᵘr̲eglu1 ᵘs̲lope xs) c̲at ('ᵘg̲eglu1 ᵘs̲lope xs) c̲at 'ᵘs̲wiglu1 ᵘs̲lope xs