sourcelibs/Activations/demos/curves.xtl

1⍝!/usr/bin/env xetal 2⍝ Activations: every curve as a table. Thirteen points from -3 to 3 3⍝ by halves across; one row per activation, in the library's order 4⍝ (ReLU, leaky 0.1, PReLU with slopes 0.1 to 0.7 by column pairs, ELU 5⍝ alpha 1, SELU, sigmoid, tanh, softplus, SiLU, Swish 1.5, GELU, exact 6⍝ GELU, Mish, hard sigmoid, hard swish), then the same for their 7⍝ derivatives; four places. Run with "just demo Activations curves". 8"act:" u̲se< "Activations" 9xs ← -3.0 + 0.5 × f̲loat o̲ffsets 13 10a ← 0.1 0.1 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 0.6 0.6 0.7 11ᵘr̲ound ← { v → (f̲loat f̲loor 0.5 + 10000.0 × v) ÷ 10000.0 } 12values ← (ᵃᶜᵗr̲elu xs) c̲at (0.1 ᵃᶜᵗl̲eaky xs) c̲at (a ᵃᶜᵗp̲relu xs) c̲at (1.0 ᵃᶜᵗe̲lu xs) c̲at (ᵃᶜᵗs̲elu xs) c̲at (ᵃᶜᵗs̲igmoid xs) c̲at (ᵃᶜᵗt̲anh xs) c̲at (ᵃᶜᵗs̲oftplus xs) c̲at (ᵃᶜᵗs̲ilu xs) c̲at (1.5 ᵃᶜᵗs̲wish xs) c̲at (ᵃᶜᵗg̲elu xs) c̲at (ᵃᶜᵗg̲eluErf xs) c̲at (ᵃᶜᵗm̲ish xs) c̲at (ᵃᶜᵗh̲ardSigmoid xs) c̲at ᵃᶜᵗh̲ardSwish xs 13ᵘr̲ound 15 13 r̲eshape values 14slopes ← (ᵃᶜᵗd̲relu xs) c̲at (0.1 ᵃᶜᵗd̲leaky xs) c̲at (a ᵃᶜᵗd̲prelu xs) c̲at (1.0 ᵃᶜᵗd̲elu xs) c̲at (ᵃᶜᵗd̲selu xs) c̲at (ᵃᶜᵗd̲sigmoid xs) c̲at (ᵃᶜᵗd̲tanh xs) c̲at (ᵃᶜᵗd̲softplus xs) c̲at (ᵃᶜᵗd̲silu xs) c̲at (1.5 ᵃᶜᵗd̲swish xs) c̲at (ᵃᶜᵗd̲gelu xs) c̲at (ᵃᶜᵗd̲geluErf xs) c̲at (ᵃᶜᵗd̲mish xs) c̲at (ᵃᶜᵗd̲hardSigmoid xs) c̲at ᵃᶜᵗd̲hardSwish xs 15ᵘr̲ound 15 13 r̲eshape slopes 16⍝ Where each smooth one dips lowest (its minimum over -6..6), four places. 17fine ← -6.0 + 0.01 × f̲loat o̲ffsets 1201 18ᵘr̲ound ('m̲in r̲/ ᵃᶜᵗg̲elu fine) c̲at ('m̲in r̲/ ᵃᶜᵗs̲ilu fine) c̲at ('m̲in r̲/ ᵃᶜᵗm̲ish fine) c̲at 'm̲in r̲/ 1.0 ᵃᶜᵗe̲lu fine