sourcelibs/Activations/src/Activations.xtl
1⍝│ Activations: activation functions, a derivative for each, and the gated layers (GLU, ReGLU, GEGLU, SwiGLU)
2⍝│
3⍝│ Import it with an alias of your choice: "act:" u_se< "Activations".
4⍝│ Put libs/Activations/src on XETAL_PATH ("just path"); the reference is libs/Activations/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ Every activation works item by item on a Float array of any shape,
8⍝│ so a batch (one example per row) goes through in one call. A
9⍝│ setting comes first (0.1 ᵃᶜᵗl̲eaky x, 1.5 ᵃᶜᵗs̲wish x); PReLU's
10⍝│ slopes are one per column (the last axis), as a bias is. Each
11⍝│ function f has its derivative d_f written in terms of f's input
12⍝│ (ᵃᶜᵗd̲gelu z, not of its output): X_eTaL has no automatic
13⍝│ differentiation, and the smooth activations cannot be written at
14⍝│ their output as ReLU, tanh and sigmoid can. A gated layer takes its
15⍝│ two weight arrays as a tuple, (W, V) ᵃᶜᵗs̲wiglu x: the activation
16⍝│ on x W, times the linear x V (Shazeer 2020's order).
17⍝│
18⍝│ tanh and erf are X_eTaL's builtins, t̲anh and e̲rf (libm's, at
19⍝│ full accuracy: docs/xetal-asks.md M19, landed); ᵃᶜᵗt̲anh is the
20⍝│ builtin under this library's name, beside its other activations.
21
22⍝━━ Helpers (private)
23
24⍝│ The inner tanh of GELU's approximation, at the point u(x) = sqrt(2/pi) (x + 0.044715 x^3).
25ʰg̲eluTanh ← { x → ˡt̲anh 0.7978845608028654 × x + 0.044715 × x × x × x }
26
27⍝━━ Activations
28
29⍝│ r̲elu x: max(x, 0), item by item.
30⍝│ >> "act:" u_se< "Activations"
31⍝│ >> act:r_elu -1.5 0.0 2.0
32⍝│ 0.0 0.0 2.0
33ˡr̲elu ← { x → x m̲ax 0.0 }
34
35⍝│ a l̲eaky x: leaky ReLU, x where positive and a times x elsewhere.
36⍝│ >> "act:" u_se< "Activations"
37⍝│ >> 0.1 act:l_eaky -2.0 3.0
38⍝│ -0.2 3.0
39ˡl̲eaky ← { a x → (x m̲ax 0.0) + a × x m̲in 0.0 }
40
41⍝│ a p̲relu x: PReLU (He et al. 2015), a leaky ReLU whose slope a is
42⍝│ one number per column (a's length is x's last extent; a scalar is
43⍝│ l_eaky). The slopes are learned; the library only applies them.
44⍝│ >> "act:" u_se< "Activations"
45⍝│ >> 0.0 0.5 act:p_relu 2 2 r_eshape -2.0 -2.0 1.0 -4.0
46⍝│ 0.0 -1.0
47⍝│ 1.0 -2.0
48ˡp̲relu ← { a x → (x m̲ax 0.0) + ((s̲hape x) r̲eshape a) × x m̲in 0.0 }
49
50⍝│ a e̲lu x: ELU (Clevert et al. 2016), x where positive and
51⍝│ a (e^x - 1) elsewhere: smooth at 0 when a is 1, bounded below by -a.
52⍝│ >> "act:" u_se< "Activations"
53⍝│ >> 1.0 act:e_lu -1.0 2.0
54⍝│ -0.6321205588285577 2.0
55ˡe̲lu ← { a x → (x m̲ax 0.0) + a × (e̲xp x m̲in 0.0) − 1.0 }
56
57⍝│ s̲elu x: SELU (Klambauer et al. 2017), lambda times an ELU with
58⍝│ alpha, the two constants that keep a standard-normal input's mean 0
59⍝│ and variance 1 through a layer (a self-normalizing network needs no
60⍝│ norm layer).
61⍝│ >> "act:" u_se< "Activations"
62⍝│ >> act:s_elu 0.0 1.0
63⍝│ 0.0 1.0507009873554805
64ˡs̲elu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡe̲lu x }
65
66⍝│ s̲igmoid x: 1 / (1 + e^-x), item by item.
67⍝│ >> "act:" u_se< "Activations"
68⍝│ >> act:s_igmoid 0.0 2.0
69⍝│ 0.5 0.8807970779778823
70ˡs̲igmoid ← { x → 1.0 ÷ 1.0 + e̲xp n̲eg x }
71
72⍝│ t̲anh x: the hyperbolic tangent, X_eTaL's builtin (exact near 0,
73⍝│ 1 without overflow far out).
74⍝│ >> "act:" u_se< "Activations"
75⍝│ >> act:t_anh 0.0 1.0
76⍝│ 0.0 0.7615941559557649
77ˡt̲anh ← { x → t̲anh x }
78
79⍝│ s̲oftplus x: log(1 + e^x), a smooth ReLU, in the form that does
80⍝│ not overflow: max(x, 0) + log(1 + e^-|x|).
81⍝│ >> "act:" u_se< "Activations"
82⍝│ >> act:s_oftplus 0.0 1000.0
83⍝│ 0.6931471805599453 1000.0
84ˡs̲oftplus ← { x → (x m̲ax 0.0) + l̲og 1.0 + e̲xp n̲eg a̲bs x }
85
86⍝│ s̲ilu x: SiLU, x sigmoid(x) (Hendrycks and Gimpel 2016; Swish with
87⍝│ beta 1, Ramachandran et al. 2017): smooth, dips to -0.278 below 0.
88⍝│ >> "act:" u_se< "Activations"
89⍝│ >> act:s_ilu -1.0 1.0
90⍝│ -0.2689414213699951 0.7310585786300049
91ˡs̲ilu ← { x → x × ˡs̲igmoid x }
92
93⍝│ b s̲wish x: Swish, x sigmoid(b x); b = 1 is s_ilu, a large b tends
94⍝│ to ReLU, b = 1.702 is close to GELU.
95⍝│ >> "act:" u_se< "Activations"
96⍝│ >> 1.702 act:s_wish 1.0
97⍝│ 0.8457957659328212
98ˡs̲wish ← { b x → x × ˡs̲igmoid b × x }
99
100⍝│ g̲elu x: GELU (Hendrycks and Gimpel 2016) in the tanh form the
101⍝│ frameworks compute, x/2 (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))),
102⍝│ within 4.73e-4 of the exact g_eluErf.
103⍝│ >> "act:" u_se< "Activations"
104⍝│ >> act:g_elu -1.0 1.0
105⍝│ -0.1588080093917233 0.8411919906082768
106ˡg̲elu ← { x → 0.5 × x × 1.0 + ʰg̲eluTanh x }
107
108⍝│ g̲eluErf x: the exact GELU, x Phi(x) = x/2 (1 + erf(x / sqrt 2)),
109⍝│ Phi the normal CDF (erf X_eTaL's builtin).
110⍝│ >> "act:" u_se< "Activations"
111⍝│ >> act:g_eluErf 1.0
112⍝│ 0.8413447460685429
113ˡg̲eluErf ← { x → 0.5 × x × 1.0 + e̲rf x ÷ 1.4142135623730951 }
114
115⍝│ m̲ish x: Mish (Misra 2019), x tanh(softplus(x)): smooth, dips to
116⍝│ -0.309 below 0, finite at any x.
117⍝│ >> "act:" u_se< "Activations"
118⍝│ >> act:m_ish -1.0 1.0
119⍝│ -0.30340146137410895 0.8650983882673103
120ˡm̲ish ← { x → x × ˡt̲anh ˡs̲oftplus x }
121
122⍝│ h̲ardSigmoid x: min(max(x + 3, 0), 6) / 6 (Howard et al. 2019,
123⍝│ MobileNetV3): a sigmoid of three straight lines, no e_xp.
124⍝│ >> "act:" u_se< "Activations"
125⍝│ >> act:h_ardSigmoid -4.0 0.0 1.5
126⍝│ 0.0 0.5 0.75
127ˡh̲ardSigmoid ← { x → (6.0 m̲in 0.0 m̲ax x + 3.0) ÷ 6.0 }
128
129⍝│ h̲ardSwish x: x h_ardSigmoid(x), the same paper's SiLU of straight
130⍝│ lines, within 0.15 of s_ilu everywhere.
131⍝│ >> "act:" u_se< "Activations"
132⍝│ >> act:h_ardSwish -4.0 1.0 4.0
133⍝│ -0.0 0.6666666666666666 4.0
134ˡh̲ardSwish ← { x → x × ˡh̲ardSigmoid x }
135
136⍝━━ The gated layers
137
138⍝│ (W, V) g̲lu x: GLU (Dauphin et al. 2017), sigmoid(x W) * (x V):
139⍝│ the activation on one projection of each row of x, times the other
140⍝│ (a gate), item by item. x is a batch (one row each), W and V both
141⍝│ inputs by hidden.
142⍝│ >> "act:" u_se< "Activations"
143⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) act:g_lu 1 2 r_eshape 0.0 1.0
144⍝│ 0.0 1.4621171572600098
145ˡg̲lu ← { (W, V) x → (ˡs̲igmoid x '+ '× i̲nner W) × x '+ '× i̲nner V }
146
147⍝│ (W, V) r̲eglu x: relu(x W) * (x V) (Shazeer 2020).
148ˡr̲eglu ← { (W, V) x → (ˡr̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
149
150⍝│ (W, V) g̲eglu x: gelu(x W) * (x V) (Shazeer 2020).
151ˡg̲eglu ← { (W, V) x → (ˡg̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V }
152
153⍝│ (W, V) s̲wiglu x: silu(x W) * (x V) (Shazeer 2020; LLaMA's
154⍝│ feed-forward layer, with a hidden width of 2/3 the usual to keep
155⍝│ the parameter count).
156⍝│ >> "act:" u_se< "Activations"
157⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) act:s_wiglu 1 2 r_eshape 0.0 1.0
158⍝│ 0.0 1.4621171572600098
159ˡs̲wiglu ← { (W, V) x → (ˡs̲ilu x '+ '× i̲nner W) × x '+ '× i̲nner V }
160
161⍝━━ Derivatives, in terms of each function's input
162
163⍝│ d̲relu x: 1 where x is positive, else 0.
164⍝│ >> "act:" u_se< "Activations"
165⍝│ >> act:d_relu -1.5 0.0 2.0
166⍝│ 0.0 0.0 1.0
167ˡd̲relu ← { x → f̲loat x > 0.0 }
168
169⍝│ a d̲leaky x: 1 where x is positive, else a.
170ˡd̲leaky ← { a x → 1.0 + (a − 1.0) × f̲loat x ≤ 0.0 }
171
172⍝│ a d̲prelu x: 1 where x is positive, else the column's slope.
173ˡd̲prelu ← { a x → 1.0 + (((s̲hape x) r̲eshape a) − 1.0) × f̲loat x ≤ 0.0 }
174
175⍝│ a d̲elu x: 1 where x is positive, else a e^x.
176ˡd̲elu ← { a x → (f̲loat x > 0.0) + a × (f̲loat x ≤ 0.0) × e̲xp x m̲in 0.0 }
177
178⍝│ d̲selu x: lambda where x is positive, else lambda alpha e^x.
179ˡd̲selu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡd̲elu x }
180
181⍝│ d̲sigmoid x: s (1 - s) for s = sigmoid(x): at most 1/4, at 0.
182⍝│ >> "act:" u_se< "Activations"
183⍝│ >> act:d_sigmoid 0.0
184⍝│ 0.25
185ˡd̲sigmoid ← { x → s ← ˡs̲igmoid x◆ s × 1.0 − s }
186
187⍝│ d̲tanh x: 1 - tanh(x)^2.
188ˡd̲tanh ← { x → t ← ˡt̲anh x◆ 1.0 − t × t }
189
190⍝│ d̲softplus x: sigmoid(x).
191ˡd̲softplus ← { x → ˡs̲igmoid x }
192
193⍝│ d̲silu x: s (1 + x (1 - s)) for s = sigmoid(x).
194ˡd̲silu ← { x → s ← ˡs̲igmoid x◆ s × 1.0 + x × 1.0 − s }
195
196⍝│ b d̲swish x: s + b x s (1 - s) for s = sigmoid(b x).
197ˡd̲swish ← { b x → s ← ˡs̲igmoid b × x◆ s + b × x × s × 1.0 − s }
198
199⍝│ d̲gelu x: the tanh form's derivative, (1 + t)/2 + x/2 (1 - t^2) sqrt(2/pi) (1 + 3 * 0.044715 x^2), t the inner tanh.
200⍝│ >> "act:" u_se< "Activations"
201⍝│ >> act:d_gelu 0.0
202⍝│ 0.5
203ˡd̲gelu ← { x → t ← ʰg̲eluTanh x◆ (0.5 × 1.0 + t) + 0.5 × x × (1.0 − t × t) × 0.7978845608028654 × 1.0 + 0.134145 × x × x }
204
205⍝│ d̲geluErf x: Phi(x) + x phi(x), the normal CDF and density.
206ˡd̲geluErf ← { x → (0.5 × 1.0 + e̲rf x ÷ 1.4142135623730951) + x × 0.3989422804014327 × e̲xp n̲eg x × x ÷ 2.0 }
207
208⍝│ d̲mish x: t + x (1 - t^2) sigmoid(x) for t = tanh(softplus(x)).
209ˡd̲mish ← { x → t ← ˡt̲anh ˡs̲oftplus x◆ t + x × (1.0 − t × t) × ˡs̲igmoid x }
210
211⍝│ d̲hardSigmoid x: 1/6 where |x| < 3, else 0.
212ˡd̲hardSigmoid ← { x → (f̲loat (a̲bs x) < 3.0) ÷ 6.0 }
213
214⍝│ d̲hardSwish x: (2x + 3)/6 where |x| < 3; 0 below, 1 from 3 on.
215ˡd̲hardSwish ← { x → (f̲loat x ≥ 3.0) + (f̲loat (a̲bs x) < 3.0) × ((2.0 × x) + 3.0) ÷ 6.0 }