sourcelibs/Activations/src/Activations.xtl

1⍝│ Activations: activation functions, a derivative for each, and the gated layers (GLU, ReGLU, GEGLU, SwiGLU) 2⍝│ 3⍝│ Import it with an alias of your choice: "act:" u_se< "Activations". 4⍝│ Put libs/Activations/src on XETAL_PATH ("just path"); the reference is libs/Activations/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ Every activation works item by item on a Float array of any shape, 8⍝│ so a batch (one example per row) goes through in one call. A 9⍝│ setting comes first (0.1 ᵃᶜᵗl̲eaky x, 1.5 ᵃᶜᵗs̲wish x); PReLU's 10⍝│ slopes are one per column (the last axis), as a bias is. Each 11⍝│ function f has its derivative d_f written in terms of f's input 12⍝│ (ᵃᶜᵗd̲gelu z, not of its output): X_eTaL has no automatic 13⍝│ differentiation, and the smooth activations cannot be written at 14⍝│ their output as ReLU, tanh and sigmoid can. A gated layer takes its 15⍝│ two weight arrays as a tuple, (W, V) ᵃᶜᵗs̲wiglu x: the activation 16⍝│ on x W, times the linear x V (Shazeer 2020's order). 17⍝│ 18⍝│ tanh and erf are X_eTaL's builtins, t̲anh and e̲rf (libm's, at 19⍝│ full accuracy: docs/xetal-asks.md M19, landed); ᵃᶜᵗt̲anh is the 20⍝│ builtin under this library's name, beside its other activations. 21 22⍝━━ Helpers (private) 23 24⍝│ The inner tanh of GELU's approximation, at the point u(x) = sqrt(2/pi) (x + 0.044715 x^3). 25ʰg̲eluTanh ← { x → ˡt̲anh 0.7978845608028654 × x + 0.044715 × x × x × x } 26 27⍝━━ Activations 28 29⍝│ r̲elu x: max(x, 0), item by item. 30⍝│ >> "act:" u_se< "Activations" 31⍝│ >> act:r_elu -1.5 0.0 2.0 32⍝│ 0.0 0.0 2.0 33ˡr̲elu ← { x → x m̲ax 0.0 } 34 35⍝│ a l̲eaky x: leaky ReLU, x where positive and a times x elsewhere. 36⍝│ >> "act:" u_se< "Activations" 37⍝│ >> 0.1 act:l_eaky -2.0 3.0 38⍝│ -0.2 3.0 39ˡl̲eaky ← { a x → (x m̲ax 0.0) + a × x m̲in 0.0 } 40 41⍝│ a p̲relu x: PReLU (He et al. 2015), a leaky ReLU whose slope a is 42⍝│ one number per column (a's length is x's last extent; a scalar is 43⍝│ l_eaky). The slopes are learned; the library only applies them. 44⍝│ >> "act:" u_se< "Activations" 45⍝│ >> 0.0 0.5 act:p_relu 2 2 r_eshape -2.0 -2.0 1.0 -4.0 46⍝│ 0.0 -1.0 47⍝│ 1.0 -2.0 48ˡp̲relu ← { a x → (x m̲ax 0.0) + ((s̲hape x) r̲eshape a) × x m̲in 0.0 } 49 50⍝│ a e̲lu x: ELU (Clevert et al. 2016), x where positive and 51⍝│ a (e^x - 1) elsewhere: smooth at 0 when a is 1, bounded below by -a. 52⍝│ >> "act:" u_se< "Activations" 53⍝│ >> 1.0 act:e_lu -1.0 2.0 54⍝│ -0.6321205588285577 2.0 55ˡe̲lu ← { a x → (x m̲ax 0.0) + a × (e̲xp x m̲in 0.0) − 1.0 } 56 57⍝│ s̲elu x: SELU (Klambauer et al. 2017), lambda times an ELU with 58⍝│ alpha, the two constants that keep a standard-normal input's mean 0 59⍝│ and variance 1 through a layer (a self-normalizing network needs no 60⍝│ norm layer). 61⍝│ >> "act:" u_se< "Activations" 62⍝│ >> act:s_elu 0.0 1.0 63⍝│ 0.0 1.0507009873554805 64ˡs̲elu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡe̲lu x } 65 66⍝│ s̲igmoid x: 1 / (1 + e^-x), item by item. 67⍝│ >> "act:" u_se< "Activations" 68⍝│ >> act:s_igmoid 0.0 2.0 69⍝│ 0.5 0.8807970779778823 70ˡs̲igmoid ← { x → 1.0 ÷ 1.0 + e̲xp n̲eg x } 71 72⍝│ t̲anh x: the hyperbolic tangent, X_eTaL's builtin (exact near 0, 73⍝│ 1 without overflow far out). 74⍝│ >> "act:" u_se< "Activations" 75⍝│ >> act:t_anh 0.0 1.0 76⍝│ 0.0 0.7615941559557649 77ˡt̲anh ← { x → t̲anh x } 78 79⍝│ s̲oftplus x: log(1 + e^x), a smooth ReLU, in the form that does 80⍝│ not overflow: max(x, 0) + log(1 + e^-|x|). 81⍝│ >> "act:" u_se< "Activations" 82⍝│ >> act:s_oftplus 0.0 1000.0 83⍝│ 0.6931471805599453 1000.0 84ˡs̲oftplus ← { x → (x m̲ax 0.0) + l̲og 1.0 + e̲xp n̲eg a̲bs x } 85 86⍝│ s̲ilu x: SiLU, x sigmoid(x) (Hendrycks and Gimpel 2016; Swish with 87⍝│ beta 1, Ramachandran et al. 2017): smooth, dips to -0.278 below 0. 88⍝│ >> "act:" u_se< "Activations" 89⍝│ >> act:s_ilu -1.0 1.0 90⍝│ -0.2689414213699951 0.7310585786300049 91ˡs̲ilu ← { x → x × ˡs̲igmoid x } 92 93⍝│ b s̲wish x: Swish, x sigmoid(b x); b = 1 is s_ilu, a large b tends 94⍝│ to ReLU, b = 1.702 is close to GELU. 95⍝│ >> "act:" u_se< "Activations" 96⍝│ >> 1.702 act:s_wish 1.0 97⍝│ 0.8457957659328212 98ˡs̲wish ← { b x → x × ˡs̲igmoid b × x } 99 100⍝│ g̲elu x: GELU (Hendrycks and Gimpel 2016) in the tanh form the 101⍝│ frameworks compute, x/2 (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))), 102⍝│ within 4.73e-4 of the exact g_eluErf. 103⍝│ >> "act:" u_se< "Activations" 104⍝│ >> act:g_elu -1.0 1.0 105⍝│ -0.1588080093917233 0.8411919906082768 106ˡg̲elu ← { x → 0.5 × x × 1.0 + ʰg̲eluTanh x } 107 108⍝│ g̲eluErf x: the exact GELU, x Phi(x) = x/2 (1 + erf(x / sqrt 2)), 109⍝│ Phi the normal CDF (erf X_eTaL's builtin). 110⍝│ >> "act:" u_se< "Activations" 111⍝│ >> act:g_eluErf 1.0 112⍝│ 0.8413447460685429 113ˡg̲eluErf ← { x → 0.5 × x × 1.0 + e̲rf x ÷ 1.4142135623730951 } 114 115⍝│ m̲ish x: Mish (Misra 2019), x tanh(softplus(x)): smooth, dips to 116⍝│ -0.309 below 0, finite at any x. 117⍝│ >> "act:" u_se< "Activations" 118⍝│ >> act:m_ish -1.0 1.0 119⍝│ -0.30340146137410895 0.8650983882673103 120ˡm̲ish ← { x → x × ˡt̲anh ˡs̲oftplus x } 121 122⍝│ h̲ardSigmoid x: min(max(x + 3, 0), 6) / 6 (Howard et al. 2019, 123⍝│ MobileNetV3): a sigmoid of three straight lines, no e_xp. 124⍝│ >> "act:" u_se< "Activations" 125⍝│ >> act:h_ardSigmoid -4.0 0.0 1.5 126⍝│ 0.0 0.5 0.75 127ˡh̲ardSigmoid ← { x → (6.0 m̲in 0.0 m̲ax x + 3.0) ÷ 6.0 } 128 129⍝│ h̲ardSwish x: x h_ardSigmoid(x), the same paper's SiLU of straight 130⍝│ lines, within 0.15 of s_ilu everywhere. 131⍝│ >> "act:" u_se< "Activations" 132⍝│ >> act:h_ardSwish -4.0 1.0 4.0 133⍝│ -0.0 0.6666666666666666 4.0 134ˡh̲ardSwish ← { x → x × ˡh̲ardSigmoid x } 135 136⍝━━ The gated layers 137 138⍝│ (W, V) g̲lu x: GLU (Dauphin et al. 2017), sigmoid(x W) * (x V): 139⍝│ the activation on one projection of each row of x, times the other 140⍝│ (a gate), item by item. x is a batch (one row each), W and V both 141⍝│ inputs by hidden. 142⍝│ >> "act:" u_se< "Activations" 143⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) act:g_lu 1 2 r_eshape 0.0 1.0 144⍝│ 0.0 1.4621171572600098 145ˡg̲lu ← { (W, V) x → (ˡs̲igmoid x '+ '× i̲nner W) × x '+ '× i̲nner V } 146 147⍝│ (W, V) r̲eglu x: relu(x W) * (x V) (Shazeer 2020). 148ˡr̲eglu ← { (W, V) x → (ˡr̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V } 149 150⍝│ (W, V) g̲eglu x: gelu(x W) * (x V) (Shazeer 2020). 151ˡg̲eglu ← { (W, V) x → (ˡg̲elu x '+ '× i̲nner W) × x '+ '× i̲nner V } 152 153⍝│ (W, V) s̲wiglu x: silu(x W) * (x V) (Shazeer 2020; LLaMA's 154⍝│ feed-forward layer, with a hidden width of 2/3 the usual to keep 155⍝│ the parameter count). 156⍝│ >> "act:" u_se< "Activations" 157⍝│ >> (2 2 r_eshape 1.0 0.0 0.0 1.0, 2 2 r_eshape 2.0 0.0 0.0 2.0) act:s_wiglu 1 2 r_eshape 0.0 1.0 158⍝│ 0.0 1.4621171572600098 159ˡs̲wiglu ← { (W, V) x → (ˡs̲ilu x '+ '× i̲nner W) × x '+ '× i̲nner V } 160 161⍝━━ Derivatives, in terms of each function's input 162 163⍝│ d̲relu x: 1 where x is positive, else 0. 164⍝│ >> "act:" u_se< "Activations" 165⍝│ >> act:d_relu -1.5 0.0 2.0 166⍝│ 0.0 0.0 1.0 167ˡd̲relu ← { x → f̲loat x > 0.0 } 168 169⍝│ a d̲leaky x: 1 where x is positive, else a. 170ˡd̲leaky ← { a x → 1.0 + (a − 1.0) × f̲loat x ≤ 0.0 } 171 172⍝│ a d̲prelu x: 1 where x is positive, else the column's slope. 173ˡd̲prelu ← { a x → 1.0 + (((s̲hape x) r̲eshape a) − 1.0) × f̲loat x ≤ 0.0 } 174 175⍝│ a d̲elu x: 1 where x is positive, else a e^x. 176ˡd̲elu ← { a x → (f̲loat x > 0.0) + a × (f̲loat x ≤ 0.0) × e̲xp x m̲in 0.0 } 177 178⍝│ d̲selu x: lambda where x is positive, else lambda alpha e^x. 179ˡd̲selu ← { x → 1.0507009873554805 × 1.6732632423543772 ˡd̲elu x } 180 181⍝│ d̲sigmoid x: s (1 - s) for s = sigmoid(x): at most 1/4, at 0. 182⍝│ >> "act:" u_se< "Activations" 183⍝│ >> act:d_sigmoid 0.0 184⍝│ 0.25 185ˡd̲sigmoid ← { x → s ← ˡs̲igmoid x◆ s × 1.0 − s } 186 187⍝│ d̲tanh x: 1 - tanh(x)^2. 188ˡd̲tanh ← { x → t ← ˡt̲anh x◆ 1.0 − t × t } 189 190⍝│ d̲softplus x: sigmoid(x). 191ˡd̲softplus ← { x → ˡs̲igmoid x } 192 193⍝│ d̲silu x: s (1 + x (1 - s)) for s = sigmoid(x). 194ˡd̲silu ← { x → s ← ˡs̲igmoid x◆ s × 1.0 + x × 1.0 − s } 195 196⍝│ b d̲swish x: s + b x s (1 - s) for s = sigmoid(b x). 197ˡd̲swish ← { b x → s ← ˡs̲igmoid b × x◆ s + b × x × s × 1.0 − s } 198 199⍝│ d̲gelu x: the tanh form's derivative, (1 + t)/2 + x/2 (1 - t^2) sqrt(2/pi) (1 + 3 * 0.044715 x^2), t the inner tanh. 200⍝│ >> "act:" u_se< "Activations" 201⍝│ >> act:d_gelu 0.0 202⍝│ 0.5 203ˡd̲gelu ← { x → t ← ʰg̲eluTanh x◆ (0.5 × 1.0 + t) + 0.5 × x × (1.0 − t × t) × 0.7978845608028654 × 1.0 + 0.134145 × x × x } 204 205⍝│ d̲geluErf x: Phi(x) + x phi(x), the normal CDF and density. 206ˡd̲geluErf ← { x → (0.5 × 1.0 + e̲rf x ÷ 1.4142135623730951) + x × 0.3989422804014327 × e̲xp n̲eg x × x ÷ 2.0 } 207 208⍝│ d̲mish x: t + x (1 - t^2) sigmoid(x) for t = tanh(softplus(x)). 209ˡd̲mish ← { x → t ← ˡt̲anh ˡs̲oftplus x◆ t + x × (1.0 − t × t) × ˡs̲igmoid x } 210 211⍝│ d̲hardSigmoid x: 1/6 where |x| < 3, else 0. 212ˡd̲hardSigmoid ← { x → (f̲loat (a̲bs x) < 3.0) ÷ 6.0 } 213 214⍝│ d̲hardSwish x: (2x + 3)/6 where |x| < 3; 0 below, 1 from 3 on. 215ˡd̲hardSwish ← { x → (f̲loat x ≥ 3.0) + (f̲loat (a̲bs x) < 3.0) × ((2.0 × x) + 3.0) ÷ 6.0 }