Optim.xtl

1⍝│ Optim: optimizers as steps: gradient descent and Adam on a weight array and its averages
2⍝│
3⍝│ Import it with an alias of your choice: "op:" u_se< "Optim".
4⍝│ Put libs/Optim/src on XETAL_PATH ("just path"); the reference is libs/Optim/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ An optimizer is a step: it takes a weight array (of any shape), its
8⍝│ gradient and whatever the optimizer keeps (Adam's two running
9⍝│ averages), and gives them back moved, as a tuple. A network's
10⍝│ training step calls it once per weight array, and p_ower repeats the
11⍝│ whole step. Settings come first, as a tuple: (lr, t) for Adam at step
12⍝│ t (from 1).
13
14⍝━━ Gradient descent
15
16⍝│ lr s̲gd (w, g): the weights moved against their gradient by the rate lr.
17⍝│ >> "op:" u_se< "Optim"
18⍝│ >> 0.5 op:s_gd (1.0 2.0, 0.2 -0.4)
19⍝│ 0.9 2.2
20ˡs̲gd ← { lr (w, g) → w − lr × g }
21
22⍝━━ Adam
23
24⍝│ (lr, b1, b2, t) a̲damWith (w, m, v, g): Adam's step t (from 1)
25⍝│ with decay rates b1 and b2: the running averages of the gradients
26⍝│ (m) and of their squares (v) brought up to date, then the weights
27⍝│ moved by the first over the square root of the second, each
28⍝│ corrected for starting at 0 (eps 1e-8): (w, m, v) after the step.
29ˡa̲damWith ← { (lr, b1, b2, t) (w, m, v, g) →
30  m2 ← (b1 × m) + (1.0 − b1) × g
31  v2 ← (b2 × v) + (1.0 − b2) × g × g
32  (w − lr × (m2 ÷ 1.0 − b1 ^ t) ÷ 0.00000001 + (v2 ÷ 1.0 − b2 ^ t) ^ 0.5, m2, v2)
33}
34
35⍝│ (lr, t) a̲dam (w, m, v, g): Adam's step with the usual decay
36⍝│ rates, 0.9 and 0.999. Its first step moves every weight by the rate
37⍝│ against its gradient's sign, whatever the gradient's size (eps makes
38⍝│ it a hair less):
39⍝│ >> "op:" u_se< "Optim"
40⍝│ >> (w, m, v) := (0.1, 1.0) op:a_dam (1.0 1.0, 0.0 0.0, 0.0 0.0, 0.5 -2.0)
41⍝│ >> w
42⍝│ 0.9000000019999999 1.0999999995
43ˡa̲dam ← { (lr, t) (w, m, v, g) →
44  m2 ← (0.9 × m) + 0.1 × g
45  v2 ← (0.999 × v) + 0.001 × g × g
46  (w − lr × (m2 ÷ 1.0 − 0.9 ^ t) ÷ 0.00000001 + (v2 ÷ 1.0 − 0.999 ^ t) ^ 0.5, m2, v2)
47}