librarywork/xetal-ml/libs/Optim/src/Optim.xtl
Optim: optimizers as steps: gradient descent and Adam on a weight array and its averages
Import it with an alias of your choice: "op:" u_se< "Optim". Put libs/Optim/src on XETAL_PATH ("just path"); the reference is libs/Optim/docs. Names with l: are exported; h: names are private to this file.
An optimizer is a step: it takes a weight array (of any shape), its gradient and whatever the optimizer keeps (Adam's two running averages), and gives them back moved, as a tuple. A network's training step calls it once per weight array, and p_ower repeats the whole step. Settings come first, as a tuple: (lr, t) for Adam at step t (from 1).
Gradient descent
ˡs̲gd : Num a => a -> (a, a) -> a
lr s̲gd (w, g): the weights moved against their gradient by the rate lr.
"op:" u̲se< "Optim"
0.5 ᵒᵖs̲gd (1.0 2.0, 0.2 -0.4) 0.9 2.2
ˡs̲gd ← { lr (w, g) → w − lr × g }
Adam
ˡa̲damWith : (Float, Float, Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float)
(lr, b1, b2, t) a̲damWith (w, m, v, g): Adam's step t (from 1)
with decay rates b1 and b2: the running averages of the gradients
(m) and of their squares (v) brought up to date, then the weights
moved by the first over the square root of the second, each
corrected for starting at 0 (eps 1e-8): (w, m, v) after the step.
ˡa̲damWith ← { (lr, b1, b2, t) (w, m, v, g) → m2 ← (b1 × m) + (1.0 − b1) × g v2 ← (b2 × v) + (1.0 − b2) × g × g (w − lr × (m2 ÷ 1.0 − b1 ^ t) ÷ 0.00000001 + (v2 ÷ 1.0 − b2 ^ t) ^ 0.5, m2, v2) }
ˡa̲dam : (Float, Float) -> (Float, Float, Float, Float) -> (Float, Float, Float)
(lr, t) a̲dam (w, m, v, g): Adam's step with the usual decay
rates, 0.9 and 0.999. Its first step moves every weight by the rate
against its gradient's sign, whatever the gradient's size (eps makes
it a hair less):
"op:" u̲se< "Optim"
(w, m, v) ← (0.1, 1.0) ᵒᵖa̲dam (1.0 1.0, 0.0 0.0, 0.0 0.0, 0.5 -2.0)
w
0.9000000019999999 1.0999999995
ˡa̲dam ← { (lr, t) (w, m, v, g) → m2 ← (0.9 × m) + 0.1 × g v2 ← (0.999 × v) + 0.001 × g × g (w − lr × (m2 ÷ 1.0 − 0.9 ^ t) ÷ 0.00000001 + (v2 ÷ 1.0 − 0.999 ^ t) ^ 0.5, m2, v2) }