programlibs/Optim/demos/bowl.xtl
Gradient descent and Adam on a long narrow bowl, where the two directions want very different step sizes. Run with "just demo-lib Optim bowl".
ᵘg̲rad : Float -> Float
The bowl: x^2 / 1000 + y^2, its bottom at 0 0; the gradient (x/500, 2y).
ᵘg̲rad ← { w → (0.002 × 1 s̲elect w) c̲at 2.0 × 2 s̲elect w }
sg : Float
Gradient descent at 0.9 (near the most the steep direction allows): after 100 steps the shallow direction has gone only from 10 to 8.35 (where it is, then the height).
sg ← 100 '{ w → 0.9 ᵒᵖs̲gd (w, ᵘg̲rad w) } p̲ower start
Used in: libs/Optim/demos/bowl.xtl:17
ad : Float
Adam at 0.1 scales each direction by its own gradients' size, so the shallow one moves too: to 2.24, the height 14 times lower.
(ad, _, _, _) ← 100 '{ (w, m, v, t) → (w2, m2, v2) ← (0.1, t + 1.0) ᵒᵖa̲dam (w, m, v, ᵘg̲rad w)◆ (w2, m2, v2, t + 1.0) } p̲ower (start, 0.0 0.0, 0.0 0.0, 0.0)
Used in: ad, libs/Optim/demos/bowl.xtl:21