sourcelibs/Optim/demos/bowl.xtl
1⍝!/usr/bin/env xetal
2⍝│ Gradient descent and Adam on a long narrow bowl, where the two
3⍝│ directions want very different step sizes. Run with "just demo-lib
4⍝│ Optim bowl".
5
6"op:" u̲se< "Optim"
7
8⍝│ The bowl: x^2 / 1000 + y^2, its bottom at 0 0; the gradient (x/500, 2y).
9ᵘg̲rad ← { w → (0.002 × 1 s̲elect w) c̲at 2.0 × 2 s̲elect w }
10ᵘh̲eight ← { w → ((1 s̲elect w) ^ 2.0) ÷ 1000.0 + (2 s̲elect w) ^ 2.0 }
11start ← 10.0 1.0
12ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }
13⍝│ Gradient descent at 0.9 (near the most the steep direction allows):
14⍝│ after 100 steps the shallow direction has gone only from 10 to 8.35
15⍝│ (where it is, then the height).
16sg ← 100 '{ w → 0.9 ᵒᵖs̲gd (w, ᵘg̲rad w) } p̲ower start
17ᵘr̲ound sg c̲at ᵘh̲eight sg
18⍝│ Adam at 0.1 scales each direction by its own gradients' size, so the
19⍝│ shallow one moves too: to 2.24, the height 14 times lower.
20(ad, _, _, _) ← 100 '{ (w, m, v, t) → (w2, m2, v2) ← (0.1, t + 1.0) ᵒᵖa̲dam (w, m, v, ᵘg̲rad w)◆ (w2, m2, v2, t + 1.0) } p̲ower (start, 0.0 0.0, 0.0 0.0, 0.0)
21ᵘr̲ound ad c̲at ᵘh̲eight ad