sourcelibs/Optim/demos/bowl.xtl

1⍝!/usr/bin/env xetal 2⍝│ Gradient descent and Adam on a long narrow bowl, where the two 3⍝│ directions want very different step sizes. Run with "just demo-lib 4⍝│ Optim bowl". 5 6"op:" u̲se< "Optim" 7 8⍝│ The bowl: x^2 / 1000 + y^2, its bottom at 0 0; the gradient (x/500, 2y). 9ᵘg̲rad ← { w → (0.002 × 1 s̲elect w) c̲at 2.0 × 2 s̲elect w } 10ᵘh̲eight ← { w → ((1 s̲elect w) ^ 2.0) ÷ 1000.0 + (2 s̲elect w) ^ 2.0 } 11start ← 10.0 1.0 12ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 } 13⍝│ Gradient descent at 0.9 (near the most the steep direction allows): 14⍝│ after 100 steps the shallow direction has gone only from 10 to 8.35 15⍝│ (where it is, then the height). 16sg ← 100 '{ w → 0.9 ᵒᵖs̲gd (w, ᵘg̲rad w) } p̲ower start 17ᵘr̲ound sg c̲at ᵘh̲eight sg 18⍝│ Adam at 0.1 scales each direction by its own gradients' size, so the 19⍝│ shallow one moves too: to 2.24, the height 14 times lower. 20(ad, _, _, _) ← 100 '{ (w, m, v, t) → (w2, m2, v2) ← (0.1, t + 1.0) ᵒᵖa̲dam (w, m, v, ᵘg̲rad w)◆ (w2, m2, v2, t + 1.0) } p̲ower (start, 0.0 0.0, 0.0 0.0, 0.0) 21ᵘr̲ound ad c̲at ᵘh̲eight ad