programlibs/Optim/demos/bowl.xtl

Gradient descent and Adam on a long narrow bowl, where the two directions want very different step sizes. Run with "just demo-lib Optim bowl".

source · imports op: libs/Optim/src/Optim.xtl

ᵘg̲rad : Float -> Float

function · line 9

The bowl: x^2 / 1000 + y^2, its bottom at 0 0; the gradient (x/500, 2y).

ᵘg̲rad ← { w → (0.002 × 1 s̲elect w) c̲at 2.0 × 2 s̲elect w }
Used in: sg, ad

ᵘh̲eight : Float -> Float

function · line 10
ᵘh̲eight ← { w → ((1 s̲elect w) ^ 2.0) ÷ 1000.0 + (2 s̲elect w) ^ 2.0 }

start : Float

value · line 11
start ← 10.0 1.0
Used in: sg, ad

ᵘr̲ound : Float -> Float

function · line 12
ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }

sg : Float

value · line 16

Gradient descent at 0.9 (near the most the steep direction allows): after 100 steps the shallow direction has gone only from 10 to 8.35 (where it is, then the height).

sg ← 100 '{ w → 0.9 ᵒᵖs̲gd (w, ᵘg̲rad w) } p̲ower start