programdemos/cnn-backprop/cnn-backprop.xtl
Training a tiny convolutional network in X_eTaL, from random weights, on 600 handwritten digits: 28 x 28 -> 8 filters of 3 x 3 -> ReLU -> 2 x 2 max-pooling -> dense -> softmax, as cnn-digits reads them. The backward pass undoes each stage in turn: the dense layer, the pooling (each block's gradient to its largest item), the ReLU and the convolution, each a few array expressions over the whole batch; it is checked against finite differences. Adam moves the weights, 20 digits a step, and p_ower repeats the step.
The digits, read from data/ (just cnn-backprop-data)
tr : Float
600 digits to train on, 200 to test with: each a label, then 784 pixels.
tr ← 600 785 r̲eshape n̲umbers ⎕N̲GET "data/train.txt"
X : Float
The pictures, 0.0 to 1.0, and the labels one-hot (digit d is class d + 1).
X ← 600 28 28 r̲eshape (f̲loat 1 d̲rop₂ tr) ÷ 255.0
Forward: the network of cnn-digits, for a batch x of n digits
ᵘp̲robs : (Float, Float, Float) -> Float -> Float
Each digit's probabilities for 0 to 9: the Conv library's convolution (every 3 x 3 window of every digit, the 8 filters over them in one matrix product) and pooling, then ReLU's and the dense layer.
ᵘp̲robs ← { (K, B, W) x → ⁿⁿs̲oftmax (((t̲ally x) c̲at 1352) r̲eshape ᶜᵛp̲ool ᵃᶜᵗr̲elu (K, B) ᶜᵛc̲onv x) ⁿⁿd̲ense W }
Backward: each stage undone, in reverse
ᵘg̲rad : (Float, Float, Float) -> Int -> (Float, Float, Float)
The gradients of the loss by K, B and W, for the digits numbered i.
ᵘg̲rad ← { (K, B, W) i → x ← i s̲elect X y ← i s̲elect Y n ← t̲ally x V ← ᶜᵛw̲indows x C ← (K, B) ᶜᵛc̲onv x A ← ᵃᶜᵗr̲elu C Q ← ᶜᵛp̲ool A F ← (n c̲at 1352) r̲eshape Q P ← ⁿⁿs̲oftmax F ⁿⁿd̲ense W D3 ← (P − y) ÷ f̲loat n GW ← (o̲\ F c̲at₂ ((n c̲at 1) r̲eshape 1.0)) '+ '× i̲nner D3 DQ ← (n c̲at 8 13 13) r̲eshape D3 '+ '× i̲nner o̲\ -1 d̲rop W DC ← (A ᶜᵛu̲npool DQ) × f̲loat C > 0.0 (GK, GB) ← V ᶜᵛg̲rads DC (GK, GB, GW) }
Adam, on a state of (K, B, W, their averages, the step count)
ᵘs̲tep : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)
One step, on the next 20 training digits (30 steps go through all 600): the Optim library's Adam on each of K, B and W.
ᵘs̲tep ← { (K, B, W, MK, MB, MW, VK, VB, VW, t) → i ← (r̲ange bs) + bs × (f̲loor t) m̲od 600 d̲iv bs n ← 1.0 + t (GK, GB, GW) ← (K, B, W) ᵘg̲rad i (k, mk, vk) ← (lr, n) ᵒᵖa̲dam (K, MK, VK, GK) (b, mb, vb) ← (lr, n) ᵒᵖa̲dam (B, MB, VB, GB) (w, mw, vw) ← (lr, n) ᵒᵖa̲dam (W, MW, VW, GW) (k, b, w, mk, mb, mw, vk, vb, vw, n) }
K : Float
The starting state: small random filters and dense weights; the filters' biases a little above 0, so that ReLU starts off its kink (a blank window's output is then 0.01, not exactly 0).
K ← 0.8 × (f̲loat (8 9 r̲eshape r̲oll! 72 r̲eshape 2001) − 1001) ÷ 1000.0
s0 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)
s0 ← (K, B, W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0)
ᵘt̲est : (Any a, Any b, Any c, Any d, Any e, Any f, Any g) => (Float, Float, Float, a, b, c, d, e, f, g) -> Float
The loss and the share right on the 200 test digits.
ᵘt̲est ← { (K, B, W, _, _, _, _, _, _, _) → p ← (K, B, W) ᵘp̲robs TX ((10 ⁿⁿo̲neHot ty) ⁿⁿc̲rossEntropy p) c̲at ty ⁿⁿa̲ccuracy p }
The check: every gradient against a finite difference
ᵘl̲oss : (Float, Float, Float) -> Int -> Float
The loss on digits i, and e at item j of an array shaped as w.
ᵘl̲oss ← { (K, B, W) i → (i s̲elect Y) ⁿⁿc̲rossEntropy (K, B, W) ᵘp̲robs i s̲elect X }
ᵘb̲ump : a -> Int -> Float
ᵘb̲ump ← { w j → e × f̲loat (s̲hape w) r̲eshape j = r̲ange t̲ally r̲avel w }
FK : Float
FK ← '{ j → (((K + K ᵘb̲ump j, B, W) ᵘl̲oss i4) − (K − K ᵘb̲ump j, B, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 9
FB : Float
FB ← '{ j → (((K, B + B ᵘb̲ump j, W) ᵘl̲oss i4) − (K, B − B ᵘb̲ump j, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 8
FW : Float
FW ← '{ j → (((K, B, W + W ᵘb̲ump j) ᵘl̲oss i4) − (K, B, W − W ᵘb̲ump j) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach wj
gap : Float
gap ← ('m̲ax r̲/ a̲bs (9 t̲ake r̲avel GK) − FK) m̲ax ('m̲ax r̲/ a̲bs GB − FB) m̲ax 'm̲ax r̲/ a̲bs (wj s̲elect r̲avel GW) − FW
Training
ᵘr̲ound : Float -> Float
The test loss and share right: at the start, after 10 steps (200 digits), after 40 (800: each training digit seen about once and a third).
ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }
s1 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)
s1 ← 10 'ᵘs̲tep p̲ower s0
s2 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)
s2 ← 30 'ᵘs̲tep p̲ower s1
K2 : Float
The first 20 test digits as the trained network reads them, and their labels.
(K2, B2, W2, _, _, _, _, _, _, _) ← s2