programdemos/cnn-backprop/cnn-backprop.xtl

Training a tiny convolutional network in X_eTaL, from random weights, on 600 handwritten digits: 28 x 28 -> 8 filters of 3 x 3 -> ReLU -> 2 x 2 max-pooling -> dense -> softmax, as cnn-digits reads them. The backward pass undoes each stage in turn: the dense layer, the pooling (each block's gradient to its largest item), the ReLU and the convolution, each a few array expressions over the whole batch; it is checked against finite differences. Adam moves the weights, 20 digits a step, and p_ower repeats the step.

source · imports nn: libs/NN/src/NN.xtl; act: libs/Activations/src/Activations.xtl; cv: libs/Conv/src/Conv.xtl; op: libs/Optim/src/Optim.xtl

The digits, read from data/ (just cnn-backprop-data)

tr : Float

value · line 16

600 digits to train on, 200 to test with: each a label, then 784 pixels.

tr ← 600 785 r̲eshape n̲umbers ⎕N̲GET "data/train.txt"
Used in: X, Y

te : Float

value · line 17
te ← 200 785 r̲eshape n̲umbers ⎕N̲GET "data/test.txt"
Used in: TX, ty

X : Float

value · line 19

The pictures, 0.0 to 1.0, and the labels one-hot (digit d is class d + 1).

X ← 600 28 28 r̲eshape (f̲loat 1 d̲rop₂ tr) ÷ 255.0

Y : Float

value · line 20
Y ← 10 ⁿⁿo̲neHot 1 + f̲loor r̲avel 1 t̲ake₂ tr

TX : Float

value · line 21
TX ← 200 28 28 r̲eshape (f̲loat 1 d̲rop₂ te) ÷ 255.0

ty : Int

value · line 22
ty ← 1 + f̲loor r̲avel 1 t̲ake₂ te

Forward: the network of cnn-digits, for a batch x of n digits

ᵘp̲robs : (Float, Float, Float) -> Float -> Float

function · line 30

Each digit's probabilities for 0 to 9: the Conv library's convolution (every 3 x 3 window of every digit, the 8 filters over them in one matrix product) and pooling, then ReLU's and the dense layer.

ᵘp̲robs ← { (K, B, W) x → ⁿⁿs̲oftmax (((t̲ally x) c̲at 1352) r̲eshape ᶜᵛp̲ool ᵃᶜᵗr̲elu (K, B) ᶜᵛc̲onv x) ⁿⁿd̲ense W }

Backward: each stage undone, in reverse

ᵘg̲rad : (Float, Float, Float) -> Int -> (Float, Float, Float)

function · line 34

The gradients of the loss by K, B and W, for the digits numbered i.

ᵘg̲rad ← { (K, B, W) i →
  x ← i s̲elect X
  y ← i s̲elect Y
  n ← t̲ally x
  V ← ᶜᵛw̲indows x
  C ← (K, B) ᶜᵛc̲onv x
  A ← ᵃᶜᵗr̲elu C
  Q ← ᶜᵛp̲ool A
  F ← (n c̲at 1352) r̲eshape Q
  P ← ⁿⁿs̲oftmax F ⁿⁿd̲ense W
  D3 ← (P − y) ÷ f̲loat n
  GW ← (o̲\ F c̲at₂ ((n c̲at 1) r̲eshape 1.0)) '+ '× i̲nner D3
  DQ ← (n c̲at 8 13 13) r̲eshape D3 '+ '× i̲nner o̲\ -1 d̲rop W
  DC ← (A ᶜᵛu̲npool DQ) × f̲loat C > 0.0
  (GK, GB) ← V ᶜᵛg̲rads DC
  (GK, GB, GW)
}
Used in: ᵘs̲tep, GK

Adam, on a state of (K, B, W, their averages, the step count)

lr : Float

value · line 55

The learning rate and the digits per step.

lr ← 0.01
Used in: ᵘs̲tep

bs : Int

value · line 56
bs ← 20
Used in: ᵘs̲tep

ᵘs̲tep : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float) -> (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)

function · line 59

One step, on the next 20 training digits (30 steps go through all 600): the Optim library's Adam on each of K, B and W.

ᵘs̲tep ← { (K, B, W, MK, MB, MW, VK, VB, VW, t) →
  i ← (r̲ange bs) + bs × (f̲loor t) m̲od 600 d̲iv bs
  n ← 1.0 + t
  (GK, GB, GW) ← (K, B, W) ᵘg̲rad i
  (k, mk, vk) ← (lr, n) ᵒᵖa̲dam (K, MK, VK, GK)
  (b, mb, vb) ← (lr, n) ᵒᵖa̲dam (B, MB, VB, GB)
  (w, mw, vw) ← (lr, n) ᵒᵖa̲dam (W, MW, VW, GW)
  (k, b, w, mk, mb, mw, vk, vb, vw, n)
}
Used in: s1, s2

K : Float

value · line 71

The starting state: small random filters and dense weights; the filters' biases a little above 0, so that ReLU starts off its kink (a blank window's output is then 0.01, not exactly 0).

K ← 0.8 × (f̲loat (8 9 r̲eshape r̲oll! 72 r̲eshape 2001) − 1001) ÷ 1000.0

B : Float

value · line 72
B ← 8 r̲eshape 0.01

W : Float

value · line 73
W ← 0.05 × (f̲loat (1353 10 r̲eshape r̲oll! 13530 r̲eshape 2001) − 1001) ÷ 1000.0

s0 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)

value · line 74
s0 ← (K, B, W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0)

ᵘt̲est : (Any a, Any b, Any c, Any d, Any e, Any f, Any g) => (Float, Float, Float, a, b, c, d, e, f, g) -> Float

function · line 76

The loss and the share right on the 200 test digits.

ᵘt̲est ← { (K, B, W, _, _, _, _, _, _, _) →
  p ← (K, B, W) ᵘp̲robs TX
  ((10 ⁿⁿo̲neHot ty) ⁿⁿc̲rossEntropy p) c̲at ty ⁿⁿa̲ccuracy p
}

The check: every gradient against a finite difference

ᵘl̲oss : (Float, Float, Float) -> Int -> Float

function · line 84

The loss on digits i, and e at item j of an array shaped as w.

ᵘl̲oss ← { (K, B, W) i → (i s̲elect Y) ⁿⁿc̲rossEntropy (K, B, W) ᵘp̲robs i s̲elect X }
Used in: FK, FB, FW

e : Float

value · line 85
e ← 0.00001
Used in: ᵘb̲ump, FK, FB, FW

ᵘb̲ump : a -> Int -> Float

function · line 86
ᵘb̲ump ← { w j → e × f̲loat (s̲hape w) r̲eshape j = r̲ange t̲ally r̲avel w }
Used in: FK, FB, FW

i4 : Int

value · line 88

On four digits: the first filter's 9 weights, the 8 biases, 6 dense weights.

i4 ← 1 2 3 4
Used in: GK, FK, FB, FW

GK : Float

value · line 89
(GK, GB, GW) ← (K, B, W) ᵘg̲rad i4

GB : Float

value · line 89
(GK, GB, GW) ← (K, B, W) ᵘg̲rad i4

GW : Float

value · line 89
(GK, GB, GW) ← (K, B, W) ᵘg̲rad i4
Used in: ᵘs̲tep, GK, gap

FK : Float

value · line 90
FK ← '{ j → (((K + K ᵘb̲ump j, B, W) ᵘl̲oss i4) − (K − K ᵘb̲ump j, B, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 9
Used in: gap

FB : Float

value · line 91
FB ← '{ j → (((K, B + B ᵘb̲ump j, W) ᵘl̲oss i4) − (K, B − B ᵘb̲ump j, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 8
Used in: gap

wj : Int

value · line 92
wj ← 1 77 500 6000 13521 13530
Used in: FW, gap

FW : Float

value · line 93
FW ← '{ j → (((K, B, W + W ᵘb̲ump j) ᵘl̲oss i4) − (K, B, W − W ᵘb̲ump j) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach wj
Used in: gap

gap : Float

value · line 94
gap ← ('m̲ax r̲/ a̲bs (9 t̲ake r̲avel GK) − FK) m̲ax ('m̲ax r̲/ a̲bs GB − FB) m̲ax 'm̲ax r̲/ a̲bs (wj s̲elect r̲avel GW) − FW

Training

ᵘr̲ound : Float -> Float

function · line 101

The test loss and share right: at the start, after 10 steps (200 digits), after 40 (800: each training digit seen about once and a third).

ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }

s1 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)

value · line 102
s1 ← 10 'ᵘs̲tep p̲ower s0

s2 : (Float, Float, Float, Float, Float, Float, Float, Float, Float, Float)

value · line 103
s2 ← 30 'ᵘs̲tep p̲ower s1

K2 : Float

value · line 106

The first 20 test digits as the trained network reads them, and their labels.

(K2, B2, W2, _, _, _, _, _, _, _) ← s2

B2 : Float

value · line 106

The first 20 test digits as the trained network reads them, and their labels.

(K2, B2, W2, _, _, _, _, _, _, _) ← s2

W2 : Float

value · line 106

The first 20 test digits as the trained network reads them, and their labels.

(K2, B2, W2, _, _, _, _, _, _, _) ← s2