sourcedemos/cnn-backprop/cnn-backprop.xtl
1⍝!/usr/bin/env xetal
2⍝│ Training a tiny convolutional network in X_eTaL, from random
3⍝│ weights, on 600 handwritten digits: 28 x 28 -> 8 filters of 3 x 3
4⍝│ -> ReLU -> 2 x 2 max-pooling -> dense -> softmax, as cnn-digits
5⍝│ reads them. The backward pass undoes each stage in turn: the dense
6⍝│ layer, the pooling (each block's gradient to its largest item), the
7⍝│ ReLU and the convolution, each a few array expressions over the
8⍝│ whole batch; it is checked against finite differences. Adam moves
9⍝│ the weights, 20 digits a step, and p_ower repeats the step.
10
11"nn:" u̲se< "NN"
12"act:" u̲se< "Activations"
13
14⍝━━ The digits, read from data/ (just cnn-backprop-data)
15⍝│ 600 digits to train on, 200 to test with: each a label, then 784 pixels.
16tr ← 600 785 r̲eshape n̲umbers ⎕N̲GET "data/train.txt"
17te ← 200 785 r̲eshape n̲umbers ⎕N̲GET "data/test.txt"
18⍝│ The pictures, 0.0 to 1.0, and the labels one-hot (digit d is class d + 1).
19X ← 600 28 28 r̲eshape (f̲loat 1 d̲rop₂ tr) ÷ 255.0
20Y ← 10 ⁿⁿo̲neHot 1 + f̲loor r̲avel 1 t̲ake₂ tr
21TX ← 200 28 28 r̲eshape (f̲loat 1 d̲rop₂ te) ÷ 255.0
22ty ← 1 + f̲loor r̲avel 1 t̲ake₂ te
23
24⍝━━ Forward: the network of cnn-digits, for a batch x of n digits
25"cv:" u̲se< "Conv"
26⍝│ Each digit's probabilities for 0 to 9: the Conv library's
27⍝│ convolution (every 3 x 3 window of every digit, the 8 filters over
28⍝│ them in one matrix product) and pooling, then ReLU's and the dense
29⍝│ layer.
30ᵘp̲robs ← { (K, B, W) x → ⁿⁿs̲oftmax (((t̲ally x) c̲at 1352) r̲eshape ᶜᵛp̲ool ᵃᶜᵗr̲elu (K, B) ᶜᵛc̲onv x) ⁿⁿd̲ense W }
31
32⍝━━ Backward: each stage undone, in reverse
33⍝│ The gradients of the loss by K, B and W, for the digits numbered i.
34ᵘg̲rad ← { (K, B, W) i →
35 x ← i s̲elect X
36 y ← i s̲elect Y
37 n ← t̲ally x
38 V ← ᶜᵛw̲indows x
39 C ← (K, B) ᶜᵛc̲onv x
40 A ← ᵃᶜᵗr̲elu C
41 Q ← ᶜᵛp̲ool A
42 F ← (n c̲at 1352) r̲eshape Q
43 P ← ⁿⁿs̲oftmax F ⁿⁿd̲ense W
44 D3 ← (P − y) ÷ f̲loat n
45 GW ← (o̲\ F c̲at₂ ((n c̲at 1) r̲eshape 1.0)) '+ '× i̲nner D3
46 DQ ← (n c̲at 8 13 13) r̲eshape D3 '+ '× i̲nner o̲\ -1 d̲rop W
47 DC ← (A ᶜᵛu̲npool DQ) × f̲loat C > 0.0
48 (GK, GB) ← V ᶜᵛg̲rads DC
49 (GK, GB, GW)
50}
51
52⍝━━ Adam, on a state of (K, B, W, their averages, the step count)
53"op:" u̲se< "Optim"
54⍝│ The learning rate and the digits per step.
55lr ← 0.01
56bs ← 20
57⍝│ One step, on the next 20 training digits (30 steps go through all
58⍝│ 600): the Optim library's Adam on each of K, B and W.
59ᵘs̲tep ← { (K, B, W, MK, MB, MW, VK, VB, VW, t) →
60 i ← (r̲ange bs) + bs × (f̲loor t) m̲od 600 d̲iv bs
61 n ← 1.0 + t
62 (GK, GB, GW) ← (K, B, W) ᵘg̲rad i
63 (k, mk, vk) ← (lr, n) ᵒᵖa̲dam (K, MK, VK, GK)
64 (b, mb, vb) ← (lr, n) ᵒᵖa̲dam (B, MB, VB, GB)
65 (w, mw, vw) ← (lr, n) ᵒᵖa̲dam (W, MW, VW, GW)
66 (k, b, w, mk, mb, mw, vk, vb, vw, n)
67}
68⍝│ The starting state: small random filters and dense weights; the
69⍝│ filters' biases a little above 0, so that ReLU starts off its kink
70⍝│ (a blank window's output is then 0.01, not exactly 0).
71K ← 0.8 × (f̲loat (8 9 r̲eshape r̲oll! 72 r̲eshape 2001) − 1001) ÷ 1000.0
72B ← 8 r̲eshape 0.01
73W ← 0.05 × (f̲loat (1353 10 r̲eshape r̲oll! 13530 r̲eshape 2001) − 1001) ÷ 1000.0
74s0 ← (K, B, W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0)
75⍝│ The loss and the share right on the 200 test digits.
76ᵘt̲est ← { (K, B, W, _, _, _, _, _, _, _) →
77 p ← (K, B, W) ᵘp̲robs TX
78 ((10 ⁿⁿo̲neHot ty) ⁿⁿc̲rossEntropy p) c̲at ty ⁿⁿa̲ccuracy p
79}
80⍝ -- end of the core ----------------------------------------
81
82⍝━━ The check: every gradient against a finite difference
83⍝│ The loss on digits i, and e at item j of an array shaped as w.
84ᵘl̲oss ← { (K, B, W) i → (i s̲elect Y) ⁿⁿc̲rossEntropy (K, B, W) ᵘp̲robs i s̲elect X }
85e ← 0.00001
86ᵘb̲ump ← { w j → e × f̲loat (s̲hape w) r̲eshape j = r̲ange t̲ally r̲avel w }
87⍝│ On four digits: the first filter's 9 weights, the 8 biases, 6 dense weights.
88i4 ← 1 2 3 4
89(GK, GB, GW) ← (K, B, W) ᵘg̲rad i4
90FK ← '{ j → (((K + K ᵘb̲ump j, B, W) ᵘl̲oss i4) − (K − K ᵘb̲ump j, B, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 9
91FB ← '{ j → (((K, B + B ᵘb̲ump j, W) ᵘl̲oss i4) − (K, B − B ᵘb̲ump j, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 8
92wj ← 1 77 500 6000 13521 13530
93FW ← '{ j → (((K, B, W + W ᵘb̲ump j) ᵘl̲oss i4) − (K, B, W − W ᵘb̲ump j) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach wj
94gap ← ('m̲ax r̲/ a̲bs (9 t̲ake r̲avel GK) − FK) m̲ax ('m̲ax r̲/ a̲bs GB − FB) m̲ax 'm̲ax r̲/ a̲bs (wj s̲elect r̲avel GW) − FW
95gap < 1e-8
96
97⍝━━ Training
98⍝│ The test loss and share right: at the start, after 10 steps (200
99⍝│ digits), after 40 (800: each training digit seen about once and a
100⍝│ third).
101ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }
102s1 ← 10 'ᵘs̲tep p̲ower s0
103s2 ← 30 'ᵘs̲tep p̲ower s1
104ᵘr̲ound (ᵘt̲est s0) c̲at (ᵘt̲est s1) c̲at ᵘt̲est s2
105⍝│ The first 20 test digits as the trained network reads them, and their labels.
106(K2, B2, W2, _, _, _, _, _, _, _) ← s2
107(ⁿⁿa̲rgmax (K2, B2, W2) ᵘp̲robs 20 t̲ake TX) − 1
108(20 t̲ake ty) − 1