sourcedemos/cnn-backprop/cnn-backprop.xtl

1⍝!/usr/bin/env xetal 2⍝│ Training a tiny convolutional network in X_eTaL, from random 3⍝│ weights, on 600 handwritten digits: 28 x 28 -> 8 filters of 3 x 3 4⍝│ -> ReLU -> 2 x 2 max-pooling -> dense -> softmax, as cnn-digits 5⍝│ reads them. The backward pass undoes each stage in turn: the dense 6⍝│ layer, the pooling (each block's gradient to its largest item), the 7⍝│ ReLU and the convolution, each a few array expressions over the 8⍝│ whole batch; it is checked against finite differences. Adam moves 9⍝│ the weights, 20 digits a step, and p_ower repeats the step. 10 11"nn:" u̲se< "NN" 12"act:" u̲se< "Activations" 13 14⍝━━ The digits, read from data/ (just cnn-backprop-data) 15⍝│ 600 digits to train on, 200 to test with: each a label, then 784 pixels. 16tr ← 600 785 r̲eshape n̲umbers ⎕N̲GET "data/train.txt" 17te ← 200 785 r̲eshape n̲umbers ⎕N̲GET "data/test.txt" 18⍝│ The pictures, 0.0 to 1.0, and the labels one-hot (digit d is class d + 1). 19X ← 600 28 28 r̲eshape (f̲loat 1 d̲rop₂ tr) ÷ 255.0 20Y ← 10 ⁿⁿo̲neHot 1 + f̲loor r̲avel 1 t̲ake₂ tr 21TX ← 200 28 28 r̲eshape (f̲loat 1 d̲rop₂ te) ÷ 255.0 22ty ← 1 + f̲loor r̲avel 1 t̲ake₂ te 23 24⍝━━ Forward: the network of cnn-digits, for a batch x of n digits 25"cv:" u̲se< "Conv" 26⍝│ Each digit's probabilities for 0 to 9: the Conv library's 27⍝│ convolution (every 3 x 3 window of every digit, the 8 filters over 28⍝│ them in one matrix product) and pooling, then ReLU's and the dense 29⍝│ layer. 30ᵘp̲robs ← { (K, B, W) x → ⁿⁿs̲oftmax (((t̲ally x) c̲at 1352) r̲eshape ᶜᵛp̲ool ᵃᶜᵗr̲elu (K, B) ᶜᵛc̲onv x) ⁿⁿd̲ense W } 31 32⍝━━ Backward: each stage undone, in reverse 33⍝│ The gradients of the loss by K, B and W, for the digits numbered i. 34ᵘg̲rad ← { (K, B, W) i → 35 x ← i s̲elect X 36 y ← i s̲elect Y 37 n ← t̲ally x 38 V ← ᶜᵛw̲indows x 39 C ← (K, B) ᶜᵛc̲onv x 40 A ← ᵃᶜᵗr̲elu C 41 Q ← ᶜᵛp̲ool A 42 F ← (n c̲at 1352) r̲eshape Q 43 P ← ⁿⁿs̲oftmax F ⁿⁿd̲ense W 44 D3 ← (P − y) ÷ f̲loat n 45 GW ← (o̲\ F c̲at₂ ((n c̲at 1) r̲eshape 1.0)) '+ '× i̲nner D3 46 DQ ← (n c̲at 8 13 13) r̲eshape D3 '+ '× i̲nner o̲\ -1 d̲rop W 47 DC ← (A ᶜᵛu̲npool DQ) × f̲loat C > 0.0 48 (GK, GB) ← V ᶜᵛg̲rads DC 49 (GK, GB, GW) 50} 51 52⍝━━ Adam, on a state of (K, B, W, their averages, the step count) 53"op:" u̲se< "Optim" 54⍝│ The learning rate and the digits per step. 55lr ← 0.01 56bs ← 20 57⍝│ One step, on the next 20 training digits (30 steps go through all 58⍝│ 600): the Optim library's Adam on each of K, B and W. 59ᵘs̲tep ← { (K, B, W, MK, MB, MW, VK, VB, VW, t) → 60 i ← (r̲ange bs) + bs × (f̲loor t) m̲od 600 d̲iv bs 61 n ← 1.0 + t 62 (GK, GB, GW) ← (K, B, W) ᵘg̲rad i 63 (k, mk, vk) ← (lr, n) ᵒᵖa̲dam (K, MK, VK, GK) 64 (b, mb, vb) ← (lr, n) ᵒᵖa̲dam (B, MB, VB, GB) 65 (w, mw, vw) ← (lr, n) ᵒᵖa̲dam (W, MW, VW, GW) 66 (k, b, w, mk, mb, mw, vk, vb, vw, n) 67} 68⍝│ The starting state: small random filters and dense weights; the 69⍝│ filters' biases a little above 0, so that ReLU starts off its kink 70⍝│ (a blank window's output is then 0.01, not exactly 0). 71K ← 0.8 × (f̲loat (8 9 r̲eshape r̲oll! 72 r̲eshape 2001) − 1001) ÷ 1000.0 72B ← 8 r̲eshape 0.01 73W ← 0.05 × (f̲loat (1353 10 r̲eshape r̲oll! 13530 r̲eshape 2001) − 1001) ÷ 1000.0 74s0 ← (K, B, W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0 × K, 0.0 × B, 0.0 × W, 0.0) 75⍝│ The loss and the share right on the 200 test digits. 76ᵘt̲est ← { (K, B, W, _, _, _, _, _, _, _) → 77 p ← (K, B, W) ᵘp̲robs TX 78 ((10 ⁿⁿo̲neHot ty) ⁿⁿc̲rossEntropy p) c̲at ty ⁿⁿa̲ccuracy p 79} 80⍝ -- end of the core ---------------------------------------- 81 82⍝━━ The check: every gradient against a finite difference 83⍝│ The loss on digits i, and e at item j of an array shaped as w. 84ᵘl̲oss ← { (K, B, W) i → (i s̲elect Y) ⁿⁿc̲rossEntropy (K, B, W) ᵘp̲robs i s̲elect X } 85e ← 0.00001 86ᵘb̲ump ← { w j → e × f̲loat (s̲hape w) r̲eshape j = r̲ange t̲ally r̲avel w } 87⍝│ On four digits: the first filter's 9 weights, the 8 biases, 6 dense weights. 88i4 ← 1 2 3 4 89(GK, GB, GW) ← (K, B, W) ᵘg̲rad i4 90FK ← '{ j → (((K + K ᵘb̲ump j, B, W) ᵘl̲oss i4) − (K − K ᵘb̲ump j, B, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 9 91FB ← '{ j → (((K, B + B ᵘb̲ump j, W) ᵘl̲oss i4) − (K, B − B ᵘb̲ump j, W) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach r̲ange 8 92wj ← 1 77 500 6000 13521 13530 93FW ← '{ j → (((K, B, W + W ᵘb̲ump j) ᵘl̲oss i4) − (K, B, W − W ᵘb̲ump j) ᵘl̲oss i4) ÷ 2.0 × e } e̲ach wj 94gap ← ('m̲ax r̲/ a̲bs (9 t̲ake r̲avel GK) − FK) m̲ax ('m̲ax r̲/ a̲bs GB − FB) m̲ax 'm̲ax r̲/ a̲bs (wj s̲elect r̲avel GW) − FW 95gap < 1e-8 96 97⍝━━ Training 98⍝│ The test loss and share right: at the start, after 10 steps (200 99⍝│ digits), after 40 (800: each training digit seen about once and a 100⍝│ third). 101ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 } 102s1 ← 10 'ᵘs̲tep p̲ower s0 103s2 ← 30 'ᵘs̲tep p̲ower s1 104ᵘr̲ound (ᵘt̲est s0) c̲at (ᵘt̲est s1) c̲at ᵘt̲est s2 105⍝│ The first 20 test digits as the trained network reads them, and their labels. 106(K2, B2, W2, _, _, _, _, _, _, _) ← s2 107(ⁿⁿa̲rgmax (K2, B2, W2) ᵘp̲robs 20 t̲ake TX) − 1 108(20 t̲ake ty) − 1