librarylibs/Conv/src/Conv.xtl
Conv: convolution for a batch of pictures: 3 x 3 windows, filters as one matrix product, max-pooling, and their backward passes
Import it with an alias of your choice: "cv:" u_se< "Conv". Put libs/Conv/src on XETAL_PATH ("just path"); the reference is libs/Conv/docs. Names with l: are exported; h: names are private to this file.
A batch is n pictures of h x w (n x h x w); a picture alone is a batch of one. Filters K are f rows of 9 (a 3 x 3 filter row by row) with biases B (f); maps are n x f x (h - 2) x (w - 2) (valid convolution), pooled by 2 x 2 blocks to half each side.
Forward
ˡw̲indows : a -> a
w̲indows x: every 3 x 3 window of every picture as 9 rows, a
column per picture and place (the picture's nine shifted copies).
"cv:" u̲se< "Conv"
ᶜᵛw̲indows 1 3 3 r̲eshape f̲loat r̲ange 9 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0
ˡw̲indows ← { x → places ← ((2 s̲elect s̲hape x) − 2) × (3 s̲elect s̲hape x) − 2 (9 c̲at places × t̲ally x) r̲eshape 1 d̲rop₅ -1 d̲rop₅ 1 d̲rop₄ -1 d̲rop₄ -1 0 1 o̲-₃ -1 0 1 o̲-₃ x }
ˡc̲onv : Num a => (a, a) -> a -> a
(K, B) c̲onv x: each filter over every window of every picture,
plus its bias: one matrix product for the whole batch.
ˡc̲onv ← { (K, B) x → n ← t̲ally x h ← (2 s̲elect s̲hape x) − 2 w ← (3 s̲elect s̲hape x) − 2 f ← t̲ally K (n c̲at f c̲at h c̲at w) r̲eshape 2 1 3 t̲ranspose (f c̲at n c̲at h × w) r̲eshape (K '+ '× i̲nner ˡw̲indows x) + B 'l̲eft t̲able o̲ffsets n × h × w }
ˡp̲ool : Num a => a -> a
p̲ool a: the largest of each 2 x 2 block of the maps a (n x f x h x w): n x f x h/2 x w/2.
"cv:" u̲se< "Conv"
ᶜᵛp̲ool 1 1 2 4 r̲eshape 1.0 5.0 2.0 2.0 3.0 4.0 2.0 2.0 5.0 2.0
ˡp̲ool ← { a → 'm̲ax r̲/₄₆ ((2 t̲ake s̲hape a) c̲at ((3 s̲elect s̲hape a) d̲iv 2) c̲at 2 c̲at ((4 s̲elect s̲hape a) d̲iv 2) c̲at 2) r̲eshape a }
Backward
ʰb̲locks : a -> a
Each 2 x 2 block of the maps as a last axis of 4, and back.
ʰb̲locks ← { a → s ← s̲hape a ((2 t̲ake s) c̲at ((3 s̲elect s) d̲iv 2) c̲at ((4 s̲elect s) d̲iv 2) c̲at 4) r̲eshape 1 2 3 5 4 6 t̲ranspose ((2 t̲ake s) c̲at ((3 s̲elect s) d̲iv 2) c̲at 2 c̲at ((4 s̲elect s) d̲iv 2) c̲at 2) r̲eshape a }
ʰm̲aps : a -> a
ʰm̲aps ← { r → s ← s̲hape r ((2 t̲ake s) c̲at (2 × 3 s̲elect s) c̲at 2 × 4 s̲elect s) r̲eshape 1 2 3 5 4 6 t̲ranspose ((4 t̲ake s) c̲at 2 2) r̲eshape r }
ˡu̲npool : Num a => a -> Float -> Float
a u̲npool d: the gradient d of the pooled maps sent back through
the pooling of the maps a: each block's to its largest item, the
first of them when several tie (MNIST's saturated strokes make ties
common, and the loss moves with one item, not all).
"cv:" u̲se< "Conv"
(1 1 2 4 r̲eshape 1.0 5.0 2.0 2.0 3.0 4.0 2.0 2.0) ᶜᵛu̲npool 1 1 1 2 r̲eshape 10.0 20.0 0.0 10.0 20.0 0.0 0.0 0.0 0.0 0.0
ˡu̲npool ← { a d → R ← ʰb̲locks a hit ← f̲loat R = (s̲hape R) r̲eshape (r̲avel ˡp̲ool a) 'l̲eft t̲able 1 2 3 4 first ← hit × f̲loat 1.0 = '+ s̲\₅ hit ʰm̲aps first × (s̲hape R) r̲eshape (r̲avel d) 'l̲eft t̲able 1 2 3 4 }
ˡg̲rads : Num a => a -> a -> (a, a)
V g̲rads D: the filters' and the biases' gradients, from the
windows V the convolution used and the gradient D of its maps (n x f
x h x w): what came back to each place, times the window there,
summed over every place of every picture: (GK, GB).
ˡg̲rads ← { V D → s ← s̲hape D DM ← ((2 s̲elect s) c̲at ('× r̲/ s) d̲iv 2 s̲elect s) r̲eshape 2 1 3 t̲ranspose ((2 t̲ake s) c̲at (3 s̲elect s) × 4 s̲elect s) r̲eshape D (DM '+ '× i̲nner o̲\ V, '+ r̲/₂ DM) }