sourcelibs/Conv/src/Conv.xtl

1⍝│ Conv: convolution for a batch of pictures: 3 x 3 windows, filters as one matrix product, max-pooling, and their backward passes 2⍝│ 3⍝│ Import it with an alias of your choice: "cv:" u_se< "Conv". 4⍝│ Put libs/Conv/src on XETAL_PATH ("just path"); the reference is libs/Conv/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ A batch is n pictures of h x w (n x h x w); a picture alone is a 8⍝│ batch of one. Filters K are f rows of 9 (a 3 x 3 filter row by row) 9⍝│ with biases B (f); maps are n x f x (h - 2) x (w - 2) (valid 10⍝│ convolution), pooled by 2 x 2 blocks to half each side. 11 12⍝━━ Forward 13 14⍝│ w̲indows x: every 3 x 3 window of every picture as 9 rows, a 15⍝│ column per picture and place (the picture's nine shifted copies). 16⍝│ >> "cv:" u_se< "Conv" 17⍝│ >> cv:w_indows 1 3 3 r_eshape f_loat r_ange 9 18⍝│ 1.0 19⍝│ 2.0 20⍝│ 3.0 21⍝│ 4.0 22⍝│ 5.0 23⍝│ 6.0 24⍝│ 7.0 25⍝│ 8.0 26⍝│ 9.0 27ˡw̲indows ← { x → 28 places ← ((2 s̲elect s̲hape x) − 2) × (3 s̲elect s̲hape x) − 2 29 (9 c̲at places × t̲ally x) r̲eshape 1 d̲rop₅ -1 d̲rop₅ 1 d̲rop₄ -1 d̲rop₄ -1 0 1 o̲-₃ -1 0 1 o̲-₃ x 30} 31 32⍝│ (K, B) c̲onv x: each filter over every window of every picture, 33⍝│ plus its bias: one matrix product for the whole batch. 34ˡc̲onv ← { (K, B) x → 35 n ← t̲ally x 36 h ← (2 s̲elect s̲hape x) − 2 37 w ← (3 s̲elect s̲hape x) − 2 38 f ← t̲ally K 39 (n c̲at f c̲at h c̲at w) r̲eshape 2 1 3 t̲ranspose (f c̲at n c̲at h × w) r̲eshape (K '+ '× i̲nner ˡw̲indows x) + B 'l̲eft t̲able o̲ffsets n × h × w 40} 41 42⍝│ p̲ool a: the largest of each 2 x 2 block of the maps a (n x f x h x w): n x f x h/2 x w/2. 43⍝│ >> "cv:" u_se< "Conv" 44⍝│ >> cv:p_ool 1 1 2 4 r_eshape 1.0 5.0 2.0 2.0 3.0 4.0 2.0 2.0 45⍝│ 5.0 2.0 46ˡp̲ool ← { a → 'm̲ax r̲/₄₆ ((2 t̲ake s̲hape a) c̲at ((3 s̲elect s̲hape a) d̲iv 2) c̲at 2 c̲at ((4 s̲elect s̲hape a) d̲iv 2) c̲at 2) r̲eshape a } 47 48⍝━━ Backward 49 50⍝│ Each 2 x 2 block of the maps as a last axis of 4, and back. 51ʰb̲locks ← { a → 52 s ← s̲hape a 53 ((2 t̲ake s) c̲at ((3 s̲elect s) d̲iv 2) c̲at ((4 s̲elect s) d̲iv 2) c̲at 4) r̲eshape 1 2 3 5 4 6 t̲ranspose ((2 t̲ake s) c̲at ((3 s̲elect s) d̲iv 2) c̲at 2 c̲at ((4 s̲elect s) d̲iv 2) c̲at 2) r̲eshape a 54} 55ʰm̲aps ← { r → 56 s ← s̲hape r 57 ((2 t̲ake s) c̲at (2 × 3 s̲elect s) c̲at 2 × 4 s̲elect s) r̲eshape 1 2 3 5 4 6 t̲ranspose ((4 t̲ake s) c̲at 2 2) r̲eshape r 58} 59 60⍝│ a u̲npool d: the gradient d of the pooled maps sent back through 61⍝│ the pooling of the maps a: each block's to its largest item, the 62⍝│ first of them when several tie (MNIST's saturated strokes make ties 63⍝│ common, and the loss moves with one item, not all). 64⍝│ >> "cv:" u_se< "Conv" 65⍝│ >> (1 1 2 4 r_eshape 1.0 5.0 2.0 2.0 3.0 4.0 2.0 2.0) cv:u_npool 1 1 1 2 r_eshape 10.0 20.0 66⍝│ 0.0 10.0 20.0 0.0 67⍝│ 0.0 0.0 0.0 0.0 68ˡu̲npool ← { a d → 69 R ← ʰb̲locks a 70 hit ← f̲loat R = (s̲hape R) r̲eshape (r̲avel ˡp̲ool a) 'l̲eft t̲able 1 2 3 4 71 first ← hit × f̲loat 1.0 = '+ s̲\₅ hit 72 ʰm̲aps first × (s̲hape R) r̲eshape (r̲avel d) 'l̲eft t̲able 1 2 3 4 73} 74 75⍝│ V g̲rads D: the filters' and the biases' gradients, from the 76⍝│ windows V the convolution used and the gradient D of its maps (n x f 77⍝│ x h x w): what came back to each place, times the window there, 78⍝│ summed over every place of every picture: (GK, GB). 79ˡg̲rads ← { V D → 80 s ← s̲hape D 81 DM ← ((2 s̲elect s) c̲at ('× r̲/ s) d̲iv 2 s̲elect s) r̲eshape 2 1 3 t̲ranspose ((2 t̲ake s) c̲at (3 s̲elect s) × 4 s̲elect s) r̲eshape D 82 (DM '+ '× i̲nner o̲\ V, '+ r̲/₂ DM) 83}