sourceextensions/blender/demos/ReluSoftmax.xtl

1⍝│ ReluSoftmax.xtl -- ReLU and softmax on a tiny network's arrays, as a 2⍝│ video (Video.xtl). The network is X_eTaL-gpu's tiny-net 3⍝│ (libs/Accel/demos/tiny-net.xtl), run with the same library the GPU 4⍝│ runs, Accel (X_eTaL-gpu at GPU_COMMIT, on XETAL_PATH by 5⍝│ scripts/libraries.sh): its dense layers, ReLU, row maximum, row sums 6⍝│ and softmax. What is new here is only how they are shown: every 7⍝│ bar's place, size, color and timing, every label and the camera's 8⍝│ path, worked out from those arrays, and Video.xtl writes them into the 9⍝│ script Blender runs. Shared by the demo (demos/relu-softmax.xtl: the 10⍝│ video), its test (a few small frames) and the literate document 11⍝│ (docs/literate/relu-softmax.org). 12⍝│ Import: "rs:" u_se< "ReluSoftmax" 13 14"ac:" u̲se< "Accel" 15"vd:" u̲se< "Video" 16 17⍝━━ The network's arrays (Accel) 18 19⍝│ x: tiny-net's input, a batch of four examples of three features (4 20⍝│ by 3) 21ˡx ← 4 3 r̲eshape 0.5 -1.0 0.25 1.0 0.0 -0.5 -0.75 0.5 1.0 0.2 0.4 -0.6 22 23⍝│ w1: the first layer's weights, a row per input and a column per 24⍝│ output, its bias the last row (4 by 5: 3 inputs and a bias, 5 25⍝│ outputs): the numbers of X_eTaL-gpu's libs/Accel/data/w1.txt 26ˡw1 ← 4 5 r̲eshape 0.5 -0.25 0.75 0.1 -0.6 -0.3 0.8 0.2 -0.5 0.4 0.6 0.1 -0.7 0.3 0.2 0.05 -0.1 0.2 0.0 0.1 27 28⍝│ w2: the second layer's (6 by 2: 5 inputs and a bias, 2 outputs): 29⍝│ libs/Accel/data/w2.txt 30ˡw2 ← 6 2 r̲eshape 0.7 -0.4 -0.2 0.9 0.5 0.3 -0.6 0.2 0.3 -0.8 0.1 -0.1 31 32⍝│ z1: the first layer's output, ac:d_ense (4 by 5), some of it negative 33⍝│ >> "rs:" u_se< "ReluSoftmax" 34⍝│ >> s_hape rs:z1 35⍝│ 4 5 36ˡz1 ← ˡx ᵃᶜd̲ense ˡw1 37 38⍝│ h: z1 through ac:r_elu (4 by 5) 39ˡh ← ᵃᶜr̲elu ˡz1 40 41⍝│ z2: the second layer's output (4 by 2): two scores for each example 42ˡz2 ← ˡh ᵃᶜd̲ense ˡw2 43 44⍝│ p: the network's answer, ac:s_oftmax: each row two shares of 1 (4 by 45⍝│ 2) 46⍝│ >> "rs:" u_se< "ReluSoftmax" 47⍝│ >> "ac:" u_se< "Accel" 48⍝│ >> 0.000000001 > a_bs 1.0 - ac:r_owSums rs:p 49⍝│ 1 1 1 1 50ˡp ← ᵃᶜs̲oftmax ˡz2 51 52⍝ a value for each row spread along it (as ac:s_oftmax spreads them) 53ʰa̲long ← { v m → v 'l̲eft t̲able o̲ffsets f̲irst -1 t̲ake s̲hape m } 54 55⍝│ Softmax's steps, to show them one at a time (ac:s_oftmax does them in 56⍝│ one expression): rs:s_hifted m, each row less its ac:r_owMax (the 57⍝│ largest becomes 0) 58ˡs̲hifted ← { m → m − (ᵃᶜr̲owMax m) ʰa̲long m } 59 60⍝│ rs:e_xps m: e to each of the shifted rows: all positive, the largest 1 61ˡe̲xps ← { m → e̲xp ˡs̲hifted m } 62 63⍝│ rs:s_ums m: each row's e's added up, ac:r_owSums 64ˡs̲ums ← { m → ᵃᶜr̲owSums ˡe̲xps m } 65 66⍝│ each row's e's divided by their sum are ac:s_oftmax's shares 67⍝│ >> "rs:" u_se< "ReluSoftmax" 68⍝│ >> '+ r_/ r_avel a_bs rs:p - (rs:e_xps rs:z2) / (rs:s_ums rs:z2) 'l_eft t_able 1 2 69⍝│ 0.0 70ˡshares ← (ˡe̲xps ˡz2) ÷ (ˡs̲ums ˡz2) ʰa̲long ˡz2 71 72⍝━━ Colors (0 to 255) 73 74⍝│ orange: a positive value, and the first of a row's two shares 75ˡorange ← 255 179 71 76⍝│ blue: a negative value 77ˡblue ← 79 168 255 78⍝│ red: a value ReLU removes 79ˡred ← 255 74 61 80⍝│ cyan: the second of a row's two shares 81ˡcyan ← 79 214 255 82⍝│ white: a sum, and the labels 83ˡwhite ← 255 255 255 84⍝│ yellow: a kept value's label after ReLU, a sum's, a step's caption 85ˡyellow ← 255 216 77 86⍝│ pink: a removed value's label, and ReLU's caption 87ˡpink ← 255 138 128 88⍝│ gray: the rows' names 89ˡgray ← 197 204 214 90⍝│ the glass floors 91ˡglassColor ← 143 211 255 92⍝│ the sky, near black 93ˡsky ← 7 8 10 94 95⍝ a color for each of the mask's values: a where 1, b where 0 (n by 3) 96ʰp̲ick ← { m (a, b) → ((f̲loat m) '× t̲able f̲loat a) + (1.0 − f̲loat m) '× t̲able f̲loat b } 97 98⍝━━ Bars, keys and labels as tables 99 100⍝│ (x, y, heights, width) rs:b_ars 0: places for bars (n by 6: x y z w 101⍝│ d h), each standing on the floor (z 0) or hanging below it as its 102⍝│ height is positive or negative, never thinner than 0.004 103⍝│ >> "rs:" u_se< "ReluSoftmax" 104⍝│ >> (0.0 1.0, 0.0 0.0, 2.0 -1.0, 0.5) rs:b_ars 0 105⍝│ 0.0 0.0 1.0 0.5 0.5 2.0 106⍝│ 1.0 0.0 -0.5 0.5 0.5 1.0 107ˡb̲ars ← { (bx, by, hh, w) z → 108 n ← t̲ally hh 109 2 1 t̲ranspose (6 c̲at n) r̲eshape bx c̲at by c̲at (hh ÷ 2.0) c̲at (n r̲eshape w) c̲at (n r̲eshape w) c̲at 0.004 m̲ax a̲bs hh 110} 111 112⍝│ (places, colors, glow) rs:t_able 0: the rows vd:b_oxes takes -- each 113⍝│ place, then its color, then how strongly it glows 114ˡt̲able ← { (pl, c, g) z → pl c̲at₂ (f̲loat c) c̲at₂ (t̲ally pl) r̲eshape g } 115 116⍝│ a list of n by 6 (or n by 3) tables, one a key, as n by k by 6 (or 3): 117⍝│ what vd:m_oves (vd:t_ints) takes. keys is k, and the tables are 118⍝│ joined one under another 119ˡk̲eyed ← { k tabs → 120 s ← s̲hape tabs 121 n ← (f̲irst s) d̲iv k 122 2 1 3 t̲ranspose (k c̲at n c̲at 2 s̲elect s) r̲eshape tabs 123} 124 125⍝ label places (n by 7: x y z, height, color) for words at x y z 126ʰl̲abelAt ← { (bx, by, bz, size, c) z → 127 n ← t̲ally bx 128 (2 1 t̲ranspose (3 c̲at n) r̲eshape bx c̲at by c̲at bz) c̲at₂ (n r̲eshape size) c̲at₂ ((n c̲at 3) r̲eshape f̲loat c) 129} 130 131⍝━━ ReLU, 0 to 11 s 132 133⍝ meters a unit of value; a cell's spacing; where the stage sits 134hs ← 2.2 135gap ← 1.4 136ox ← -12.0 137 138⍝│ where each cell of z1 stands: x (4 by 5), 1.4 apart, about x -12 139ˡcellX ← 4 5 r̲eshape ox + gap × (f̲loat r̲ange 5) − 3.0 140⍝│ y (4 by 5), a row of z1 a row of the floor 141ˡcellY ← (gap × f̲loat (r̲ange 4) − 1) 'l̲eft t̲able r̲ange 5 142 143⍝│ the ReLU part of the script: z1 as bars on a glass floor, labeled; 144⍝│ the negatives turn red and shrink to 0, outlines left where they 145⍝│ were; then h's values 146ˡr̲eluScript ← { z → 147 v ← r̲avel ˡz1 148 bx ← r̲avel ˡcellX 149 by ← r̲avel ˡcellY 150 neg ← v < 0.0 151 n ← t̲ally v 152 k ← '+ r̲/ neg 153 p0 ← (bx, by, v × hs, 0.62) ˡb̲ars 0 154 p1 ← (bx, by, hs × ᵃᶜr̲elu v, 0.62) ˡb̲ars 0 155 c0 ← neg ʰp̲ick (ˡblue, ˡorange) 156 c1 ← neg ʰp̲ick (ˡred, ˡorange) 157 gv ← neg r̲eplicate v 158 top ← 0.45 + hs × ᵃᶜr̲elu v 159 hl ← ((bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) c̲at₂ 0.0 160 marked ← (4 t̲ake₂ hl) c̲at₂ neg ʰp̲ick (ˡpink, ˡyellow) 161 ((("floor", (1 6 r̲eshape ox c̲at 2.1 0.0 7.4 6.0 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at 162 (("floor", 1 2 r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at 163 (("z1", (p0, c0, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at 164 (("z1", 0.0 4.0 5.6, 3 ˡk̲eyed p0 c̲at p0 c̲at p1) ᵛᵈm̲oves 0) c̲at 165 (("z1", 0.0 3.0 3.6, 3 ˡk̲eyed c0 c̲at c0 c̲at c1) ᵛᵈt̲ints 0) c̲at 166 (("z1", (n c̲at 2) r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at 167 (("gone", ((neg r̲eplicate bx, neg r̲eplicate by, gv × hs, 0.62) ˡb̲ars 0, (k c̲at 3) r̲eshape ˡred, 1.5) ˡt̲able 0) ᵛᵈo̲utlines 0) c̲at 168 (("gone", (k c̲at 2) r̲eshape 5.6 11.0) ᵛᵈs̲hown 0) c̲at 169 (("z1 values", 2 ᵛᵈf̲ixed v, (bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at 170 (("z1 values", (n c̲at 2) r̲eshape 0.5 5.0) ᵛᵈs̲hown 0) c̲at 171 (("h values", 2 ᵛᵈf̲ixed ᵃᶜr̲elu v, marked) ᵛᵈl̲abels 0) c̲at 172 ("h values", (n c̲at 2) r̲eshape 6.0 11.0) ᵛᵈs̲hown 0) 173} 174 175⍝━━ Softmax, 11.5 to 30 s 176 177⍝│ when each step of softmax starts: the scores, less the largest, e to 178⍝│ them, the sums, divided, the shares; the video ends at 30 179ˡsteps ← 11.5 15.0 18.0 21.0 23.5 26.0 180⍝│ the video's length, 30 seconds 181ˡends ← 30.0 182 183⍝ meters a unit of value; each row's group's width; the result's height 184eh ← 2.0 185gw ← 4.4 186ph ← 3.2 187 188⍝│ the middle of each row's group, left to right (4), 4.4 apart 189ˡgroupX ← gw × (f̲loat r̲ange 4) − 2.5 190⍝│ where each score's bar stands (4 by 2): its group's middle less 0.9, 191⍝│ then 1.2 to the right 192ˡbarX ← (ˡgroupX − 0.9) '+ t̲able 0.0 1.2 193 194⍝ a step's start 195ʰa̲t ← { i → i s̲elect ˡsteps } 196 197⍝│ the softmax part of the script: each row's two bars become the 198⍝│ shifted scores, the e's, the shares; each row's sum stands beside 199⍝│ them; each row's result is a column of its two shares, 1 tall 200ˡs̲oftmaxScript ← { z → 201 sc ← r̲avel ˡz2 202 sh ← r̲avel ˡs̲hifted ˡz2 203 e ← r̲avel ˡe̲xps ˡz2 204 sm ← ˡs̲ums ˡz2 205 pp ← r̲avel ˡp 206 bx ← r̲avel ˡbarX 207 n ← t̲ally bx 208 by ← n r̲eshape 0.0 209 ks ← (ʰa̲t 1) c̲at ((ʰa̲t 2) − 1.0) c̲at (ʰa̲t 2) c̲at ((ʰa̲t 3) − 1.0) c̲at (ʰa̲t 3) c̲at ((ʰa̲t 5) − 1.0) c̲at ʰa̲t 5 210 hh ← sc c̲at sc c̲at sh c̲at sh c̲at e c̲at e c̲at pp 211 pl ← ((7 × n) c̲at 6) r̲eshape (r̲avel (7 × n) r̲eshape bx, (7 × n) r̲eshape by, eh × hh, 0.8) ˡb̲ars 0 212 first ← n t̲ake pl 213 share ← (r̲avel 4 2 r̲eshape 1 0) ʰp̲ick (ˡorange, ˡcyan) 214 cz ← (sc ≥ 0.0) ʰp̲ick (ˡorange, ˡblue) 215 sx ← 1.2 + r̲avel 2 s̲elect₂ ˡbarX 216 cx ← r̲avel 2 r̲eplicate 0.3 + ˡgroupX 217 below ← r̲avel (4 1 r̲eshape 0.0) c̲at₂ 1 t̲ake₂ ˡp 218 mid ← ph × below + pp ÷ 2.0 219 segs ← 2 1 t̲ranspose (6 c̲at n) r̲eshape cx c̲at by c̲at mid c̲at (n r̲eshape 1.6) c̲at (n r̲eshape 0.9) c̲at (ph × pp) − 0.04 220 gx ← 0.3 + ˡgroupX 221 ((("floor2", (1 6 r̲eshape 0.6 0.0 0.0 18.2 1.8 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at 222 (("floor2", 1 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) c̲at 223 (("scores", (first, cz, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at 224 (("scores", ks, 7 ˡk̲eyed pl) ᵛᵈm̲oves 0) c̲at 225 (("scores", (ʰa̲t 1) c̲at ((ʰa̲t 3) − 0.6) c̲at ʰa̲t 3, 3 ˡk̲eyed cz c̲at cz c̲at share) ᵛᵈt̲ints 0) c̲at 226 (("scores", (n c̲at 2) r̲eshape (ʰa̲t 1) c̲at ʰa̲t 6) ᵛᵈs̲hown 0) c̲at 227 ((("scores", sc, 1, 2, -0.4, bx) ʰv̲alues 0)) c̲at 228 ((("shifted", sh, 2, 3, -0.4, bx) ʰv̲alues 0)) c̲at 229 ((("e", e, 3, 5, -0.4, bx) ʰv̲alues 0)) c̲at 230 ((("shares", pp, 5, 6, 0.0, bx) ʰv̲alues 0)) c̲at 231 (("sums", ((sx, 4 r̲eshape 0.0, eh × sm, 0.8) ˡb̲ars 0, 4 3 r̲eshape ˡwhite, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at 232 (("sums", 4 2 r̲eshape (ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at 233 (("sum values", 2 ᵛᵈf̲ixed sm, (sx, 4 r̲eshape -0.7, 0.4 + eh × sm, 0.46, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at 234 (("sum values", 4 2 r̲eshape (0.3 + ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at 235 (("result", (segs, share, 1.3) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at 236 (("result", (n c̲at 2) r̲eshape (ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at 237 (("result values", 3 ᵛᵈf̲ixed pp, (cx, n r̲eshape -0.6, mid, 0.46, 0 0 0) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at 238 (("result values", (n c̲at 2) r̲eshape (0.3 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at 239 (("ones", "1|1|1|1", (gx, 4 r̲eshape -0.6, 4 r̲eshape ph + 0.45, 0.5, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at 240 (("ones", 4 2 r̲eshape (0.6 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at 241 (("rows", "row 1|row 2|row 3|row 4", (gx, 4 r̲eshape -1.3, 4 r̲eshape -0.1, 0.42, ˡgray) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at 242 ("rows", 4 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) 243} 244 245⍝ a step's values over its bars, white, shown from step i to step j 246⍝ (less a margin), lifted above the bars 247ʰv̲alues ← { (name, v, i, j, less, bx) z → 248 n ← t̲ally v 249 t ← ((name c̲at " values"), 2 ᵛᵈf̲ixed v, (bx, n r̲eshape -0.7, 0.4 + eh × v m̲ax 0.0, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0 250 t c̲at ((name c̲at " values"), (n c̲at 2) r̲eshape (0.3 + ʰa̲t i) c̲at less + ʰa̲t j) ᵛᵈs̲hown 0 251} 252 253⍝━━ The captions and the camera 254 255⍝ an array's name and shape, as [4 5] 256ʰs̲hape ← { name a → name c̲at " [" c̲at (f̲ormat s̲hape a) c̲at "]" } 257 258⍝│ the captions: what each part shows, | between them; the shapes and 259⍝│ the count of negatives are the arrays' 260ˡcaptionTexts ← (("z1" ʰs̲hape ˡz1) c̲at ": " c̲at (f̲ormat t̲ally r̲avel ˡz1) c̲at " values, " c̲at (f̲ormat '+ r̲/ (r̲avel ˡz1) < 0.0) c̲at " of them negative|" c̲at 261 "relu: every negative becomes 0|" c̲at 262 ("h" ʰs̲hape ˡh) c̲at ": the negatives are gone; the rest unchanged|" c̲at 263 ("z2" ʰs̲hape ˡz2) c̲at ": two scores in each row|" c̲at 264 "1. subtract each row's largest|2. exp: every value positive, the largest 1|3. add up each row|4. divide by the sum|" c̲at 265 ("p" ʰs̲hape ˡp) c̲at ": each row shares out 1") 266 267⍝│ when each caption shows (from, to) and its color 268ˡc̲aptionTable ← { z → 269 s ← ˡsteps 270 from ← 0.3 3.1 7.1 c̲at (0.3 + ʰa̲t 1) c̲at 1 d̲rop s 271 to ← 3.0 7.0 11.0 c̲at ((1 d̲rop s) − 0.2) c̲at ˡends 272 c ← 9 3 r̲eshape ˡwhite c̲at ˡpink c̲at ˡyellow c̲at ˡwhite c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡwhite 273 (2 1 t̲ranspose 2 9 r̲eshape from c̲at to) c̲at₂ f̲loat c 274} 275 276⍝│ the camera: over the ReLU part, then gliding back to see the four 277⍝│ rows of softmax side by side: at 0, 10.5, 12 and 30 s, each eye 278⍝│ looking at its aim 279ˡc̲ameraKeys ← { z → 280 rc ← ox c̲at 2.1 0.6 281 sc ← 0.3 0.0 1.4 282 ts ← 0.0 10.5 12.0 c̲at ˡends 283 eyes ← 4 3 r̲eshape (rc + -1.5 -10.5 11.5) c̲at (rc + 1.0 -11.0 11.0) c̲at (sc + -1.0 -20.5 7.0) c̲at sc + 1.0 -20.5 6.5 284 aims ← 4 3 r̲eshape rc c̲at rc c̲at sc c̲at sc 285 (ts, eyes, aims) 286} 287 288⍝━━ The scene 289 290⍝│ (width, height) rs:s_cene 0: the whole script Blender runs, but its 291⍝│ output: the stage (30 frames a second, 30 seconds, the sky), the 292⍝│ ReLU part, the softmax part, the captions and the camera, and the 293⍝│ report. The caller adds vd:r_ender or vd:s_tills. 294ˡs̲cene ← { (w, h) z → 295 ((ᵛᵈhead c̲at ((w, h, 30, ˡends, ˡsky, 38) ᵛᵈs̲tage 0) c̲at (ˡr̲eluScript 0) c̲at ˡs̲oftmaxScript 0) c̲at 296 ((ˡcaptionTexts, ˡc̲aptionTable 0) ᵛᵈc̲aptions 0) c̲at 297 (ˡc̲ameraKeys 0) ᵛᵈc̲amera 0) 298}