sourceextensions/blender/demos/ReluSoftmax.xtl
1⍝│ ReluSoftmax.xtl -- ReLU and softmax on a tiny network's arrays, as a
2⍝│ video (Video.xtl). The network is X_eTaL-gpu's tiny-net
3⍝│ (libs/Accel/demos/tiny-net.xtl), run with the same library the GPU
4⍝│ runs, Accel (X_eTaL-gpu at GPU_COMMIT, on XETAL_PATH by
5⍝│ scripts/libraries.sh): its dense layers, ReLU, row maximum, row sums
6⍝│ and softmax. What is new here is only how they are shown: every
7⍝│ bar's place, size, color and timing, every label and the camera's
8⍝│ path, worked out from those arrays, and Video.xtl writes them into the
9⍝│ script Blender runs. Shared by the demo (demos/relu-softmax.xtl: the
10⍝│ video), its test (a few small frames) and the literate document
11⍝│ (docs/literate/relu-softmax.org).
12⍝│ Import: "rs:" u_se< "ReluSoftmax"
13
14"ac:" u̲se< "Accel"
15"vd:" u̲se< "Video"
16
17⍝━━ The network's arrays (Accel)
18
19⍝│ x: tiny-net's input, a batch of four examples of three features (4
20⍝│ by 3)
21ˡx ← 4 3 r̲eshape 0.5 -1.0 0.25 1.0 0.0 -0.5 -0.75 0.5 1.0 0.2 0.4 -0.6
22
23⍝│ w1: the first layer's weights, a row per input and a column per
24⍝│ output, its bias the last row (4 by 5: 3 inputs and a bias, 5
25⍝│ outputs): the numbers of X_eTaL-gpu's libs/Accel/data/w1.txt
26ˡw1 ← 4 5 r̲eshape 0.5 -0.25 0.75 0.1 -0.6 -0.3 0.8 0.2 -0.5 0.4 0.6 0.1 -0.7 0.3 0.2 0.05 -0.1 0.2 0.0 0.1
27
28⍝│ w2: the second layer's (6 by 2: 5 inputs and a bias, 2 outputs):
29⍝│ libs/Accel/data/w2.txt
30ˡw2 ← 6 2 r̲eshape 0.7 -0.4 -0.2 0.9 0.5 0.3 -0.6 0.2 0.3 -0.8 0.1 -0.1
31
32⍝│ z1: the first layer's output, ac:d_ense (4 by 5), some of it negative
33⍝│ >> "rs:" u_se< "ReluSoftmax"
34⍝│ >> s_hape rs:z1
35⍝│ 4 5
36ˡz1 ← ˡx ᵃᶜd̲ense ˡw1
37
38⍝│ h: z1 through ac:r_elu (4 by 5)
39ˡh ← ᵃᶜr̲elu ˡz1
40
41⍝│ z2: the second layer's output (4 by 2): two scores for each example
42ˡz2 ← ˡh ᵃᶜd̲ense ˡw2
43
44⍝│ p: the network's answer, ac:s_oftmax: each row two shares of 1 (4 by
45⍝│ 2)
46⍝│ >> "rs:" u_se< "ReluSoftmax"
47⍝│ >> "ac:" u_se< "Accel"
48⍝│ >> 0.000000001 > a_bs 1.0 - ac:r_owSums rs:p
49⍝│ 1 1 1 1
50ˡp ← ᵃᶜs̲oftmax ˡz2
51
52⍝ a value for each row spread along it (as ac:s_oftmax spreads them)
53ʰa̲long ← { v m → v 'l̲eft t̲able o̲ffsets f̲irst -1 t̲ake s̲hape m }
54
55⍝│ Softmax's steps, to show them one at a time (ac:s_oftmax does them in
56⍝│ one expression): rs:s_hifted m, each row less its ac:r_owMax (the
57⍝│ largest becomes 0)
58ˡs̲hifted ← { m → m − (ᵃᶜr̲owMax m) ʰa̲long m }
59
60⍝│ rs:e_xps m: e to each of the shifted rows: all positive, the largest 1
61ˡe̲xps ← { m → e̲xp ˡs̲hifted m }
62
63⍝│ rs:s_ums m: each row's e's added up, ac:r_owSums
64ˡs̲ums ← { m → ᵃᶜr̲owSums ˡe̲xps m }
65
66⍝│ each row's e's divided by their sum are ac:s_oftmax's shares
67⍝│ >> "rs:" u_se< "ReluSoftmax"
68⍝│ >> '+ r_/ r_avel a_bs rs:p - (rs:e_xps rs:z2) / (rs:s_ums rs:z2) 'l_eft t_able 1 2
69⍝│ 0.0
70ˡshares ← (ˡe̲xps ˡz2) ÷ (ˡs̲ums ˡz2) ʰa̲long ˡz2
71
72⍝━━ Colors (0 to 255)
73
74⍝│ orange: a positive value, and the first of a row's two shares
75ˡorange ← 255 179 71
76⍝│ blue: a negative value
77ˡblue ← 79 168 255
78⍝│ red: a value ReLU removes
79ˡred ← 255 74 61
80⍝│ cyan: the second of a row's two shares
81ˡcyan ← 79 214 255
82⍝│ white: a sum, and the labels
83ˡwhite ← 255 255 255
84⍝│ yellow: a kept value's label after ReLU, a sum's, a step's caption
85ˡyellow ← 255 216 77
86⍝│ pink: a removed value's label, and ReLU's caption
87ˡpink ← 255 138 128
88⍝│ gray: the rows' names
89ˡgray ← 197 204 214
90⍝│ the glass floors
91ˡglassColor ← 143 211 255
92⍝│ the sky, near black
93ˡsky ← 7 8 10
94
95⍝ a color for each of the mask's values: a where 1, b where 0 (n by 3)
96ʰp̲ick ← { m (a, b) → ((f̲loat m) '× t̲able f̲loat a) + (1.0 − f̲loat m) '× t̲able f̲loat b }
97
98⍝━━ Bars, keys and labels as tables
99
100⍝│ (x, y, heights, width) rs:b_ars 0: places for bars (n by 6: x y z w
101⍝│ d h), each standing on the floor (z 0) or hanging below it as its
102⍝│ height is positive or negative, never thinner than 0.004
103⍝│ >> "rs:" u_se< "ReluSoftmax"
104⍝│ >> (0.0 1.0, 0.0 0.0, 2.0 -1.0, 0.5) rs:b_ars 0
105⍝│ 0.0 0.0 1.0 0.5 0.5 2.0
106⍝│ 1.0 0.0 -0.5 0.5 0.5 1.0
107ˡb̲ars ← { (bx, by, hh, w) z →
108 n ← t̲ally hh
109 2 1 t̲ranspose (6 c̲at n) r̲eshape bx c̲at by c̲at (hh ÷ 2.0) c̲at (n r̲eshape w) c̲at (n r̲eshape w) c̲at 0.004 m̲ax a̲bs hh
110}
111
112⍝│ (places, colors, glow) rs:t_able 0: the rows vd:b_oxes takes -- each
113⍝│ place, then its color, then how strongly it glows
114ˡt̲able ← { (pl, c, g) z → pl c̲at₂ (f̲loat c) c̲at₂ (t̲ally pl) r̲eshape g }
115
116⍝│ a list of n by 6 (or n by 3) tables, one a key, as n by k by 6 (or 3):
117⍝│ what vd:m_oves (vd:t_ints) takes. keys is k, and the tables are
118⍝│ joined one under another
119ˡk̲eyed ← { k tabs →
120 s ← s̲hape tabs
121 n ← (f̲irst s) d̲iv k
122 2 1 3 t̲ranspose (k c̲at n c̲at 2 s̲elect s) r̲eshape tabs
123}
124
125⍝ label places (n by 7: x y z, height, color) for words at x y z
126ʰl̲abelAt ← { (bx, by, bz, size, c) z →
127 n ← t̲ally bx
128 (2 1 t̲ranspose (3 c̲at n) r̲eshape bx c̲at by c̲at bz) c̲at₂ (n r̲eshape size) c̲at₂ ((n c̲at 3) r̲eshape f̲loat c)
129}
130
131⍝━━ ReLU, 0 to 11 s
132
133⍝ meters a unit of value; a cell's spacing; where the stage sits
134hs ← 2.2
135gap ← 1.4
136ox ← -12.0
137
138⍝│ where each cell of z1 stands: x (4 by 5), 1.4 apart, about x -12
139ˡcellX ← 4 5 r̲eshape ox + gap × (f̲loat r̲ange 5) − 3.0
140⍝│ y (4 by 5), a row of z1 a row of the floor
141ˡcellY ← (gap × f̲loat (r̲ange 4) − 1) 'l̲eft t̲able r̲ange 5
142
143⍝│ the ReLU part of the script: z1 as bars on a glass floor, labeled;
144⍝│ the negatives turn red and shrink to 0, outlines left where they
145⍝│ were; then h's values
146ˡr̲eluScript ← { z →
147 v ← r̲avel ˡz1
148 bx ← r̲avel ˡcellX
149 by ← r̲avel ˡcellY
150 neg ← v < 0.0
151 n ← t̲ally v
152 k ← '+ r̲/ neg
153 p0 ← (bx, by, v × hs, 0.62) ˡb̲ars 0
154 p1 ← (bx, by, hs × ᵃᶜr̲elu v, 0.62) ˡb̲ars 0
155 c0 ← neg ʰp̲ick (ˡblue, ˡorange)
156 c1 ← neg ʰp̲ick (ˡred, ˡorange)
157 gv ← neg r̲eplicate v
158 top ← 0.45 + hs × ᵃᶜr̲elu v
159 hl ← ((bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) c̲at₂ 0.0
160 marked ← (4 t̲ake₂ hl) c̲at₂ neg ʰp̲ick (ˡpink, ˡyellow)
161 ((("floor", (1 6 r̲eshape ox c̲at 2.1 0.0 7.4 6.0 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at
162 (("floor", 1 2 r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at
163 (("z1", (p0, c0, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
164 (("z1", 0.0 4.0 5.6, 3 ˡk̲eyed p0 c̲at p0 c̲at p1) ᵛᵈm̲oves 0) c̲at
165 (("z1", 0.0 3.0 3.6, 3 ˡk̲eyed c0 c̲at c0 c̲at c1) ᵛᵈt̲ints 0) c̲at
166 (("z1", (n c̲at 2) r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at
167 (("gone", ((neg r̲eplicate bx, neg r̲eplicate by, gv × hs, 0.62) ˡb̲ars 0, (k c̲at 3) r̲eshape ˡred, 1.5) ˡt̲able 0) ᵛᵈo̲utlines 0) c̲at
168 (("gone", (k c̲at 2) r̲eshape 5.6 11.0) ᵛᵈs̲hown 0) c̲at
169 (("z1 values", 2 ᵛᵈf̲ixed v, (bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
170 (("z1 values", (n c̲at 2) r̲eshape 0.5 5.0) ᵛᵈs̲hown 0) c̲at
171 (("h values", 2 ᵛᵈf̲ixed ᵃᶜr̲elu v, marked) ᵛᵈl̲abels 0) c̲at
172 ("h values", (n c̲at 2) r̲eshape 6.0 11.0) ᵛᵈs̲hown 0)
173}
174
175⍝━━ Softmax, 11.5 to 30 s
176
177⍝│ when each step of softmax starts: the scores, less the largest, e to
178⍝│ them, the sums, divided, the shares; the video ends at 30
179ˡsteps ← 11.5 15.0 18.0 21.0 23.5 26.0
180⍝│ the video's length, 30 seconds
181ˡends ← 30.0
182
183⍝ meters a unit of value; each row's group's width; the result's height
184eh ← 2.0
185gw ← 4.4
186ph ← 3.2
187
188⍝│ the middle of each row's group, left to right (4), 4.4 apart
189ˡgroupX ← gw × (f̲loat r̲ange 4) − 2.5
190⍝│ where each score's bar stands (4 by 2): its group's middle less 0.9,
191⍝│ then 1.2 to the right
192ˡbarX ← (ˡgroupX − 0.9) '+ t̲able 0.0 1.2
193
194⍝ a step's start
195ʰa̲t ← { i → i s̲elect ˡsteps }
196
197⍝│ the softmax part of the script: each row's two bars become the
198⍝│ shifted scores, the e's, the shares; each row's sum stands beside
199⍝│ them; each row's result is a column of its two shares, 1 tall
200ˡs̲oftmaxScript ← { z →
201 sc ← r̲avel ˡz2
202 sh ← r̲avel ˡs̲hifted ˡz2
203 e ← r̲avel ˡe̲xps ˡz2
204 sm ← ˡs̲ums ˡz2
205 pp ← r̲avel ˡp
206 bx ← r̲avel ˡbarX
207 n ← t̲ally bx
208 by ← n r̲eshape 0.0
209 ks ← (ʰa̲t 1) c̲at ((ʰa̲t 2) − 1.0) c̲at (ʰa̲t 2) c̲at ((ʰa̲t 3) − 1.0) c̲at (ʰa̲t 3) c̲at ((ʰa̲t 5) − 1.0) c̲at ʰa̲t 5
210 hh ← sc c̲at sc c̲at sh c̲at sh c̲at e c̲at e c̲at pp
211 pl ← ((7 × n) c̲at 6) r̲eshape (r̲avel (7 × n) r̲eshape bx, (7 × n) r̲eshape by, eh × hh, 0.8) ˡb̲ars 0
212 first ← n t̲ake pl
213 share ← (r̲avel 4 2 r̲eshape 1 0) ʰp̲ick (ˡorange, ˡcyan)
214 cz ← (sc ≥ 0.0) ʰp̲ick (ˡorange, ˡblue)
215 sx ← 1.2 + r̲avel 2 s̲elect₂ ˡbarX
216 cx ← r̲avel 2 r̲eplicate 0.3 + ˡgroupX
217 below ← r̲avel (4 1 r̲eshape 0.0) c̲at₂ 1 t̲ake₂ ˡp
218 mid ← ph × below + pp ÷ 2.0
219 segs ← 2 1 t̲ranspose (6 c̲at n) r̲eshape cx c̲at by c̲at mid c̲at (n r̲eshape 1.6) c̲at (n r̲eshape 0.9) c̲at (ph × pp) − 0.04
220 gx ← 0.3 + ˡgroupX
221 ((("floor2", (1 6 r̲eshape 0.6 0.0 0.0 18.2 1.8 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at
222 (("floor2", 1 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
223 (("scores", (first, cz, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
224 (("scores", ks, 7 ˡk̲eyed pl) ᵛᵈm̲oves 0) c̲at
225 (("scores", (ʰa̲t 1) c̲at ((ʰa̲t 3) − 0.6) c̲at ʰa̲t 3, 3 ˡk̲eyed cz c̲at cz c̲at share) ᵛᵈt̲ints 0) c̲at
226 (("scores", (n c̲at 2) r̲eshape (ʰa̲t 1) c̲at ʰa̲t 6) ᵛᵈs̲hown 0) c̲at
227 ((("scores", sc, 1, 2, -0.4, bx) ʰv̲alues 0)) c̲at
228 ((("shifted", sh, 2, 3, -0.4, bx) ʰv̲alues 0)) c̲at
229 ((("e", e, 3, 5, -0.4, bx) ʰv̲alues 0)) c̲at
230 ((("shares", pp, 5, 6, 0.0, bx) ʰv̲alues 0)) c̲at
231 (("sums", ((sx, 4 r̲eshape 0.0, eh × sm, 0.8) ˡb̲ars 0, 4 3 r̲eshape ˡwhite, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
232 (("sums", 4 2 r̲eshape (ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at
233 (("sum values", 2 ᵛᵈf̲ixed sm, (sx, 4 r̲eshape -0.7, 0.4 + eh × sm, 0.46, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
234 (("sum values", 4 2 r̲eshape (0.3 + ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at
235 (("result", (segs, share, 1.3) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
236 (("result", (n c̲at 2) r̲eshape (ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
237 (("result values", 3 ᵛᵈf̲ixed pp, (cx, n r̲eshape -0.6, mid, 0.46, 0 0 0) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
238 (("result values", (n c̲at 2) r̲eshape (0.3 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
239 (("ones", "1|1|1|1", (gx, 4 r̲eshape -0.6, 4 r̲eshape ph + 0.45, 0.5, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
240 (("ones", 4 2 r̲eshape (0.6 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
241 (("rows", "row 1|row 2|row 3|row 4", (gx, 4 r̲eshape -1.3, 4 r̲eshape -0.1, 0.42, ˡgray) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
242 ("rows", 4 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0)
243}
244
245⍝ a step's values over its bars, white, shown from step i to step j
246⍝ (less a margin), lifted above the bars
247ʰv̲alues ← { (name, v, i, j, less, bx) z →
248 n ← t̲ally v
249 t ← ((name c̲at " values"), 2 ᵛᵈf̲ixed v, (bx, n r̲eshape -0.7, 0.4 + eh × v m̲ax 0.0, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0
250 t c̲at ((name c̲at " values"), (n c̲at 2) r̲eshape (0.3 + ʰa̲t i) c̲at less + ʰa̲t j) ᵛᵈs̲hown 0
251}
252
253⍝━━ The captions and the camera
254
255⍝ an array's name and shape, as [4 5]
256ʰs̲hape ← { name a → name c̲at " [" c̲at (f̲ormat s̲hape a) c̲at "]" }
257
258⍝│ the captions: what each part shows, | between them; the shapes and
259⍝│ the count of negatives are the arrays'
260ˡcaptionTexts ← (("z1" ʰs̲hape ˡz1) c̲at ": " c̲at (f̲ormat t̲ally r̲avel ˡz1) c̲at " values, " c̲at (f̲ormat '+ r̲/ (r̲avel ˡz1) < 0.0) c̲at " of them negative|" c̲at
261 "relu: every negative becomes 0|" c̲at
262 ("h" ʰs̲hape ˡh) c̲at ": the negatives are gone; the rest unchanged|" c̲at
263 ("z2" ʰs̲hape ˡz2) c̲at ": two scores in each row|" c̲at
264 "1. subtract each row's largest|2. exp: every value positive, the largest 1|3. add up each row|4. divide by the sum|" c̲at
265 ("p" ʰs̲hape ˡp) c̲at ": each row shares out 1")
266
267⍝│ when each caption shows (from, to) and its color
268ˡc̲aptionTable ← { z →
269 s ← ˡsteps
270 from ← 0.3 3.1 7.1 c̲at (0.3 + ʰa̲t 1) c̲at 1 d̲rop s
271 to ← 3.0 7.0 11.0 c̲at ((1 d̲rop s) − 0.2) c̲at ˡends
272 c ← 9 3 r̲eshape ˡwhite c̲at ˡpink c̲at ˡyellow c̲at ˡwhite c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡwhite
273 (2 1 t̲ranspose 2 9 r̲eshape from c̲at to) c̲at₂ f̲loat c
274}
275
276⍝│ the camera: over the ReLU part, then gliding back to see the four
277⍝│ rows of softmax side by side: at 0, 10.5, 12 and 30 s, each eye
278⍝│ looking at its aim
279ˡc̲ameraKeys ← { z →
280 rc ← ox c̲at 2.1 0.6
281 sc ← 0.3 0.0 1.4
282 ts ← 0.0 10.5 12.0 c̲at ˡends
283 eyes ← 4 3 r̲eshape (rc + -1.5 -10.5 11.5) c̲at (rc + 1.0 -11.0 11.0) c̲at (sc + -1.0 -20.5 7.0) c̲at sc + 1.0 -20.5 6.5
284 aims ← 4 3 r̲eshape rc c̲at rc c̲at sc c̲at sc
285 (ts, eyes, aims)
286}
287
288⍝━━ The scene
289
290⍝│ (width, height) rs:s_cene 0: the whole script Blender runs, but its
291⍝│ output: the stage (30 frames a second, 30 seconds, the sky), the
292⍝│ ReLU part, the softmax part, the captions and the camera, and the
293⍝│ report. The caller adds vd:r_ender or vd:s_tills.
294ˡs̲cene ← { (w, h) z →
295 ((ᵛᵈhead c̲at ((w, h, 30, ˡends, ˡsky, 38) ᵛᵈs̲tage 0) c̲at (ˡr̲eluScript 0) c̲at ˡs̲oftmaxScript 0) c̲at
296 ((ˡcaptionTexts, ˡc̲aptionTable 0) ᵛᵈc̲aptions 0) c̲at
297 (ˡc̲ameraKeys 0) ᵛᵈc̲amera 0)
298}