libraryextensions/blender/demos/ReluSoftmax.xtl
ReluSoftmax.xtl -- ReLU and softmax on a tiny network's arrays, as a video (Video.xtl). The network is X_eTaL-gpu's tiny-net (libs/Accel/demos/tiny-net.xtl), run with the same library the GPU runs, Accel (X_eTaL-gpu at GPU_COMMIT, on XETAL_PATH by scripts/libraries.sh): its dense layers, ReLU, row maximum, row sums and softmax. What is new here is only how they are shown: every bar's place, size, color and timing, every label and the camera's path, worked out from those arrays, and Video.xtl writes them into the script Blender runs. Shared by the demo (demos/relu-softmax.xtl: the video), its test (a few small frames) and the literate document (docs/literate/relu-softmax.org). Import: "rs:" u_se< "ReluSoftmax"
The network's arrays (Accel)
ˡx : Float
x: tiny-net's input, a batch of four examples of three features (4 by 3)
ˡx ← 4 3 r̲eshape 0.5 -1.0 0.25 1.0 0.0 -0.5 -0.75 0.5 1.0 0.2 0.4 -0.6
ˡw1 : Float
w1: the first layer's weights, a row per input and a column per output, its bias the last row (4 by 5: 3 inputs and a bias, 5 outputs): the numbers of X_eTaL-gpu's libs/Accel/data/w1.txt
ˡw1 ← 4 5 r̲eshape 0.5 -0.25 0.75 0.1 -0.6 -0.3 0.8 0.2 -0.5 0.4 0.6 0.1 -0.7 0.3 0.2 0.05 -0.1 0.2 0.0 0.1
ˡw2 : Float
w2: the second layer's (6 by 2: 5 inputs and a bias, 2 outputs): libs/Accel/data/w2.txt
ˡw2 ← 6 2 r̲eshape 0.7 -0.4 -0.2 0.9 0.5 0.3 -0.6 0.2 0.3 -0.8 0.1 -0.1
ˡz1 : Float
z1: the first layer's output, ac:d_ense (4 by 5), some of it negative
"rs:" u̲se< "ReluSoftmax"
s̲hape ʳˢz1 4 5
ˡz1 ← ˡx ᵃᶜd̲ense ˡw1
ˡz2 : Float
z2: the second layer's output (4 by 2): two scores for each example
ˡz2 ← ˡh ᵃᶜd̲ense ˡw2
ˡp : Float
p: the network's answer, ac:s_oftmax: each row two shares of 1 (4 by 2)
"rs:" u̲se< "ReluSoftmax"
"ac:" u̲se< "Accel"
0.000000001 > a̲bs 1.0 − ᵃᶜr̲owSums ʳˢp 1 1 1 1
ˡp ← ᵃᶜs̲oftmax ˡz2
ˡs̲hifted : Num a => a -> a
Softmax's steps, to show them one at a time (ac:s_oftmax does them in one expression): rs:s_hifted m, each row less its ac:r_owMax (the largest becomes 0)
ˡs̲hifted ← { m → m − (ᵃᶜr̲owMax m) ʰa̲long m }
ˡe̲xps : Num a => a -> Float
rs:e_xps m: e to each of the shifted rows: all positive, the largest 1
ˡe̲xps ← { m → e̲xp ˡs̲hifted m }
ˡs̲ums : Num a => a -> Float
rs:s_ums m: each row's e's added up, ac:r_owSums
ˡs̲ums ← { m → ᵃᶜr̲owSums ˡe̲xps m }
Colors (0 to 255)
ˡorange : Int
orange: a positive value, and the first of a row's two shares
ˡorange ← 255 179 71
ˡyellow : Int
yellow: a kept value's label after ReLU, a sum's, a step's caption
ˡyellow ← 255 216 77
ˡpink : Int
pink: a removed value's label, and ReLU's caption
ˡpink ← 255 138 128
ʰp̲ick : (Num a, Num b, Num c) => a -> (b, c) -> Float
ʰp̲ick ← { m (a, b) → ((f̲loat m) '× t̲able f̲loat a) + (1.0 − f̲loat m) '× t̲able f̲loat b }
Bars, keys and labels as tables
ˡb̲ars : Any a => (Float, Float, Float, Float) -> a -> Float
(x, y, heights, width) rs:b_ars 0: places for bars (n by 6: x y z w d h), each standing on the floor (z 0) or hanging below it as its height is positive or negative, never thinner than 0.004
"rs:" u̲se< "ReluSoftmax"
(0.0 1.0, 0.0 0.0, 2.0 -1.0, 0.5) ʳˢb̲ars 0 0.0 0.0 1.0 0.5 0.5 2.0 1.0 0.0 -0.5 0.5 0.5 1.0
ˡb̲ars ← { (bx, by, hh, w) z → n ← t̲ally hh 2 1 t̲ranspose (6 c̲at n) r̲eshape bx c̲at by c̲at (hh ÷ 2.0) c̲at (n r̲eshape w) c̲at (n r̲eshape w) c̲at 0.004 m̲ax a̲bs hh }
ˡt̲able : (Num a, Any b) => (Float, a, Float) -> b -> Float
(places, colors, glow) rs:t_able 0: the rows vd:b_oxes takes -- each place, then its color, then how strongly it glows
ˡt̲able ← { (pl, c, g) z → pl c̲at₂ (f̲loat c) c̲at₂ (t̲ally pl) r̲eshape g }
ˡk̲eyed : Int -> a -> a
a list of n by 6 (or n by 3) tables, one a key, as n by k by 6 (or 3): what vd:m_oves (vd:t_ints) takes. keys is k, and the tables are joined one under another
ˡk̲eyed ← { k tabs → s ← s̲hape tabs n ← (f̲irst s) d̲iv k 2 1 3 t̲ranspose (k c̲at n c̲at 2 s̲elect s) r̲eshape tabs }
ʰl̲abelAt : (Num a, Any b) => (Float, Float, Float, Float, a) -> b -> Float
ʰl̲abelAt ← { (bx, by, bz, size, c) z → n ← t̲ally bx (2 1 t̲ranspose (3 c̲at n) r̲eshape bx c̲at by c̲at bz) c̲at₂ (n r̲eshape size) c̲at₂ ((n c̲at 3) r̲eshape f̲loat c) }
ReLU, 0 to 11 s
ˡcellX : Float
where each cell of z1 stands: x (4 by 5), 1.4 apart, about x -12
ˡcellX ← 4 5 r̲eshape ox + gap × (f̲loat r̲ange 5) − 3.0
ˡcellY : Float
y (4 by 5), a row of z1 a row of the floor
ˡcellY ← (gap × f̲loat (r̲ange 4) − 1) 'l̲eft t̲able r̲ange 5
ˡr̲eluScript : Any a => a -> Char
the ReLU part of the script: z1 as bars on a glass floor, labeled; the negatives turn red and shrink to 0, outlines left where they were; then h's values
ˡr̲eluScript ← { z → v ← r̲avel ˡz1 bx ← r̲avel ˡcellX by ← r̲avel ˡcellY neg ← v < 0.0 n ← t̲ally v k ← '+ r̲/ neg p0 ← (bx, by, v × hs, 0.62) ˡb̲ars 0 p1 ← (bx, by, hs × ᵃᶜr̲elu v, 0.62) ˡb̲ars 0 c0 ← neg ʰp̲ick (ˡblue, ˡorange) c1 ← neg ʰp̲ick (ˡred, ˡorange) gv ← neg r̲eplicate v top ← 0.45 + hs × ᵃᶜr̲elu v hl ← ((bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) c̲at₂ 0.0 marked ← (4 t̲ake₂ hl) c̲at₂ neg ʰp̲ick (ˡpink, ˡyellow) ((("floor", (1 6 r̲eshape ox c̲at 2.1 0.0 7.4 6.0 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at (("floor", 1 2 r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at (("z1", (p0, c0, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at (("z1", 0.0 4.0 5.6, 3 ˡk̲eyed p0 c̲at p0 c̲at p1) ᵛᵈm̲oves 0) c̲at (("z1", 0.0 3.0 3.6, 3 ˡk̲eyed c0 c̲at c0 c̲at c1) ᵛᵈt̲ints 0) c̲at (("z1", (n c̲at 2) r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at (("gone", ((neg r̲eplicate bx, neg r̲eplicate by, gv × hs, 0.62) ˡb̲ars 0, (k c̲at 3) r̲eshape ˡred, 1.5) ˡt̲able 0) ᵛᵈo̲utlines 0) c̲at (("gone", (k c̲at 2) r̲eshape 5.6 11.0) ᵛᵈs̲hown 0) c̲at (("z1 values", 2 ᵛᵈf̲ixed v, (bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at (("z1 values", (n c̲at 2) r̲eshape 0.5 5.0) ᵛᵈs̲hown 0) c̲at (("h values", 2 ᵛᵈf̲ixed ᵃᶜr̲elu v, marked) ᵛᵈl̲abels 0) c̲at ("h values", (n c̲at 2) r̲eshape 6.0 11.0) ᵛᵈs̲hown 0) }
Softmax, 11.5 to 30 s
ˡsteps : Float
when each step of softmax starts: the scores, less the largest, e to them, the sums, divided, the shares; the video ends at 30
ˡsteps ← 11.5 15.0 18.0 21.0 23.5 26.0
ˡgroupX : Float
the middle of each row's group, left to right (4), 4.4 apart
ˡgroupX ← gw × (f̲loat r̲ange 4) − 2.5
ˡbarX : Float
where each score's bar stands (4 by 2): its group's middle less 0.9, then 1.2 to the right
ˡbarX ← (ˡgroupX − 0.9) '+ t̲able 0.0 1.2
ˡs̲oftmaxScript : Any a => a -> Char
the softmax part of the script: each row's two bars become the shifted scores, the e's, the shares; each row's sum stands beside them; each row's result is a column of its two shares, 1 tall
ˡs̲oftmaxScript ← { z → sc ← r̲avel ˡz2 sh ← r̲avel ˡs̲hifted ˡz2 e ← r̲avel ˡe̲xps ˡz2 sm ← ˡs̲ums ˡz2 pp ← r̲avel ˡp bx ← r̲avel ˡbarX n ← t̲ally bx by ← n r̲eshape 0.0 ks ← (ʰa̲t 1) c̲at ((ʰa̲t 2) − 1.0) c̲at (ʰa̲t 2) c̲at ((ʰa̲t 3) − 1.0) c̲at (ʰa̲t 3) c̲at ((ʰa̲t 5) − 1.0) c̲at ʰa̲t 5 hh ← sc c̲at sc c̲at sh c̲at sh c̲at e c̲at e c̲at pp pl ← ((7 × n) c̲at 6) r̲eshape (r̲avel (7 × n) r̲eshape bx, (7 × n) r̲eshape by, eh × hh, 0.8) ˡb̲ars 0 first ← n t̲ake pl share ← (r̲avel 4 2 r̲eshape 1 0) ʰp̲ick (ˡorange, ˡcyan) cz ← (sc ≥ 0.0) ʰp̲ick (ˡorange, ˡblue) sx ← 1.2 + r̲avel 2 s̲elect₂ ˡbarX cx ← r̲avel 2 r̲eplicate 0.3 + ˡgroupX below ← r̲avel (4 1 r̲eshape 0.0) c̲at₂ 1 t̲ake₂ ˡp mid ← ph × below + pp ÷ 2.0 segs ← 2 1 t̲ranspose (6 c̲at n) r̲eshape cx c̲at by c̲at mid c̲at (n r̲eshape 1.6) c̲at (n r̲eshape 0.9) c̲at (ph × pp) − 0.04 gx ← 0.3 + ˡgroupX ((("floor2", (1 6 r̲eshape 0.6 0.0 0.0 18.2 1.8 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at (("floor2", 1 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) c̲at (("scores", (first, cz, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at (("scores", ks, 7 ˡk̲eyed pl) ᵛᵈm̲oves 0) c̲at (("scores", (ʰa̲t 1) c̲at ((ʰa̲t 3) − 0.6) c̲at ʰa̲t 3, 3 ˡk̲eyed cz c̲at cz c̲at share) ᵛᵈt̲ints 0) c̲at (("scores", (n c̲at 2) r̲eshape (ʰa̲t 1) c̲at ʰa̲t 6) ᵛᵈs̲hown 0) c̲at ((("scores", sc, 1, 2, -0.4, bx) ʰv̲alues 0)) c̲at ((("shifted", sh, 2, 3, -0.4, bx) ʰv̲alues 0)) c̲at ((("e", e, 3, 5, -0.4, bx) ʰv̲alues 0)) c̲at ((("shares", pp, 5, 6, 0.0, bx) ʰv̲alues 0)) c̲at (("sums", ((sx, 4 r̲eshape 0.0, eh × sm, 0.8) ˡb̲ars 0, 4 3 r̲eshape ˡwhite, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at (("sums", 4 2 r̲eshape (ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at (("sum values", 2 ᵛᵈf̲ixed sm, (sx, 4 r̲eshape -0.7, 0.4 + eh × sm, 0.46, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at (("sum values", 4 2 r̲eshape (0.3 + ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at (("result", (segs, share, 1.3) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at (("result", (n c̲at 2) r̲eshape (ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at (("result values", 3 ᵛᵈf̲ixed pp, (cx, n r̲eshape -0.6, mid, 0.46, 0 0 0) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at (("result values", (n c̲at 2) r̲eshape (0.3 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at (("ones", "1|1|1|1", (gx, 4 r̲eshape -0.6, 4 r̲eshape ph + 0.45, 0.5, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at (("ones", 4 2 r̲eshape (0.6 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at (("rows", "row 1|row 2|row 3|row 4", (gx, 4 r̲eshape -1.3, 4 r̲eshape -0.1, 0.42, ˡgray) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at ("rows", 4 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) }
ʰv̲alues : Num a => (Char, Float, Int, Int, Float, Float) -> a -> Char
ʰv̲alues ← { (name, v, i, j, less, bx) z → n ← t̲ally v t ← ((name c̲at " values"), 2 ᵛᵈf̲ixed v, (bx, n r̲eshape -0.7, 0.4 + eh × v m̲ax 0.0, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0 t c̲at ((name c̲at " values"), (n c̲at 2) r̲eshape (0.3 + ʰa̲t i) c̲at less + ʰa̲t j) ᵛᵈs̲hown 0 }
The captions and the camera
ʰs̲hape : Char -> a -> Char
ʰs̲hape ← { name a → name c̲at " [" c̲at (f̲ormat s̲hape a) c̲at "]" }
ˡcaptionTexts : Char
the captions: what each part shows, | between them; the shapes and the count of negatives are the arrays'
ˡcaptionTexts ← (("z1" ʰs̲hape ˡz1) c̲at ": " c̲at (f̲ormat t̲ally r̲avel ˡz1) c̲at " values, " c̲at (f̲ormat '+ r̲/ (r̲avel ˡz1) < 0.0) c̲at " of them negative|" c̲at "relu: every negative becomes 0|" c̲at ("h" ʰs̲hape ˡh) c̲at ": the negatives are gone; the rest unchanged|" c̲at ("z2" ʰs̲hape ˡz2) c̲at ": two scores in each row|" c̲at "1. subtract each row's largest|2. exp: every value positive, the largest 1|3. add up each row|4. divide by the sum|" c̲at ("p" ʰs̲hape ˡp) c̲at ": each row shares out 1")
ˡc̲aptionTable : Any a => a -> Float
when each caption shows (from, to) and its color
ˡc̲aptionTable ← { z → s ← ˡsteps from ← 0.3 3.1 7.1 c̲at (0.3 + ʰa̲t 1) c̲at 1 d̲rop s to ← 3.0 7.0 11.0 c̲at ((1 d̲rop s) − 0.2) c̲at ˡends c ← 9 3 r̲eshape ˡwhite c̲at ˡpink c̲at ˡyellow c̲at ˡwhite c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡwhite (2 1 t̲ranspose 2 9 r̲eshape from c̲at to) c̲at₂ f̲loat c }
ˡc̲ameraKeys : Any a => a -> (Float, Float, Float)
the camera: over the ReLU part, then gliding back to see the four rows of softmax side by side: at 0, 10.5, 12 and 30 s, each eye looking at its aim
ˡc̲ameraKeys ← { z → rc ← ox c̲at 2.1 0.6 sc ← 0.3 0.0 1.4 ts ← 0.0 10.5 12.0 c̲at ˡends eyes ← 4 3 r̲eshape (rc + -1.5 -10.5 11.5) c̲at (rc + 1.0 -11.0 11.0) c̲at (sc + -1.0 -20.5 7.0) c̲at sc + 1.0 -20.5 6.5 aims ← 4 3 r̲eshape rc c̲at rc c̲at sc c̲at sc (ts, eyes, aims) }
The scene
ˡs̲cene : Any c => (a, b) -> c -> Char
(width, height) rs:s_cene 0: the whole script Blender runs, but its output: the stage (30 frames a second, 30 seconds, the sky), the ReLU part, the softmax part, the captions and the camera, and the report. The caller adds vd:r_ender or vd:s_tills.
ˡs̲cene ← { (w, h) z → ((ᵛᵈhead c̲at ((w, h, 30, ˡends, ˡsky, 38) ᵛᵈs̲tage 0) c̲at (ˡr̲eluScript 0) c̲at ˡs̲oftmaxScript 0) c̲at ((ˡcaptionTexts, ˡc̲aptionTable 0) ᵛᵈc̲aptions 0) c̲at (ˡc̲ameraKeys 0) ᵛᵈc̲amera 0) }