libraryextensions/blender/demos/ReluSoftmax.xtl

ReluSoftmax.xtl -- ReLU and softmax on a tiny network's arrays, as a video (Video.xtl). The network is X_eTaL-gpu's tiny-net (libs/Accel/demos/tiny-net.xtl), run with the same library the GPU runs, Accel (X_eTaL-gpu at GPU_COMMIT, on XETAL_PATH by scripts/libraries.sh): its dense layers, ReLU, row maximum, row sums and softmax. What is new here is only how they are shown: every bar's place, size, color and timing, every label and the camera's path, worked out from those arrays, and Video.xtl writes them into the script Blender runs. Shared by the demo (demos/relu-softmax.xtl: the video), its test (a few small frames) and the literate document (docs/literate/relu-softmax.org). Import: "rs:" u_se< "ReluSoftmax"

source · imports ac: work/gpu/libs/Accel/src/Accel.xtl; vd: extensions/blender/lib/Video.xtl

The network's arrays (Accel)

ˡx : Float

value · line 21

x: tiny-net's input, a batch of four examples of three features (4 by 3)

ˡx ← 4 3 r̲eshape 0.5 -1.0 0.25  1.0 0.0 -0.5  -0.75 0.5 1.0  0.2 0.4 -0.6
Used in: ˡz1

ˡw1 : Float

value · line 26

w1: the first layer's weights, a row per input and a column per output, its bias the last row (4 by 5: 3 inputs and a bias, 5 outputs): the numbers of X_eTaL-gpu's libs/Accel/data/w1.txt

ˡw1 ← 4 5 r̲eshape 0.5 -0.25 0.75 0.1 -0.6  -0.3 0.8 0.2 -0.5 0.4  0.6 0.1 -0.7 0.3 0.2  0.05 -0.1 0.2 0.0 0.1
Used in: ˡz1

ˡw2 : Float

value · line 30

w2: the second layer's (6 by 2: 5 inputs and a bias, 2 outputs): libs/Accel/data/w2.txt

ˡw2 ← 6 2 r̲eshape 0.7 -0.4  -0.2 0.9  0.5 0.3  -0.6 0.2  0.3 -0.8  0.1 -0.1
Used in: ˡz2

ˡz1 : Float

value · line 36

z1: the first layer's output, ac:d_ense (4 by 5), some of it negative

      "rs:" u̲se< "ReluSoftmax"
      s̲hape ʳˢz1
4 5
ˡz1 ← ˡx ᵃᶜd̲ense ˡw1

ˡh : Float

value · line 39

h: z1 through ac:r_elu (4 by 5)

ˡh ← ᵃᶜr̲elu ˡz1

ˡz2 : Float

value · line 42

z2: the second layer's output (4 by 2): two scores for each example

ˡz2 ← ˡh ᵃᶜd̲ense ˡw2

ˡp : Float

value · line 50

p: the network's answer, ac:s_oftmax: each row two shares of 1 (4 by 2)

      "rs:" u̲se< "ReluSoftmax"
      "ac:" u̲se< "Accel"
      0.000000001 > a̲bs 1.0 − ᵃᶜr̲owSums ʳˢp
1 1 1 1
ˡp ← ᵃᶜs̲oftmax ˡz2

ʰa̲long : a -> b -> a

function (private) · line 53
ʰa̲long ← { v m → v 'l̲eft t̲able o̲ffsets f̲irst -1 t̲ake s̲hape m }

ˡs̲hifted : Num a => a -> a

function · line 58

Softmax's steps, to show them one at a time (ac:s_oftmax does them in one expression): rs:s_hifted m, each row less its ac:r_owMax (the largest becomes 0)

ˡs̲hifted ← { m → m − (ᵃᶜr̲owMax m) ʰa̲long m }

ˡe̲xps : Num a => a -> Float

function · line 61

rs:e_xps m: e to each of the shifted rows: all positive, the largest 1

ˡe̲xps ← { m → e̲xp ˡs̲hifted m }

ˡs̲ums : Num a => a -> Float

function · line 64

rs:s_ums m: each row's e's added up, ac:r_owSums

ˡs̲ums ← { m → ᵃᶜr̲owSums ˡe̲xps m }

ˡshares : Float

value · line 70

each row's e's divided by their sum are ac:s_oftmax's shares

      "rs:" u̲se< "ReluSoftmax"
      '+ r̲/ r̲avel a̲bs ʳˢp − (ʳˢe̲xps ʳˢz2) ÷ (ʳˢs̲ums ʳˢz2) 'l̲eft t̲able 1 2
0.0
ˡshares ← (ˡe̲xps ˡz2) ÷ (ˡs̲ums ˡz2) ʰa̲long ˡz2

Colors (0 to 255)

ˡorange : Int

value · line 75

orange: a positive value, and the first of a row's two shares

ˡorange ← 255 179 71

ˡblue : Int

value · line 77

blue: a negative value

ˡblue ← 79 168 255

ˡred : Int

value · line 79

red: a value ReLU removes

ˡred ← 255 74 61

ˡcyan : Int

value · line 81

cyan: the second of a row's two shares

ˡcyan ← 79 214 255

ˡwhite : Int

value · line 83

white: a sum, and the labels

ˡwhite ← 255 255 255

ˡyellow : Int

value · line 85

yellow: a kept value's label after ReLU, a sum's, a step's caption

ˡyellow ← 255 216 77

ˡpink : Int

value · line 87

pink: a removed value's label, and ReLU's caption

ˡpink ← 255 138 128

ˡgray : Int

value · line 89

gray: the rows' names

ˡgray ← 197 204 214

ˡglassColor : Int

value · line 91

the glass floors

ˡglassColor ← 143 211 255

ˡsky : Int

value · line 93

the sky, near black

ˡsky ← 7 8 10
Used in: ˡs̲cene

ʰp̲ick : (Num a, Num b, Num c) => a -> (b, c) -> Float

function (private) · line 96
ʰp̲ick ← { m (a, b) → ((f̲loat m) '× t̲able f̲loat a) + (1.0 − f̲loat m) '× t̲able f̲loat b }

Bars, keys and labels as tables

ˡb̲ars : Any a => (Float, Float, Float, Float) -> a -> Float

function · line 107

(x, y, heights, width) rs:b_ars 0: places for bars (n by 6: x y z w d h), each standing on the floor (z 0) or hanging below it as its height is positive or negative, never thinner than 0.004

      "rs:" u̲se< "ReluSoftmax"
      (0.0 1.0, 0.0 0.0, 2.0 -1.0, 0.5) ʳˢb̲ars 0
0.0 0.0  1.0 0.5 0.5 2.0
1.0 0.0 -0.5 0.5 0.5 1.0
ˡb̲ars ← { (bx, by, hh, w) z →
  n ← t̲ally hh
  2 1 t̲ranspose (6 c̲at n) r̲eshape bx c̲at by c̲at (hh ÷ 2.0) c̲at (n r̲eshape w) c̲at (n r̲eshape w) c̲at 0.004 m̲ax a̲bs hh
}

ˡt̲able : (Num a, Any b) => (Float, a, Float) -> b -> Float

function · line 114

(places, colors, glow) rs:t_able 0: the rows vd:b_oxes takes -- each place, then its color, then how strongly it glows

ˡt̲able ← { (pl, c, g) z → pl c̲at₂ (f̲loat c) c̲at₂ (t̲ally pl) r̲eshape g }

ˡk̲eyed : Int -> a -> a

function · line 119

a list of n by 6 (or n by 3) tables, one a key, as n by k by 6 (or 3): what vd:m_oves (vd:t_ints) takes. keys is k, and the tables are joined one under another

ˡk̲eyed ← { k tabs →
  s ← s̲hape tabs
  n ← (f̲irst s) d̲iv k
  2 1 3 t̲ranspose (k c̲at n c̲at 2 s̲elect s) r̲eshape tabs
}

ʰl̲abelAt : (Num a, Any b) => (Float, Float, Float, Float, a) -> b -> Float

function (private) · line 126
ʰl̲abelAt ← { (bx, by, bz, size, c) z →
  n ← t̲ally bx
  (2 1 t̲ranspose (3 c̲at n) r̲eshape bx c̲at by c̲at bz) c̲at₂ (n r̲eshape size) c̲at₂ ((n c̲at 3) r̲eshape f̲loat c)
}

ReLU, 0 to 11 s

hs : Float

value (private) · line 134
hs ← 2.2

gap : Float

value (private) · line 135
gap ← 1.4
Used in: ˡcellX, ˡcellY

ox : Float

value (private) · line 136
ox ← -12.0

ˡcellX : Float

value · line 139

where each cell of z1 stands: x (4 by 5), 1.4 apart, about x -12

ˡcellX ← 4 5 r̲eshape ox + gap × (f̲loat r̲ange 5) − 3.0

ˡcellY : Float

value · line 141

y (4 by 5), a row of z1 a row of the floor

ˡcellY ← (gap × f̲loat (r̲ange 4) − 1) 'l̲eft t̲able r̲ange 5

ˡr̲eluScript : Any a => a -> Char

function · line 146

the ReLU part of the script: z1 as bars on a glass floor, labeled; the negatives turn red and shrink to 0, outlines left where they were; then h's values

ˡr̲eluScript ← { z →
  v ← r̲avel ˡz1
  bx ← r̲avel ˡcellX
  by ← r̲avel ˡcellY
  neg ← v < 0.0
  n ← t̲ally v
  k ← '+ r̲/ neg
  p0 ← (bx, by, v × hs, 0.62) ˡb̲ars 0
  p1 ← (bx, by, hs × ᵃᶜr̲elu v, 0.62) ˡb̲ars 0
  c0 ← neg ʰp̲ick (ˡblue, ˡorange)
  c1 ← neg ʰp̲ick (ˡred, ˡorange)
  gv ← neg r̲eplicate v
  top ← 0.45 + hs × ᵃᶜr̲elu v
  hl ← ((bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) c̲at₂ 0.0
  marked ← (4 t̲ake₂ hl) c̲at₂ neg ʰp̲ick (ˡpink, ˡyellow)
  ((("floor", (1 6 r̲eshape ox c̲at 2.1 0.0 7.4 6.0 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at
    (("floor", 1 2 r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at
    (("z1", (p0, c0, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
    (("z1", 0.0 4.0 5.6, 3 ˡk̲eyed p0 c̲at p0 c̲at p1) ᵛᵈm̲oves 0) c̲at
    (("z1", 0.0 3.0 3.6, 3 ˡk̲eyed c0 c̲at c0 c̲at c1) ᵛᵈt̲ints 0) c̲at
    (("z1", (n c̲at 2) r̲eshape 0.0 11.0) ᵛᵈs̲hown 0) c̲at
    (("gone", ((neg r̲eplicate bx, neg r̲eplicate by, gv × hs, 0.62) ˡb̲ars 0, (k c̲at 3) r̲eshape ˡred, 1.5) ˡt̲able 0) ᵛᵈo̲utlines 0) c̲at
    (("gone", (k c̲at 2) r̲eshape 5.6 11.0) ᵛᵈs̲hown 0) c̲at
    (("z1 values", 2 ᵛᵈf̲ixed v, (bx, by − 0.6, top, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
    (("z1 values", (n c̲at 2) r̲eshape 0.5 5.0) ᵛᵈs̲hown 0) c̲at
    (("h values", 2 ᵛᵈf̲ixed ᵃᶜr̲elu v, marked) ᵛᵈl̲abels 0) c̲at
    ("h values", (n c̲at 2) r̲eshape 6.0 11.0) ᵛᵈs̲hown 0)
}
Used in: ˡs̲cene

Softmax, 11.5 to 30 s

ˡsteps : Float

value · line 179

when each step of softmax starts: the scores, less the largest, e to them, the sums, divided, the shares; the video ends at 30

ˡsteps ← 11.5 15.0 18.0 21.0 23.5 26.0

ˡends : Float

value · line 181

the video's length, 30 seconds

ˡends ← 30.0

eh : Float

value (private) · line 184
eh ← 2.0

gw : Float

value (private) · line 185
gw ← 4.4
Used in: ˡgroupX

ph : Float

value (private) · line 186
ph ← 3.2

ˡgroupX : Float

value · line 189

the middle of each row's group, left to right (4), 4.4 apart

ˡgroupX ← gw × (f̲loat r̲ange 4) − 2.5

ˡbarX : Float

value · line 192

where each score's bar stands (4 by 2): its group's middle less 0.9, then 1.2 to the right

ˡbarX ← (ˡgroupX − 0.9) '+ t̲able 0.0 1.2

ʰa̲t : Int -> Float

function (private) · line 195
ʰa̲t ← { i → i s̲elect ˡsteps }

ˡs̲oftmaxScript : Any a => a -> Char

function · line 200

the softmax part of the script: each row's two bars become the shifted scores, the e's, the shares; each row's sum stands beside them; each row's result is a column of its two shares, 1 tall

ˡs̲oftmaxScript ← { z →
  sc ← r̲avel ˡz2
  sh ← r̲avel ˡs̲hifted ˡz2
  e ← r̲avel ˡe̲xps ˡz2
  sm ← ˡs̲ums ˡz2
  pp ← r̲avel ˡp
  bx ← r̲avel ˡbarX
  n ← t̲ally bx
  by ← n r̲eshape 0.0
  ks ← (ʰa̲t 1) c̲at ((ʰa̲t 2) − 1.0) c̲at (ʰa̲t 2) c̲at ((ʰa̲t 3) − 1.0) c̲at (ʰa̲t 3) c̲at ((ʰa̲t 5) − 1.0) c̲at ʰa̲t 5
  hh ← sc c̲at sc c̲at sh c̲at sh c̲at e c̲at e c̲at pp
  pl ← ((7 × n) c̲at 6) r̲eshape (r̲avel (7 × n) r̲eshape bx, (7 × n) r̲eshape by, eh × hh, 0.8) ˡb̲ars 0
  first ← n t̲ake pl
  share ← (r̲avel 4 2 r̲eshape 1 0) ʰp̲ick (ˡorange, ˡcyan)
  cz ← (sc ≥ 0.0) ʰp̲ick (ˡorange, ˡblue)
  sx ← 1.2 + r̲avel 2 s̲elect₂ ˡbarX
  cx ← r̲avel 2 r̲eplicate 0.3 + ˡgroupX
  below ← r̲avel (4 1 r̲eshape 0.0) c̲at₂ 1 t̲ake₂ ˡp
  mid ← ph × below + pp ÷ 2.0
  segs ← 2 1 t̲ranspose (6 c̲at n) r̲eshape cx c̲at by c̲at mid c̲at (n r̲eshape 1.6) c̲at (n r̲eshape 0.9) c̲at (ph × pp) − 0.04
  gx ← 0.3 + ˡgroupX
  ((("floor2", (1 6 r̲eshape 0.6 0.0 0.0 18.2 1.8 0.02, 1 3 r̲eshape ˡglassColor, 0.6) ˡt̲able 0) ᵛᵈg̲lass 0) c̲at
    (("floor2", 1 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
    (("scores", (first, cz, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
    (("scores", ks, 7 ˡk̲eyed pl) ᵛᵈm̲oves 0) c̲at
    (("scores", (ʰa̲t 1) c̲at ((ʰa̲t 3) − 0.6) c̲at ʰa̲t 3, 3 ˡk̲eyed cz c̲at cz c̲at share) ᵛᵈt̲ints 0) c̲at
    (("scores", (n c̲at 2) r̲eshape (ʰa̲t 1) c̲at ʰa̲t 6) ᵛᵈs̲hown 0) c̲at
    ((("scores", sc, 1, 2, -0.4, bx) ʰv̲alues 0)) c̲at
    ((("shifted", sh, 2, 3, -0.4, bx) ʰv̲alues 0)) c̲at
    ((("e", e, 3, 5, -0.4, bx) ʰv̲alues 0)) c̲at
    ((("shares", pp, 5, 6, 0.0, bx) ʰv̲alues 0)) c̲at
    (("sums", ((sx, 4 r̲eshape 0.0, eh × sm, 0.8) ˡb̲ars 0, 4 3 r̲eshape ˡwhite, 1.2) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
    (("sums", 4 2 r̲eshape (ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at
    (("sum values", 2 ᵛᵈf̲ixed sm, (sx, 4 r̲eshape -0.7, 0.4 + eh × sm, 0.46, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
    (("sum values", 4 2 r̲eshape (0.3 + ʰa̲t 4) c̲at 0.8 + ʰa̲t 5) ᵛᵈs̲hown 0) c̲at
    (("result", (segs, share, 1.3) ˡt̲able 0) ᵛᵈb̲oxes 0) c̲at
    (("result", (n c̲at 2) r̲eshape (ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
    (("result values", 3 ᵛᵈf̲ixed pp, (cx, n r̲eshape -0.6, mid, 0.46, 0 0 0) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
    (("result values", (n c̲at 2) r̲eshape (0.3 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
    (("ones", "1|1|1|1", (gx, 4 r̲eshape -0.6, 4 r̲eshape ph + 0.45, 0.5, ˡyellow) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
    (("ones", 4 2 r̲eshape (0.6 + ʰa̲t 6) c̲at ˡends) ᵛᵈs̲hown 0) c̲at
    (("rows", "row 1|row 2|row 3|row 4", (gx, 4 r̲eshape -1.3, 4 r̲eshape -0.1, 0.42, ˡgray) ʰl̲abelAt 0) ᵛᵈl̲abels 0) c̲at
    ("rows", 4 2 r̲eshape (ʰa̲t 1) c̲at ˡends) ᵛᵈs̲hown 0)
}
Used in: ˡs̲cene

ʰv̲alues : Num a => (Char, Float, Int, Int, Float, Float) -> a -> Char

function (private) · line 247
ʰv̲alues ← { (name, v, i, j, less, bx) z →
  n ← t̲ally v
  t ← ((name c̲at " values"), 2 ᵛᵈf̲ixed v, (bx, n r̲eshape -0.7, 0.4 + eh × v m̲ax 0.0, 0.46, ˡwhite) ʰl̲abelAt 0) ᵛᵈl̲abels 0
  t c̲at ((name c̲at " values"), (n c̲at 2) r̲eshape (0.3 + ʰa̲t i) c̲at less + ʰa̲t j) ᵛᵈs̲hown 0
}

The captions and the camera

ʰs̲hape : Char -> a -> Char

function (private) · line 256
ʰs̲hape ← { name a → name c̲at " [" c̲at (f̲ormat s̲hape a) c̲at "]" }

ˡcaptionTexts : Char

value · line 260

the captions: what each part shows, | between them; the shapes and the count of negatives are the arrays'

ˡcaptionTexts ← (("z1" ʰs̲hape ˡz1) c̲at ": " c̲at (f̲ormat t̲ally r̲avel ˡz1) c̲at " values, " c̲at (f̲ormat '+ r̲/ (r̲avel ˡz1) < 0.0) c̲at " of them negative|" c̲at
  "relu: every negative becomes 0|" c̲at
  ("h" ʰs̲hape ˡh) c̲at ": the negatives are gone; the rest unchanged|" c̲at
  ("z2" ʰs̲hape ˡz2) c̲at ": two scores in each row|" c̲at
  "1. subtract each row's largest|2. exp: every value positive, the largest 1|3. add up each row|4. divide by the sum|" c̲at
  ("p" ʰs̲hape ˡp) c̲at ": each row shares out 1")
Used in: ˡs̲cene

ˡc̲aptionTable : Any a => a -> Float

function · line 268

when each caption shows (from, to) and its color

ˡc̲aptionTable ← { z →
  s ← ˡsteps
  from ← 0.3 3.1 7.1 c̲at (0.3 + ʰa̲t 1) c̲at 1 d̲rop s
  to ← 3.0 7.0 11.0 c̲at ((1 d̲rop s) − 0.2) c̲at ˡends
  c ← 9 3 r̲eshape ˡwhite c̲at ˡpink c̲at ˡyellow c̲at ˡwhite c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡyellow c̲at ˡwhite
  (2 1 t̲ranspose 2 9 r̲eshape from c̲at to) c̲at₂ f̲loat c
}
Used in: ˡs̲cene

ˡc̲ameraKeys : Any a => a -> (Float, Float, Float)

function · line 279

the camera: over the ReLU part, then gliding back to see the four rows of softmax side by side: at 0, 10.5, 12 and 30 s, each eye looking at its aim

ˡc̲ameraKeys ← { z →
  rc ← ox c̲at 2.1 0.6
  sc ← 0.3 0.0 1.4
  ts ← 0.0 10.5 12.0 c̲at ˡends
  eyes ← 4 3 r̲eshape (rc + -1.5 -10.5 11.5) c̲at (rc + 1.0 -11.0 11.0) c̲at (sc + -1.0 -20.5 7.0) c̲at sc + 1.0 -20.5 6.5
  aims ← 4 3 r̲eshape rc c̲at rc c̲at sc c̲at sc
  (ts, eyes, aims)
}
Used in: ˡs̲cene

The scene

ˡs̲cene : Any c => (a, b) -> c -> Char

function · line 294

(width, height) rs:s_cene 0: the whole script Blender runs, but its output: the stage (30 frames a second, 30 seconds, the sky), the ReLU part, the softmax part, the captions and the camera, and the report. The caller adds vd:r_ender or vd:s_tills.

ˡs̲cene ← { (w, h) z →
  ((ᵛᵈhead c̲at ((w, h, 30, ˡends, ˡsky, 38) ᵛᵈs̲tage 0) c̲at (ˡr̲eluScript 0) c̲at ˡs̲oftmaxScript 0) c̲at
    ((ˡcaptionTexts, ˡc̲aptionTable 0) ᵛᵈc̲aptions 0) c̲at
    (ˡc̲ameraKeys 0) ᵛᵈc̲amera 0)
}
Used in: script