sourcelibs/Flow/demos/mlp.xtl

1⍝!/usr/bin/env xetal 2⍝ Flow: a two-layer network and a stable softmax written as 3⍝ pipelines, each beside the same computation by hand. Run with 4⍝ "just demo Flow"; "just expand-lib Flow mlp" shows what each 5⍝ call became. 6"nn:" u̲se< "NN" 7"act:" u̲se< "Activations" 8"flow:" u̲se< "Flow" 9⍝ Four points of XOR, a 2-3-2 network set by hand: a dense layer's 10⍝ weights carry the bias as their last row (NN). The hidden units are 11⍝ relu(a + b - 0.5) and relu(a + b - 1.5), the third unused; their 12⍝ difference u1 - 3 u2 is 0.5 exactly at the two XOR points, and the 13⍝ output layer turns that into the class. 14X ← 4 2 r̲eshape 0.0 0.0 0.0 1.0 1.0 0.0 1.0 1.0 15W1 ← 3 3 r̲eshape 1.0 1.0 0.0 1.0 1.0 0.0 -0.5 -1.5 0.0 16W2 ← 4 2 r̲eshape -8.0 8.0 24.0 -24.0 0.0 0.0 2.0 -2.0 17⍝ The network as a pipeline: the data first, then each layer in turn. 18⍝ A function stage (act:r_elu, nn:s_oftmax) is applied to the previous 19⍝ value; a stage naming it (it nn:d_ense W1) says where it goes. 20Y ← @ ᶠˡᵒʷp̲ipe< c"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g1:stage ← X
  g2:stage ← { @ →
    it ← g1:stage
    it ⁿⁿd̲ense W1
  } @
  g3:stage ← ᵃᶜᵗr̲elu g2:stage
  g4:stage ← { @ →
    it ← g3:stage
    it ⁿⁿd̲ense W2
  } @
  g5:stage ← ⁿⁿs̲oftmax g4:stage
  g5:stage
} @)
21 X 22▷ it ⁿⁿd̲ense W1 23▷ ᵃᶜᵗr̲elu 24▷ it ⁿⁿd̲ense W2 25▷ ⁿⁿs̲oftmax" 26Y 27⍝ The same by hand, right to left. 28ⁿⁿs̲oftmax (ᵃᶜᵗr̲elu X ⁿⁿd̲ense W1) ⁿⁿd̲ense W2 29⍝ Which class each point gets: 1 for (0, 0) and (1, 1), 2 for the 30⍝ other two: XOR. 31ⁿⁿa̲rgmax Y 32⍝ A stable softmax over a vector, with its intermediates named: the 33⍝ maximum m, the exponentials e; a later stage uses a name directly. 34z ← f̲loat 1 2 3 35@ ᶠˡᵒʷp̲ipe< c"z ▷ 'm̲ax r̲/ it ▹m ▷ e̲xp z − m ▹e ▷ e ÷ '+ r̲/ e"
ᶠˡᵒʷp̲ipe< expands to
{ @ →
  g6:stage ← z
  m ← { @ →
    it ← g6:stage
    'm̲ax r̲/ it
  } @
  e ← e̲xp z − m
  g7:stage ← e ÷ '+ r̲/ e
  g7:stage
} @
36ⁿⁿs̲oftmax z 37⍝ A shape contract: the hidden layer's output is 4 by 3, said and checked. 38H ← @ ᶠˡᵒʷp̲ipe< c"X ▷ ᵃᶜᵗr̲elu it ⁿⁿd̲ense W1 ▹h [4 3] ▷ h"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
  g8:stage ← X
  h ← { @ →
    it ← g8:stage
    ᵃᶜᵗr̲elu it ⁿⁿd̲ense W1
  } @
  _ ← { @ →
    ((s̲hape h) m̲atch r̲avel 4 c̲at 3) ? @
    "shape" ⎕S̲IGNAL "stage 2 (h): shape " c̲at (f̲ormat s̲hape h) c̲at ", expected " c̲at f̲ormat r̲avel 4 c̲at 3
  } @
  g9:stage ← h
  g9:stage
} @)
39s̲hape H