sourcelibs/Flow/demos/mlp.xtl
1⍝!/usr/bin/env xetal
2⍝ Flow: a two-layer network and a stable softmax written as
3⍝ pipelines, each beside the same computation by hand. Run with
4⍝ "just demo Flow"; "just expand-lib Flow mlp" shows what each
5⍝ call became.
6"nn:" u̲se< "NN"
7"act:" u̲se< "Activations"
8"flow:" u̲se< "Flow"
9⍝ Four points of XOR, a 2-3-2 network set by hand: a dense layer's
10⍝ weights carry the bias as their last row (NN). The hidden units are
11⍝ relu(a + b - 0.5) and relu(a + b - 1.5), the third unused; their
12⍝ difference u1 - 3 u2 is 0.5 exactly at the two XOR points, and the
13⍝ output layer turns that into the class.
14X ← 4 2 r̲eshape 0.0 0.0 0.0 1.0 1.0 0.0 1.0 1.0
15W1 ← 3 3 r̲eshape 1.0 1.0 0.0 1.0 1.0 0.0 -0.5 -1.5 0.0
16W2 ← 4 2 r̲eshape -8.0 8.0 24.0 -24.0 0.0 0.0 2.0 -2.0
17⍝ The network as a pipeline: the data first, then each layer in turn.
18⍝ A function stage (act:r_elu, nn:s_oftmax) is applied to the previous
19⍝ value; a stage naming it (it nn:d_ense W1) says where it goes.
20Y ← @ ᶠˡᵒʷp̲ipe< c"
21 X
22▷ it ⁿⁿd̲ense W1
23▷ ᵃᶜᵗr̲elu
24▷ it ⁿⁿd̲ense W2
25▷ ⁿⁿs̲oftmax"
26Y
27⍝ The same by hand, right to left.
28ⁿⁿs̲oftmax (ᵃᶜᵗr̲elu X ⁿⁿd̲ense W1) ⁿⁿd̲ense W2
29⍝ Which class each point gets: 1 for (0, 0) and (1, 1), 2 for the
30⍝ other two: XOR.
31ⁿⁿa̲rgmax Y
32⍝ A stable softmax over a vector, with its intermediates named: the
33⍝ maximum m, the exponentials e; a later stage uses a name directly.
34z ← f̲loat 1 2 3
35@ ᶠˡᵒʷp̲ipe< c"z ▷ 'm̲ax r̲/ it ▹m ▷ e̲xp z − m ▹e ▷ e ÷ '+ r̲/ e"
36ⁿⁿs̲oftmax z
37⍝ A shape contract: the hidden layer's output is 4 by 3, said and checked.
38H ← @ ᶠˡᵒʷp̲ipe< c"X ▷ ᵃᶜᵗr̲elu it ⁿⁿd̲ense W1 ▹h [4 3] ▷ h"
39s̲hape H
ᶠˡᵒʷp̲ipe< expands to
({ @ →
g1:stage ← X
g2:stage ← { @ →
it ← g1:stage
it ⁿⁿd̲ense W1
} @
g3:stage ← ᵃᶜᵗr̲elu g2:stage
g4:stage ← { @ →
it ← g3:stage
it ⁿⁿd̲ense W2
} @
g5:stage ← ⁿⁿs̲oftmax g4:stage
g5:stage
} @)ᶠˡᵒʷp̲ipe< expands to
{ @ →
g6:stage ← z
m ← { @ →
it ← g6:stage
'm̲ax r̲/ it
} @
e ← e̲xp z − m
g7:stage ← e ÷ '+ r̲/ e
g7:stage
} @ᶠˡᵒʷp̲ipe< expands to
({ @ →
g8:stage ← X
h ← { @ →
it ← g8:stage
ᵃᶜᵗr̲elu it ⁿⁿd̲ense W1
} @
_ ← { @ →
((s̲hape h) m̲atch r̲avel 4 c̲at 3) ? @
"shape" ⎕S̲IGNAL "stage 2 (h): shape " c̲at (f̲ormat s̲hape h) c̲at ", expected " c̲at f̲ormat r̲avel 4 c̲at 3
} @
g9:stage ← h
g9:stage
} @)