NN.xtl

1⍝│ NN: neural-network building blocks -- softmax by row, dense layers, one-hot, loss, accuracy
2⍝│
3⍝│ Import it with an alias of your choice: "nn:" u_se< "NN".
4⍝│ Put libs/NN/src on XETAL_PATH ("just path"); the reference is libs/NN/docs.
5⍝│ Names with l: are exported; the others are private to this file.
6⍝│
7⍝│ A batch is a matrix, one example per row; a layer's weights are a
8⍝│ matrix with one row per input and one column per output, and its
9⍝│ bias is one more row at the bottom (so a layer is one array, and
10⍝│ x ⁿⁿd̲ense wb takes two arguments). Softmax, log-softmax and
11⍝│ argmax work along the last axis of any array: a vector is one
12⍝│ example, a matrix one per row, a rank-3 array a batch of sequences.
13⍝│ The item-by-item activations (ReLU, sigmoid, tanh, GELU, ...) are in
14⍝│ the Activations library: "act:" u_se< "Activations".
15
16⍝━━ Helpers (private)
17
18⍝│ The row values r (one per row of m) spread across m's columns.
19ʰs̲pread ← { r m → o̲\ ((-1 t̲ake s̲hape m) c̲at t̲ally r) r̲eshape r }
20
21⍝│ m as a matrix of its last-axis rows (a vector becomes one row).
22ʰr̲ows ← { m → (('× r̲/ -1 d̲rop s̲hape m) c̲at -1 t̲ake s̲hape m) r̲eshape m }
23
24⍝━━ Softmax and argmax: along the last axis
25
26⍝│ s̲oftmax m: each row into probabilities; the row's largest is taken
27⍝│ off first (the same result, no overflow), the row's sum divides.
28⍝│ >> "nn:" u_se< "NN"
29⍝│ >> nn:s_oftmax 1.0 2.0 3.0
30⍝│ 0.09003057317038046 0.24472847105479764 0.6652409557748218
31ˡs̲oftmax ← { a →
32  m ← ʰr̲ows a
33  e ← e̲xp m − ('m̲ax r̲/₂ m) ʰs̲pread m
34  (s̲hape a) r̲eshape e ÷ ('+ r̲/₂ e) ʰs̲pread e
35}
36
37⍝│ l̲ogSoftmax m: the logarithm of each row's softmax, computed stably.
38ˡl̲ogSoftmax ← { a →
39  m ← ʰr̲ows a
40  z ← m − ('m̲ax r̲/₂ m) ʰs̲pread m
41  (s̲hape a) r̲eshape z − (l̲og '+ r̲/₂ e̲xp z) ʰs̲pread z
42}
43
44⍝━━ Layers
45
46⍝│ x d̲ense wb: the layer x W + b for a batch x (one row each) and wb,
47⍝│ the weights W with the bias b as one more row at the bottom.
48⍝│ >> "nn:" u_se< "NN"
49⍝│ >> (2 2 r_eshape 1.0 2.0 3.0 4.0) nn:d_ense 3 2 r_eshape 1.0 0.0 0.0 2.0 10.0 20.0
50⍝│ 11.0 24.0
51⍝│ 13.0 28.0
52ˡd̲ense ← { x wb →
53  (x '+ '× i̲nner -1 d̲rop wb) + (o̲ffsets t̲ally x) 'r̲ight t̲able f̲irst -1 t̲ake wb
54}
55
56⍝━━ Adding a vector along an axis
57
58⍝│ (k, b) a̲ddAxis V: the vector b added to V along its axis k (every
59⍝│ element of V gets the b of its position on that axis). X_eTaL has
60⍝│ no implicit broadcasting, and the APL family disagrees on which axis
61⍝│ a vector would match (J and BQN the first, NumPy the last), so the
62⍝│ axis is written. b's length must be that axis's extent, else
63⍝│ error[shape] naming both; an axis V does not have is error[axis].
64⍝│ >> "nn:" u_se< "NN"
65⍝│ >> (2, 10.0 20.0 30.0) nn:a_ddAxis 2 3 r_eshape f_loat r_ange 6
66⍝│ 11.0 22.0 33.0
67⍝│ 14.0 25.0 36.0
68⍝│ >> (1, 10.0 20.0) nn:a_ddAxis 2 3 r_eshape f_loat r_ange 6
69⍝│ 11.0 12.0 13.0
70⍝│ 24.0 25.0 26.0
71ˡa̲ddAxis ← { (k, b) V →
72  s ← s̲hape V
73  _ ← (k, b) ʰa̲xisChecked s
74  r ← t̲ally s
75  rest ← ((k − 1) t̲ake s) c̲at k d̲rop s
76  B ← b '+ t̲able rest r̲eshape (f̲irst b) − f̲irst b
77  j ← r̲ange r − 1
78  perm ← k c̲at j + 0 + j ≥ k
79  V + perm t̲ranspose B
80}
81
82⍝│ Whether axis k of an array of shape s exists and b fits it; else the
83⍝│ error, naming the axis and both lengths.
84ʰa̲xisChecked ← { (k, b) s →
85  r ← t̲ally s
86  n̲ot (k ≥ 1) ∧ k ≤ r ? "axis" ⎕S̲IGNAL "a_ddAxis: axis " c̲at (f̲ormat k) c̲at " of a rank " c̲at (f̲ormat r) c̲at " array"
87  (t̲ally b) ≠ k s̲elect s ? "shape" ⎕S̲IGNAL "a_ddAxis: " c̲at (f̲ormat t̲ally b) c̲at " values for axis " c̲at (f̲ormat k) c̲at " of extent " c̲at f̲ormat k s̲elect s
88  ""
89}
90
91⍝│ b b̲ias V: the bias vector b added where a layer's bias goes, by
92⍝│ V's rank: a batch of rows or one row (rank 2 or 1) on the last axis,
93⍝│ one feature per column, as d_ense keeps it; a channel-first picture
94⍝│ (rank 3: channels, height, width, as Conv's maps for one picture) on
95⍝│ the first axis; a batch of them (rank 4) on the second. Any other
96⍝│ axis is written out with a_ddAxis.
97⍝│ >> "nn:" u_se< "NN"
98⍝│ >> 10.0 20.0 30.0 nn:b_ias 2 3 r_eshape f_loat r_ange 6
99⍝│ 11.0 22.0 33.0
100⍝│ 14.0 25.0 36.0
101⍝│ >> r_avel 1.0 2.0 nn:b_ias 2 2 2 r_eshape 0.0
102⍝│ 1.0 1.0 1.0 1.0 2.0 2.0 2.0 2.0
103ˡb̲ias ← { b V →
104  r ← t̲ally s̲hape V
105  r ≤ 2 ? (r, b) ˡa̲ddAxis V
106  r = 3 ? (1, b) ˡa̲ddAxis V
107  (2, b) ˡa̲ddAxis V
108}
109
110⍝━━ Labels
111
112⍝│ a̲rgmax a: for each row (along the last axis), the position (from
113⍝│ 1) of its largest item (the first, on a tie); a vector gives one Int.
114⍝│ >> "nn:" u_se< "NN"
115⍝│ >> nn:a_rgmax 2 3 r_eshape 1.0 2.0 3.0 -1.0 0.0 5.0
116⍝│ 3 3
117ˡa̲rgmax ← { a →
118  m ← ʰr̲ows a
119  n ← f̲irst -1 t̲ake s̲hape m
120  hit ← f̲loat m = ('m̲ax r̲/₂ m) ʰs̲pread m
121  (-1 d̲rop s̲hape a) r̲eshape f̲loor 'm̲in r̲/₂ (hit × (o̲ffsets t̲ally m) 'r̲ight t̲able f̲loat r̲ange n) + (1.0 − hit) × f̲loat n + 1
122}
123
124⍝│ k o̲neHot y: the labels y (1 to k) as rows of k Floats, 1.0 at the label.
125⍝│ >> "nn:" u_se< "NN"
126⍝│ >> 3 nn:o_neHot 2 1 3
127⍝│ 0.0 1.0 0.0
128⍝│ 1.0 0.0 0.0
129⍝│ 0.0 0.0 1.0
130ˡo̲neHot ← { k y → f̲loat y '= t̲able r̲ange k }
131
132⍝━━ Loss and accuracy
133
134⍝│ y c̲rossEntropy p: the mean over rows of -sum(y log p), for one-hot
135⍝│ (or probability) rows y and predicted probabilities p; p is clamped
136⍝│ at 1e-12 first, so a probability of 0 costs 27.6, not a log error.
137⍝│ >> "nn:" u_se< "NN"
138⍝│ >> (2 nn:o_neHot 1 2) nn:c_rossEntropy 2 2 r_eshape 0.9 0.1 0.2 0.8
139⍝│ 0.164252033486018
140ˡc̲rossEntropy ← { y p → ('+ r̲/ '+ r̲/₂ n̲eg y × l̲og p m̲ax 1e-12) ÷ f̲loat t̲ally p }
141
142⍝│ y m̲se p: the mean squared error over every item.
143⍝│ >> "nn:" u_se< "NN"
144⍝│ >> 1.0 2.0 nn:m_se 1.0 4.0
145⍝│ 2.0
146ˡm̲se ← { y p → ('+ r̲/ r̲avel (y − p) ^ 2) ÷ f̲loat t̲ally r̲avel p }
147
148⍝│ y a̲ccuracy p: the fraction of rows of p whose argmax is the label y.
149⍝│ >> "nn:" u_se< "NN"
150⍝│ >> 3 2 nn:a_ccuracy 2 3 r_eshape 1.0 2.0 3.0 -1.0 0.0 5.0
151⍝│ 0.5
152ˡa̲ccuracy ← { y p → ('+ r̲/ f̲loat y = ˡa̲rgmax p) ÷ f̲loat t̲ally y }