librarywork/xetal-ml/libs/NN/src/NN.xtl
NN: neural-network building blocks -- softmax by row, dense layers, one-hot, loss, accuracy
Import it with an alias of your choice: "nn:" u_se< "NN". Put libs/NN/src on XETAL_PATH ("just path"); the reference is libs/NN/docs. Names with l: are exported; the others are private to this file.
A batch is a matrix, one example per row; a layer's weights are a
matrix with one row per input and one column per output, and its
bias is one more row at the bottom (so a layer is one array, and
x ⁿⁿd̲ense wb takes two arguments). Softmax, log-softmax and
argmax work along the last axis of any array: a vector is one
example, a matrix one per row, a rank-3 array a batch of sequences.
The item-by-item activations (ReLU, sigmoid, tanh, GELU, ...) are in
the Activations library: "act:" u_se< "Activations".
Helpers (private)
ʰs̲pread : a -> b -> a
The row values r (one per row of m) spread across m's columns.
ʰs̲pread ← { r m → o̲\ ((-1 t̲ake s̲hape m) c̲at t̲ally r) r̲eshape r }
ʰr̲ows : a -> a
m as a matrix of its last-axis rows (a vector becomes one row).
ʰr̲ows ← { m → (('× r̲/ -1 d̲rop s̲hape m) c̲at -1 t̲ake s̲hape m) r̲eshape m }
Softmax and argmax: along the last axis
ˡs̲oftmax : Num a => a -> Float
s̲oftmax m: each row into probabilities; the row's largest is taken
off first (the same result, no overflow), the row's sum divides.
"nn:" u̲se< "NN"
ⁿⁿs̲oftmax 1.0 2.0 3.0 0.09003057317038046 0.24472847105479764 0.6652409557748218
ˡs̲oftmax ← { a → m ← ʰr̲ows a e ← e̲xp m − ('m̲ax r̲/₂ m) ʰs̲pread m (s̲hape a) r̲eshape e ÷ ('+ r̲/₂ e) ʰs̲pread e }
ˡl̲ogSoftmax : Float -> Float
l̲ogSoftmax m: the logarithm of each row's softmax, computed stably.
ˡl̲ogSoftmax ← { a → m ← ʰr̲ows a z ← m − ('m̲ax r̲/₂ m) ʰs̲pread m (s̲hape a) r̲eshape z − (l̲og '+ r̲/₂ e̲xp z) ʰs̲pread z }
Layers
ˡd̲ense : Num a => a -> a -> a
x d̲ense wb: the layer x W + b for a batch x (one row each) and wb,
the weights W with the bias b as one more row at the bottom.
"nn:" u̲se< "NN"
(2 2 r̲eshape 1.0 2.0 3.0 4.0) ⁿⁿd̲ense 3 2 r̲eshape 1.0 0.0 0.0 2.0 10.0 20.0 11.0 24.0 13.0 28.0
ˡd̲ense ← { x wb → (x '+ '× i̲nner -1 d̲rop wb) + (o̲ffsets t̲ally x) 'r̲ight t̲able f̲irst -1 t̲ake wb }
Adding a vector along an axis
ˡa̲ddAxis : Num a => (Int, a) -> a -> a
(k, b) a̲ddAxis V: the vector b added to V along its axis k (every
element of V gets the b of its position on that axis). X_eTaL has
no implicit broadcasting, and the APL family disagrees on which axis
a vector would match (J and BQN the first, NumPy the last), so the
axis is written. b's length must be that axis's extent, else
error[shape] naming both; an axis V does not have is error[axis].
"nn:" u̲se< "NN"
(2, 10.0 20.0 30.0) ⁿⁿa̲ddAxis 2 3 r̲eshape f̲loat r̲ange 6 11.0 22.0 33.0 14.0 25.0 36.0
(1, 10.0 20.0) ⁿⁿa̲ddAxis 2 3 r̲eshape f̲loat r̲ange 6 11.0 12.0 13.0 24.0 25.0 26.0
ˡa̲ddAxis ← { (k, b) V → s ← s̲hape V _ ← (k, b) ʰa̲xisChecked s r ← t̲ally s rest ← ((k − 1) t̲ake s) c̲at k d̲rop s B ← b '+ t̲able rest r̲eshape (f̲irst b) − f̲irst b j ← r̲ange r − 1 perm ← k c̲at j + 0 + j ≥ k V + perm t̲ranspose B }
ʰa̲xisChecked : Num a => (Int, a) -> Int -> Char
Whether axis k of an array of shape s exists and b fits it; else the error, naming the axis and both lengths.
ʰa̲xisChecked ← { (k, b) s → r ← t̲ally s n̲ot (k ≥ 1) ∧ k ≤ r ? "axis" ⎕S̲IGNAL "a_ddAxis: axis " c̲at (f̲ormat k) c̲at " of a rank " c̲at (f̲ormat r) c̲at " array" (t̲ally b) ≠ k s̲elect s ? "shape" ⎕S̲IGNAL "a_ddAxis: " c̲at (f̲ormat t̲ally b) c̲at " values for axis " c̲at (f̲ormat k) c̲at " of extent " c̲at f̲ormat k s̲elect s "" }
ˡb̲ias : Num a => a -> a -> a
b b̲ias V: the bias vector b added where a layer's bias goes, by
V's rank: a batch of rows or one row (rank 2 or 1) on the last axis,
one feature per column, as d_ense keeps it; a channel-first picture
(rank 3: channels, height, width, as Conv's maps for one picture) on
the first axis; a batch of them (rank 4) on the second. Any other
axis is written out with a_ddAxis.
"nn:" u̲se< "NN"
10.0 20.0 30.0 ⁿⁿb̲ias 2 3 r̲eshape f̲loat r̲ange 6 11.0 22.0 33.0 14.0 25.0 36.0
r̲avel 1.0 2.0 ⁿⁿb̲ias 2 2 2 r̲eshape 0.0 1.0 1.0 1.0 1.0 2.0 2.0 2.0 2.0
ˡb̲ias ← { b V → r ← t̲ally s̲hape V r ≤ 2 ? (r, b) ˡa̲ddAxis V r = 3 ? (1, b) ˡa̲ddAxis V (2, b) ˡa̲ddAxis V }
Labels
ˡa̲rgmax : Num a => a -> Int
a̲rgmax a: for each row (along the last axis), the position (from
1) of its largest item (the first, on a tie); a vector gives one Int.
"nn:" u̲se< "NN"
ⁿⁿa̲rgmax 2 3 r̲eshape 1.0 2.0 3.0 -1.0 0.0 5.0 3 3
ˡa̲rgmax ← { a → m ← ʰr̲ows a n ← f̲irst -1 t̲ake s̲hape m hit ← f̲loat m = ('m̲ax r̲/₂ m) ʰs̲pread m (-1 d̲rop s̲hape a) r̲eshape f̲loor 'm̲in r̲/₂ (hit × (o̲ffsets t̲ally m) 'r̲ight t̲able f̲loat r̲ange n) + (1.0 − hit) × f̲loat n + 1 }
ˡo̲neHot : Int -> Int -> Float
k o̲neHot y: the labels y (1 to k) as rows of k Floats, 1.0 at the label.
"nn:" u̲se< "NN"
3 ⁿⁿo̲neHot 2 1 3 0.0 1.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0
ˡo̲neHot ← { k y → f̲loat y '= t̲able r̲ange k }
Loss and accuracy
ˡc̲rossEntropy : Float -> Float -> Float
y c̲rossEntropy p: the mean over rows of -sum(y log p), for one-hot
(or probability) rows y and predicted probabilities p; p is clamped
at 1e-12 first, so a probability of 0 costs 27.6, not a log error.
"nn:" u̲se< "NN"
(2 ⁿⁿo̲neHot 1 2) ⁿⁿc̲rossEntropy 2 2 r̲eshape 0.9 0.1 0.2 0.8 0.164252033486018
ˡc̲rossEntropy ← { y p → ('+ r̲/ '+ r̲/₂ n̲eg y × l̲og p m̲ax 1e-12) ÷ f̲loat t̲ally p }