X_eTaL documentation

Files

• library libs/NN/src/NN.xtl: NN: neural-network building blocks -- activations, softmax by row, dense layers, one-hot, loss, accuracy
• macro library libs/Net/src/Net.xtlm: Net: a network written as one line -- its layer sizes and activations become ordinary NN calls
• program libs/Net/demos/train-xor.xtl: XOR learned, the network and its training each written as one line: t_rain< writes the training step (backpropagation and Adam) from the same spec n_etwork< reads. Small random weights at the start. Run with "just demo-lib Net train-xor"; see what was written with "just expand-lib Net train-xor".
• program libs/Net/demos/xor.xtl: XOR, the network written as one line. NN's own demo builds the same two layers by hand; here the spec "2 2 relu 2 softmax" is the network, and xetal expand shows the function the macro wrote. Run with "just demo-lib Net xor"; see it with "just expand-lib Net xor".
• program libs/NN/demos/xor.xtl: XOR by a two-layer network with hand-set weights: no straight line separates XOR's classes, but one hidden ReLU layer does. Every layer is one nn:d_ense on the whole batch of four points at once. Run with "just demo-lib NN xor".
• program demos/attention/attention.xtl: Attention as array algebra: every token's query meets every token's key in one matrix product, S = Q K^T / sqrt d; softmax turns each row into weights; the output is the weighted sum of the values, Y = A V. One head, hand-set so that a word looks for what it can describe.
• program demos/backprop/backprop.xtl: One step of training, every array shown: a small network (2 inputs, 4 tanh units, softmax over 3 classes) on six points of a spiral. The forward pass, the loss, each layer's gradient as one expression, the step; then every gradient checked against a finite difference.
• program demos/cnn-digits/cnn-digits.xtl: A tiny convolutional network reads a handwritten digit: 28 x 28 -> 8 filters of 3 x 3 -> ReLU -> 2 x 2 max-pooling -> dense -> softmax. The convolution is the picture's nine shifted copies times the filters: one matrix product. Trained offline on MNIST (97.82% of the 10,000 test digits right); just cnn-train writes data/.
• program demos/moe-router/moe-router.xtl: A mixture-of-experts router: each token of a sentence is scored against 16 experts by one matrix product, the scores become probabilities (softmax), each token keeps its two most likely experts (top-2), and their probabilities, renormalized, are the gates. The experts' load is how many tokens each one got.
• program demos/net-macro/net-macro.xtl: Three networks for one task (which of three spiral arms is a point on?), each described once, by one line: the Net macro library turns a spec, sizes and activations, into the loading of the weights and an ordinary function of NN calls before the program is compiled. xetal expand shows what it wrote.
• program demos/net-macro/train-it.xtl: Training a network from its one-line spec: the Net macro library writes, from "2 16 relu 16 relu 3 softmax", the network's backward pass and an Adam step (net:t_rain<) and the starting state (net:s_tate<); p_ower repeats the step. The task is net-macro's: which of three spiral arms is a point on? Here the 300 training points are made in X_eTaL, and the weights start small and random.
• program demos/ternary-net/ternary-net.xtl: A 1.58-bit network: a tiny classifier (which of three spiral arms a point is on, 2 -> 16 -> 16 -> 3) run with its weights stored four ways: 32-bit and 16-bit floats (simulated by rounding), 8-bit integers, and ternary -1 / 0 / +1 (log2 3 = 1.58 bits each), where a layer is additions and subtractions of its inputs and one scale. The same points go through all four; the measures compare them.
• program demos/train-live/train-live.xtl: Training in X_eTaL: a network (2 inputs, 16 tanh units, softmax over 3) learns which of three spiral arms a point is on. The points are made here; the backward pass is the backprop microscope's; Adam, written out, moves the weights; p_ower repeats the step.

Items