Live
Tiny CNN
A tiny convolutional network trained on MNIST reads a handwritten digit: convolution as the picture's nine shifted copies times eight filters, ReLU, 2 x 2 max-pooling by reshape, a dense layer and softmax, every stage an array program.
-1 0 1 o̲-₂ -1 0 1 o̲-₂ x
Every 3 x 3 window of the picture at once: the picture rotated by all nine offsets, then one matrix product with all eight filters, where other code needs four nested loops.
windows by rotationinner productreshape for poolingsoftmax
Live
Activation functions
Every activation of the Activations library and its slope, graphed over -6..6 by X_eTaL with X_eTaL-libraries' Plot, a family at a time: classical, rectifiers, smooth, gated. Move a slider to change a parameter; write an activation of your own and see it beside a built-in.
ᵘs̲lope ← { f̲ x → 5000.0 × (f̲ x + 0.0001) − f̲ x − 0.0001 }
A slope by central differences for any function: the function is the left argument (a name ending in _), so one line serves every gate and the reader's own activation.
activation functionsderivativesfunction plots with a real x axisa function passed to a function
Live
Backprop microscope
One step of training a small network, every array shown: the forward pass, the loss, each layer's gradient as one expression, the step. Each of the 27 gradients is checked by nudging its weight and watching the loss. Take steps and watch the loss fall.
G1 ← (o̲\ ᵘo̲nes X) '+ '× i̲nner D1
A layer's gradient is its input transposed times what comes back from the layer above: one matrix product, for every weight and every example at once, where other code loops over each weight's contribution.
backpropagationmatrix product with a transposefinite differencesgradient descent
Live
LoRA explorer
Low-rank adaptation of one 8 x 8 projection: choose the rank, see the adapted layer y = W x + s B (A x) as a pipeline of equations drawn as math and each stage's arrays, a backward step through it stage by stage, the update s B A and its rank, the merge, what adapters cost on a large mixture-of-experts model, and LoRA beside LoRA-FA trained to fit a known update against the best possible at each rank (the truncated SVD), W in 4 bits as QLoRA stores it, uniform and NF4, QLoRA's adapter trained on it, its merge not exact, and VeRA, two vectors on frozen random projections shared by every layer, forward, back and trained beside them.
ˡf̲orward ← { (W, A, B, s) x → (W '+ '× i̲nner x) + s × B '+ '× i̲nner A '+ '× i̲nner x }
The adapted layer in one line: right to left, A x first, then B of that, so the m x n matrix B A is never made.
LoRAlow-rank updatesa pipeline of equationsrank by Gram-Schmidttuples as argumentsgradients written as equationsLoRA-FAEckart-YoungNF4QLoRAVeRA
Live
MoE fine-tuning
A tiny mixture-of-experts, trained in X_eTaL, fine-tuned to a changed task by each method in turn, in your browser: the shared layer, every expert, both, selected experts, the router, a new expert. Is the new task learned and the old one kept? Only a new expert with its own router row keeps the old task within 3 points. Then four adapters compared at equal ranks and budgets: shared LoRA, expert LoRA, LoRA-FA and VeRA with its projections shared by the experts. Then deployed: LoRA merges exactly and QLoRA does not, two tasks switch adapters on one base, and an expert cache shows keeping adapters apart moving less. Last, SFT then DPO: pairs of two valid answers teach which is better, pairs of right and wrong only what is right.
(method, tokens, 120) ᵘt̲une seed
One function fine-tunes the base by any method: a method is a flag per part, and a frozen part's gradient is multiplied by 0, so the same step trains any of them.
inner productsoftmax by columntop-k by masksp_ower for traininggradients masked by a flag per partreading data filesVeRA as LoRA's factorsa least-recently-used cache by p_owerDPO's gradient at the softmax's input
Live
Training live
X_eTaL trains a small network in your browser: 300 points of a three-arm spiral made in X_eTaL, the backprop microscope's gradient, Adam written out, 25 steps a frame. Watch the decision regions bend to follow the spiral and the loss fall.
s ← 25 'ᵘa̲dam p̲ower s
Training is a function iterated: one Adam step maps the state (the weights and two running averages) to the next, and p_ower repeats it, with no loop written; every step updates all 99 weights at once.
trainingAdamp_ower for iterationbackpropagation
Live
CNN training
X_eTaL trains a tiny convolutional network in your browser, from random weights, on 600 handwritten digits: the backward pass through softmax, the dense layer, max-pooling, ReLU and the convolution, checked against finite differences; Adam, 20 digits a step. Watch the filters form and the test digits come to be read right, from 12% to about 90% in 40 steps.
DC ← (A ᶜᵛu̲npool DQ) × f̲loat C > 0.0
Back through max-pooling and ReLU for a whole batch at once: each 2 x 2 block's gradient goes to its first largest item, and only where the convolution's output was positive; then one matrix product with the same windows gives every filter's gradient.
trainingbackpropagationconvolutionmax-poolingtuples
Live
Attention microscope
One head of attention as array algebra: every query against every key in one matrix product with a transpose, softmax by row, the output as a weighted mix. The head is set by hand so a word looks for what it can describe: tired finds the animal, wide the street. A causal mask; click a word to see its query against every key.
ᵘs̲cores ← { X → ((X '+ '× i̲nner Wq) '+ '× i̲nner o̲\ X '+ '× i̲nner Wk) ÷ 2.0 ^ 0.5 }
Every word's query against every word's key is one matrix product with a transpose; softmax by row and the weighted mix follow, with no loop over words.
matrix producttransposesoftmax by rowmasks
Live
Tiny GPT
Karpathy's microgpt in X_eTaL: one transformer layer, 16 numbers wide, 4 attention heads, 4,192 weights trained on 32,033 baby names, inventing new names a letter at a time in your browser. Set the temperature and the first letters; look into a name: the five likeliest next letters at each step, and where each head looks.
x1 ← x0 + heads ᵘl̲inear Wo
The whole name goes through attention at once: every position's four heads computed together under a causal mask, joined and projected, then added back to what came in. The same scores microgpt computes one token at a time with its cache.
transformerattention headsRMS normsamplingcausal mask
Live
Tiny BERT
The encoder half of a transformer in X_eTaL: 2 layers, 16 numbers wide, 2 attention heads, 7,594 weights trained to fill a blank letter in baby names, each letter looking both ways. Type a name with a blank and see the letters it would put there and where each head looks; one encoder layer is a pipeline applied twice, and the model is drawn in one line of math.
(hL, _) ← L 'ᵘl̲ayer p̲ower (ᵘe̲mbed ids, 1)
BERT is L copies of one layer: the layer is a pipeline (attention, added back and normed, the GELU feed-forward layer, added back and normed), and p_ower applies it L times, carrying the layer number to pick each layer's weights.
transformer encodermasked language modelattention both wayslayer normGELUpipelinesp_ower over layersequations as macros
Live
MoE routing microscope
A mixture-of-experts router: a sentence's tokens scored against 16 experts by one matrix product, softmax, top-2 by masks, gates and the experts' load; words route by meaning. Nudge a token towards another word and watch the experts switch abruptly where two scores tie.
gates ← ᵘt̲op2 ᵘs̲oftmax ᵘs̲cores x
Every token against every expert is one matrix product, softmax works on every row at once, and top-2 is two masks over the whole score matrix: no sort per token.
inner productsoftmax by rowtop-k by masksreduceouter product for a path and a plane of inputs
Live
1.58-bit network
One tiny classifier run with its weights stored four ways (FP32, FP16, INT8, ternary -1/0/+1) and compared: maps, accuracy, error, storage, operations. A ternary layer is additions and subtractions only; ternary works when the network was trained for it.
(f̲loat x > s) − f̲loat x < n̲eg s
Quantizing a whole layer to -1, 0 and +1 is one comparison over the weight array, not a loop over weights; a ternary layer then needs only additions and subtractions.
inner productquantization as array expressionsreduce over two axesmasks
Live
k-means
Clustering without labels, step by step: 300 points in five blobs, every point given its nearest center, each center moved to the mean of its points. From a poor start three centers crowd one blob and k-means settles there; from a farthest-first start they find all five. Change k and the start.
(((o̲\ O) '+ '× i̲nner X) + C × f̲loat n = 0.0) ÷ 1.0 m̲ax n
A whole step's new centers at once: O is each point's nearest center as one-hot rows, so O's columns times the points sum each cluster's points; divided by each cluster's count, a center with no points kept where it was.
k-meansone-hot as a matrixp_owerthe Learn library
Live
Embedding explorer
Word embeddings made from counts, in X_eTaL: 4,000 short sentences over 70 words, each word's embedding 64 numbers from which words occur near it. Turn the cloud in 3-D (Learn's PCA) and click a word for its nearest: the animals, foods, colors, numbers and places gather, and k-means finds them without being told.
E ← ᵉᵐu̲nit 64 ᵉᵐp̲roject ᵉᵐp̲pmi (v, 2) ᵉᵐc̲ooccur ids
A whole embedding table in one line read right to left: count which words occur near which, weight the counts by how much more often than chance, project every row to 64 numbers, scale to length 1. No loop over words or sentences.
embeddingsco-occurrencePPMIPCAsorting to count
Live
Network macro
A network written as one line: a macro library reads a spec such as "2 16 relu 16 relu 3 softmax" when the program is compiled and writes the forward function in its place, as ordinary code you can read. Three networks on one task, a spiral: a line, a small network, a deep one. Type a spec of your own and see what the macro writes, or refuses; then train it in your browser from random weights, by the training step the macro writes from the same spec.
"u:d_eep c" ⁿᵉᵗm̲odel< "2 16 relu 16 relu 3 softmax"
The line is a macro call: before the program is compiled it becomes the loading of three weight files, shaped and checked as the spec says, and the forward function written out, three matrix products and their activations, type-checked like any code. The spec is the one place the network is described.
macro libraries (.xtlm)xetal expandmatrix productsoftmax by rowbackpropagationp_ower
Live
World model
A tiny world model: a network that has watched a 2 x 2 ball bounce in a 12 x 12 box predicts the next frame from the last three, 432 pixels in and 144 out, trained offline on four fifths of the world's 1,936 states. It gets 1,897 next frames exactly, 348 of the 387 it never saw. Place the ball, pick its speed, and let it dream: fed its own predictions, it plays the ball forward, bounces included.
ᵘp̲redict ← { X → ᵃᶜᵗs̲igmoid (ᵃᶜᵗr̲elu X ⁿⁿd̲ense wb1) ⁿⁿd̲ense wb2 }
The whole world at once: X is 1,936 windows, one row per state, three frames side by side; two dense layers give every state's next frame together, a probability per pixel, and one count says how many are exactly right.
world modelnext-frame predictiona batch through dense layersdreaming on its own output
Live
Pipelines
Layers written as data flow, left to right: a dense layer with its bias, a stable softmax, layer norm and one attention head as Flow pipelines on this repository's libraries, each beside what the macro wrote and the same layer in plain X_eTaL. A CNN layer and a latent diffusion sampler are each shown four ways, as a paper prints them (equations, drawn as math; the sampler's sum from T down to 1 is a recurrence, one step applied T times), as a pipeline, as its expansion and as plain X_eTaL; all compute the same values, checked as the page runs. Every example is also a pipeline of equations, drawn as math a row a stage, an attention head and a BERT layer among them. Compose your own from a palette of the notations' parts and see it run, drawn as math and expanded.
@ ᶠˡᵒʷp̲ipe< c"z ▷ 'm̲ax r̲/ it ▹m ▷ e̲xp z − m ▹e ▷ e ÷ '+ r̲/ e"
A stable softmax as data flow: each stage takes the one before as it, a label names a stage's value for later stages, and the macro writes it out as ordinary code, a line per stage, before the program is compiled.
macro libraries (.xtlm)code stringsxetal expand|> and $ labelsequations as macrosconvolutiondiffusion samplinga palette of the notations' partspipelines of equations, drawn as math
Live
Learned denoiser
Pictures from noise, step by step: a small network trained to see the clean picture in a noisy one at ten levels of noise, run backward from pure noise (DDIM: no new noise, so the same noise always gives the same picture). Eight 10 x 10 pictures at a time, every level of the way shown; the shapes it learned are a disk, a hollow square and a cross.
(((b ^ 0.5) × C) + ((1.0 − b) ^ 0.5) × (X − (a ^ 0.5) × C) ÷ (1.0 − a) ^ 0.5, t − 1)
One denoising step for all the pictures at once: C is what the network sees in the noisy pictures X at level t; the noise that would make X of it is inferred and mixed back at the next level's share. Ten of these, by p_ower over the tuple, turn noise into shapes.
diffusiona learned denoisera noise scheduleDDIM samplingp_ower over a tuple