librarywork/xetal-ml/libs/Learn/src/Learn.xtl
Learn: classic machine learning: k-means, k nearest neighbors, PCA, logistic regression
Import it with an alias of your choice: "ml:" u_se< "Learn". Put libs/Learn/src on XETAL_PATH ("just path"); the reference is libs/Learn/docs. Names with l: are exported; h: names are private to this file.
After APLearn's list (k-means, kNN, PCA, logistic regression), each a fit and a predict: the data X is a matrix, one example per row; labels count from 1. Every model is whole-array arithmetic, and an iterative fit is a step repeated by p_ower.
Helpers (private)
ʰd̲ist : Float -> Float -> Float
The squared distance between every row of A (n x d) and every row of B (k x d): n x k.
ʰd̲ist ← { A B → (('+ r̲/₂ A × A) '+ t̲able '+ r̲/₂ B × B) − 2.0 × A '+ '× i̲nner o̲\ B }
ʰo̲nes : Float -> Float
X with a column of 1s on the right (for a bias row).
ʰo̲nes ← { X → X c̲at₂ ((t̲ally X) c̲at 1) r̲eshape 1.0 }
k-means: clusters by their centers
ˡa̲ssign : Float -> Float -> Int
C a̲ssign X: the nearest of the centers C (one per row) for each row of X, from 1.
"ml:" u̲se< "Learn"
(2 2 r̲eshape 0.0 0.0 10.0 10.0) ᵐˡa̲ssign 3 2 r̲eshape 1.0 0.0 9.0 9.0 0.0 2.0 1 2 1
ˡa̲ssign ← { C X → ⁿⁿa̲rgmax n̲eg X ʰd̲ist C }
ˡk̲step : Float -> Float -> Float
X k̲step C: one step of k-means (Lloyd's): each center moved to
the mean of the rows of X nearest it; a center no row is nearest
stays where it is. Repeat it with p_ower.
"ml:" u̲se< "Learn"
X ← 4 2 r̲eshape 0.0 0.0 0.0 2.0 10.0 10.0 10.0 12.0
X ᵐˡk̲step 2 2 r̲eshape 1.0 1.0 9.0 9.0 0.0 1.0 10.0 11.0
ˡk̲step ← { X C → O ← (t̲ally C) ⁿⁿo̲neHot C ˡa̲ssign X n ← ('+ r̲/₁ O) 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape X (((o̲\ O) '+ '× i̲nner X) + C × f̲loat n = 0.0) ÷ 1.0 m̲ax n }
ˡi̲nertia : Float -> Float -> Float
C i̲nertia X: the sum of each row's squared distance to its nearest center.
ˡi̲nertia ← { C X → '+ r̲/ 'm̲in r̲/₂ X ʰd̲ist C }
ˡf̲arthest : Int -> Float -> Float
k f̲arthest X: k starting centers, chosen without chance: the
first row, then each time the row farthest from the centers so far.
ˡf̲arthest ← { k X → d ← 2 s̲elect s̲hape X (k − 1) '{ C → C c̲at₁ (1 c̲at d) r̲eshape (ⁿⁿa̲rgmax 'm̲in r̲/₂ X ʰd̲ist C) s̲elect X } p̲ower 1 t̲ake X }
ˡk̲means : (Int, Int) -> Float -> Float
(k, n) k̲means X: k centers after n steps from the farthest-first start.
ˡk̲means ← { (k, n) X → n '{ C → X ˡk̲step C } p̲ower k ˡf̲arthest X }
k nearest neighbors: a vote
ˡk̲nn : (Int, Float, Int) -> Float -> Int
(k, X, y) k̲nn Q: for each row of Q, the label most of its k
nearest rows of X have (y: X's labels, from 1); on a tie the
smallest label. The neighbors are found nearest first, each shifted
beyond the rest once taken (X_eTaL has no grade along a row yet); the
shift is finite, 1 + the largest distance, so it runs in 32-bit
floats too.
"ml:" u̲se< "Learn"
X ← 4 1 r̲eshape 0.0 1.0 10.0 11.0
(3, X, 1 1 2 2) ᵐˡk̲nn 2 1 r̲eshape 0.5 10.5 1 2
ˡk̲nn ← { (k, X, y) Q → c ← 'm̲ax r̲/ y D ← Q ʰd̲ist X (_, V) ← k '{ (d, v) → j ← ⁿⁿa̲rgmax n̲eg d (d + (1.0 + 'm̲ax r̲/ r̲avel d) × f̲loat j '= t̲able r̲ange t̲ally X, v + c ⁿⁿo̲neHot j s̲elect y) } p̲ower (D, ((t̲ally Q) c̲at c) r̲eshape 0.0) ⁿⁿa̲rgmax V }
PCA: the directions of most variance
ˡp̲ca : Int -> Float -> (Float, Float, Float)
n p̲ca X: the mean of X's rows, the first n principal directions
(a column each, unit length) and the variance along each: (mu, V,
ev). Power iteration on the covariance, each direction then taken
out of it (deflation); a direction's sign makes its items sum
positive.
ˡp̲ca ← { n X → d ← 2 s̲elect s̲hape X mu ← ('+ r̲/₁ X) ÷ f̲loat t̲ally X Z ← X − (s̲hape X) r̲eshape mu S0 ← ((o̲\ Z) '+ '× i̲nner Z) ÷ f̲loat t̲ally X u0 ← 1.0 + 0.1 × f̲loat o̲ffsets d (_, V, ev, _) ← n '{ (S, V, ev, i) → v ← 200 '{ u → w ← S '+ '× i̲nner u◆ w ÷ ('+ r̲/ w × w) ^ 0.5 } p̲ower u0 v1 ← v × 1.0 − 2.0 × f̲loat 0.0 > '+ r̲/ v lam ← '+ r̲/ v1 × S '+ '× i̲nner v1 (S − lam × v1 '× t̲able v1, V + v1 '× t̲able f̲loat i = r̲ange n, ev + lam × f̲loat i = r̲ange n, i + 1) } p̲ower (S0, (d c̲at n) r̲eshape 0.0, n r̲eshape 0.0, 1) (mu, V, ev) }
ˡp̲roject : (Num a, Any b) => (a, a, b) -> a -> a
(mu, V, ev) p̲roject X: each row of X in the principal directions: n columns.
ˡp̲roject ← { (mu, V, _) X → (X − (s̲hape X) r̲eshape mu) '+ '× i̲nner V }
Logistic regression: classes by a softmax
ˡl̲ogistic : (Int, Float) -> (Float, Int) -> Float
(n, lr) l̲ogistic (X, y): the weights of a softmax over X's
columns (a row per column, the bias the last row; one column per
label), after n steps of gradient descent on the cross-entropy at
rate lr, from zeros.
ˡl̲ogistic ← { (n, lr) (X, y) → Y ← ('m̲ax r̲/ y) ⁿⁿo̲neHot y G ← o̲\ ʰo̲nes X n '{ W → W − lr × (G '+ '× i̲nner (ⁿⁿs̲oftmax X ⁿⁿd̲ense W) − Y) ÷ f̲loat t̲ally X } p̲ower ((1 + 2 s̲elect s̲hape X) c̲at 2 s̲elect s̲hape Y) r̲eshape 0.0 }
ˡp̲redict : Num a => a -> a -> Int
W p̲redict X: the label (from 1) the weights W give each row of X.
ˡp̲redict ← { W X → ⁿⁿa̲rgmax ⁿⁿs̲oftmax X ⁿⁿd̲ense W }