sourcelibs/Embed/src/Embed.xtl
1⍝│ Embed: word embeddings from counts: co-occurrence, PPMI, a fixed projection, cosine neighbors
2⍝│
3⍝│ Import it with an alias of your choice: "em:" u_se< "Embed".
4⍝│ Put libs/Embed/src on XETAL_PATH ("just path"); the reference is libs/Embed/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ A corpus is a vector of word numbers (from 1), each sentence ended
8⍝│ by a 0. A word's embedding is a row: which words occur near it,
9⍝│ weighted by how much more often than chance (PPMI), then projected
10⍝│ to fewer dimensions. Words used alike get rows that point alike.
11
12"ls:" u̲se< "Lists"
13
14⍝━━ Counting
15
16⍝│ n c̲ounts k: how many times each of 1 .. n occurs in k, by sorting
17⍝│ (the keys in order, then the length of each run of equal keys).
18⍝│ >> "em:" u_se< "Embed"
19⍝│ >> 5 em:c_ounts 3 1 3 3 5
20⍝│ 1 0 3 0 1
21ˡc̲ounts ← { n k →
22 s ← (g̲rade k) s̲elect k
23 first ← 1 c̲at (1 d̲rop s) ≠ -1 d̲rop s
24 at ← w̲here first
25 runs ← ((1 d̲rop at) c̲at 1 + t̲ally s) − at
26 ((first r̲eplicate s) i̲ndexOf r̲ange n) s̲elect runs c̲at 0
27}
28
29⍝│ (v, w) c̲ooccur ids: a v x v matrix of how often word i occurs
30⍝│ within w words of word j in the same sentence (both ways).
31⍝│ >> "em:" u_se< "Embed"
32⍝│ >> (3, 1) em:c_ooccur 1 2 3 0 1 2 0
33⍝│ 0 2 0
34⍝│ 2 0 1
35⍝│ 0 1 0
36ˡc̲ooccur ← { (v, w) ids →
37 sent ← '+ s̲\ ids = 0
38 pairs ← '{ d →
39 a ← (n̲eg d) d̲rop ids
40 b ← d d̲rop ids
41 ok ← (((n̲eg d) d̲rop sent) = d d̲rop sent) ∧ (a > 0) ∧ b > 0
42 (ok r̲eplicate (v × a − 1) + b) c̲at ok r̲eplicate (v × b − 1) + a
43 } m̲ap r̲ange w
44 (v c̲at v) r̲eshape (v × v) ˡc̲ounts ˡˢr̲aze pairs
45}
46
47⍝━━ Weighting and projecting
48
49⍝│ p̲pmi C: positive pointwise mutual information: the log of how
50⍝│ much more often each pair occurs than if the words were independent,
51⍝│ 0 where less (or never; a word with no company has a row of 0s).
52ˡp̲pmi ← { C →
53 c ← f̲loat C
54 r ← '+ r̲/₂ c
55 0.0 m̲ax l̲og 1e-300 m̲ax (c × '+ r̲/ r) ÷ 1e-300 m̲ax r '× t̲able r
56}
57
58⍝│ d p̲roject M: each row of M projected to d dimensions by a fixed
59⍝│ matrix of +1 and -1 over the square root of d (random indexing):
60⍝│ distances and angles between rows are roughly kept. The signs come
61⍝│ from a hash of each place, the same every run, no chance.
62ˡp̲roject ← { d M →
63 n ← 2 s̲elect s̲hape M
64 h ← 43758.5453 × s̲in (12.9898 × f̲loat r̲ange n) '+ t̲able 78.233 × f̲loat r̲ange d
65 M '+ '× i̲nner (1.0 − 2.0 × f̲loat (h − f̲loat f̲loor h) < 0.5) ÷ (f̲loat d) ^ 0.5
66}
67
68⍝━━ Neighbors
69
70⍝│ u̲nit E: each row scaled to length 1 (a row of 0s stays 0).
71ˡu̲nit ← { E →
72 len ← (1e-300 m̲ax '+ r̲/₂ E × E) ^ 0.5
73 E ÷ len 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape E
74}
75
76⍝│ E c̲osines i: the cosine of row i with every row of E.
77ˡc̲osines ← { E i → (ˡu̲nit E) '+ '× i̲nner r̲avel i s̲elect ˡu̲nit E }
78
79⍝│ (E, k) n̲earest i: the k rows of E most like row i (by cosine),
80⍝│ nearest first, row i itself left out.
81⍝│ >> "em:" u_se< "Embed"
82⍝│ >> E := 4 2 r_eshape 1.0 0.0 0.9 0.1 0.0 1.0 0.7 0.7
83⍝│ >> (E, 2) em:n_earest 1
84⍝│ 2 4
85ˡn̲earest ← { (E, k) i →
86 order ← g̲rade n̲eg E ˡc̲osines i
87 k t̲ake (order ≠ i) r̲eplicate order
88}