sourcelibs/Embed/src/Embed.xtl

1⍝│ Embed: word embeddings from counts: co-occurrence, PPMI, a fixed projection, cosine neighbors 2⍝│ 3⍝│ Import it with an alias of your choice: "em:" u_se< "Embed". 4⍝│ Put libs/Embed/src on XETAL_PATH ("just path"); the reference is libs/Embed/docs. 5⍝│ Names with l: are exported; h: names are private to this file. 6⍝│ 7⍝│ A corpus is a vector of word numbers (from 1), each sentence ended 8⍝│ by a 0. A word's embedding is a row: which words occur near it, 9⍝│ weighted by how much more often than chance (PPMI), then projected 10⍝│ to fewer dimensions. Words used alike get rows that point alike. 11 12"ls:" u̲se< "Lists" 13 14⍝━━ Counting 15 16⍝│ n c̲ounts k: how many times each of 1 .. n occurs in k, by sorting 17⍝│ (the keys in order, then the length of each run of equal keys). 18⍝│ >> "em:" u_se< "Embed" 19⍝│ >> 5 em:c_ounts 3 1 3 3 5 20⍝│ 1 0 3 0 1 21ˡc̲ounts ← { n k → 22 s ← (g̲rade k) s̲elect k 23 first ← 1 c̲at (1 d̲rop s) ≠ -1 d̲rop s 24 at ← w̲here first 25 runs ← ((1 d̲rop at) c̲at 1 + t̲ally s) − at 26 ((first r̲eplicate s) i̲ndexOf r̲ange n) s̲elect runs c̲at 0 27} 28 29⍝│ (v, w) c̲ooccur ids: a v x v matrix of how often word i occurs 30⍝│ within w words of word j in the same sentence (both ways). 31⍝│ >> "em:" u_se< "Embed" 32⍝│ >> (3, 1) em:c_ooccur 1 2 3 0 1 2 0 33⍝│ 0 2 0 34⍝│ 2 0 1 35⍝│ 0 1 0 36ˡc̲ooccur ← { (v, w) ids → 37 sent ← '+ s̲\ ids = 0 38 pairs ← '{ d → 39 a ← (n̲eg d) d̲rop ids 40 b ← d d̲rop ids 41 ok ← (((n̲eg d) d̲rop sent) = d d̲rop sent) ∧ (a > 0) ∧ b > 0 42 (ok r̲eplicate (v × a − 1) + b) c̲at ok r̲eplicate (v × b − 1) + a 43 } m̲ap r̲ange w 44 (v c̲at v) r̲eshape (v × v) ˡc̲ounts ˡˢr̲aze pairs 45} 46 47⍝━━ Weighting and projecting 48 49⍝│ p̲pmi C: positive pointwise mutual information: the log of how 50⍝│ much more often each pair occurs than if the words were independent, 51⍝│ 0 where less (or never; a word with no company has a row of 0s). 52ˡp̲pmi ← { C → 53 c ← f̲loat C 54 r ← '+ r̲/₂ c 55 0.0 m̲ax l̲og 1e-300 m̲ax (c × '+ r̲/ r) ÷ 1e-300 m̲ax r '× t̲able r 56} 57 58⍝│ d p̲roject M: each row of M projected to d dimensions by a fixed 59⍝│ matrix of +1 and -1 over the square root of d (random indexing): 60⍝│ distances and angles between rows are roughly kept. The signs come 61⍝│ from a hash of each place, the same every run, no chance. 62ˡp̲roject ← { d M → 63 n ← 2 s̲elect s̲hape M 64 h ← 43758.5453 × s̲in (12.9898 × f̲loat r̲ange n) '+ t̲able 78.233 × f̲loat r̲ange d 65 M '+ '× i̲nner (1.0 − 2.0 × f̲loat (h − f̲loat f̲loor h) < 0.5) ÷ (f̲loat d) ^ 0.5 66} 67 68⍝━━ Neighbors 69 70⍝│ u̲nit E: each row scaled to length 1 (a row of 0s stays 0). 71ˡu̲nit ← { E → 72 len ← (1e-300 m̲ax '+ r̲/₂ E × E) ^ 0.5 73 E ÷ len 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape E 74} 75 76⍝│ E c̲osines i: the cosine of row i with every row of E. 77ˡc̲osines ← { E i → (ˡu̲nit E) '+ '× i̲nner r̲avel i s̲elect ˡu̲nit E } 78 79⍝│ (E, k) n̲earest i: the k rows of E most like row i (by cosine), 80⍝│ nearest first, row i itself left out. 81⍝│ >> "em:" u_se< "Embed" 82⍝│ >> E := 4 2 r_eshape 1.0 0.0 0.9 0.1 0.0 1.0 0.7 0.7 83⍝│ >> (E, 2) em:n_earest 1 84⍝│ 2 4 85ˡn̲earest ← { (E, k) i → 86 order ← g̲rade n̲eg E ˡc̲osines i 87 k t̲ake (order ≠ i) r̲eplicate order 88}