sourcedemos/embedding-explorer/embedding-explorer.xtl
1⍝!/usr/bin/env xetal
2⍝│ Word embeddings from counts, and a 3-D look at them: 4,000 short
3⍝│ sentences over 70 words, each word's embedding a row of 64 numbers
4⍝│ made from which words occur near it (the Embed library: counts,
5⍝│ PPMI, a fixed projection). Words used alike point alike: the
6⍝│ animals together, the foods, the colors, the numbers, the places.
7⍝│ Learn's PCA turns the 64 dimensions into the 3 that spread the
8⍝│ words most, for a cloud to look at.
9
10"em:" u̲se< "Embed"
11"ml:" u̲se< "Learn"
12"t:" u̲se< "Strings"
13
14⍝━━ The corpus, read from data/ (just embedding-data)
15⍝│ The words, one a line, and the sentences as word numbers, each ended by 0.
16words ← ᵗl̲ines ⎕N̲GET "data/words.txt"
17ids ← f̲loor n̲umbers ⎕N̲GET "data/ids.txt"
18v ← t̲ally words
19
20⍝━━ The embeddings
21⍝│ How often each word occurs within 2 words of each other, then PPMI.
22C ← (v, 2) ᵉᵐc̲ooccur ids
23P ← ᵉᵐp̲pmi C
24⍝│ 64 numbers a word, each row unit length.
25E ← ᵉᵐu̲nit 64 ᵉᵐp̲roject P
26⍝│ The 3 directions that spread the words most.
27(mu, V, ev) ← 3 ᵐˡp̲ca E
28xyz ← (mu, V, ev) ᵐˡp̲roject E
29⍝ -- end of the core ----------------------------------------
30
31⍝━━ What it shows
32ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }
33⍝│ A word's number, and its 5 nearest words by cosine in 64 dimensions.
34ᵘn̲umber ← { w → f̲irst w̲here '{ x → w m̲atch d̲isclose x } e̲ach words }
35ᵘn̲ear ← { w → w c̲at ": " c̲at " " ᵗj̲oin '{ j → d̲isclose j s̲elect words } m̲ap (E, 5) ᵉᵐn̲earest ᵘn̲umber w }
36ᵘn̲ear "cat"
37ᵘn̲ear "apple"
38ᵘn̲ear "red"
39ᵘn̲ear "three"
40ᵘn̲ear "park"
41⍝│ The share of the words' variance the 3 directions keep.
42Z ← E − (s̲hape E) r̲eshape mu
43ᵘr̲ound ('+ r̲/ ev) ÷ ('+ r̲/ r̲avel Z × Z) ÷ f̲loat v