sourcedemos/embedding-explorer/embedding-explorer.xtl

1⍝!/usr/bin/env xetal 2⍝│ Word embeddings from counts, and a 3-D look at them: 4,000 short 3⍝│ sentences over 70 words, each word's embedding a row of 64 numbers 4⍝│ made from which words occur near it (the Embed library: counts, 5⍝│ PPMI, a fixed projection). Words used alike point alike: the 6⍝│ animals together, the foods, the colors, the numbers, the places. 7⍝│ Learn's PCA turns the 64 dimensions into the 3 that spread the 8⍝│ words most, for a cloud to look at. 9 10"em:" u̲se< "Embed" 11"ml:" u̲se< "Learn" 12"t:" u̲se< "Strings" 13 14⍝━━ The corpus, read from data/ (just embedding-data) 15⍝│ The words, one a line, and the sentences as word numbers, each ended by 0. 16words ← ᵗl̲ines ⎕N̲GET "data/words.txt" 17ids ← f̲loor n̲umbers ⎕N̲GET "data/ids.txt" 18v ← t̲ally words 19 20⍝━━ The embeddings 21⍝│ How often each word occurs within 2 words of each other, then PPMI. 22C ← (v, 2) ᵉᵐc̲ooccur ids 23P ← ᵉᵐp̲pmi C 24⍝│ 64 numbers a word, each row unit length. 25E ← ᵉᵐu̲nit 64 ᵉᵐp̲roject P 26⍝│ The 3 directions that spread the words most. 27(mu, V, ev) ← 3 ᵐˡp̲ca E 28xyz ← (mu, V, ev) ᵐˡp̲roject E 29⍝ -- end of the core ---------------------------------------- 30 31⍝━━ What it shows 32ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 } 33⍝│ A word's number, and its 5 nearest words by cosine in 64 dimensions. 34ᵘn̲umber ← { w → f̲irst w̲here '{ x → w m̲atch d̲isclose x } e̲ach words } 35ᵘn̲ear ← { w → w c̲at ": " c̲at " " ᵗj̲oin '{ j → d̲isclose j s̲elect words } m̲ap (E, 5) ᵉᵐn̲earest ᵘn̲umber w } 36ᵘn̲ear "cat" 37ᵘn̲ear "apple" 38ᵘn̲ear "red" 39ᵘn̲ear "three" 40ᵘn̲ear "park" 41⍝│ The share of the words' variance the 3 directions keep. 42Z ← E − (s̲hape E) r̲eshape mu 43ᵘr̲ound ('+ r̲/ ev) ÷ ('+ r̲/ r̲avel Z × Z) ÷ f̲loat v