programdemos/embedding-explorer/embedding-explorer.xtl

Word embeddings from counts, and a 3-D look at them: 4,000 short sentences over 70 words, each word's embedding a row of 64 numbers made from which words occur near it (the Embed library: counts, PPMI, a fixed projection). Words used alike point alike: the animals together, the foods, the colors, the numbers, the places. Learn's PCA turns the 64 dimensions into the 3 that spread the words most, for a cloud to look at.

source · imports em: libs/Embed/src/Embed.xtl; ml: libs/Learn/src/Learn.xtl; t: work/libs/Strings/src/Strings.xtl

The corpus, read from data/ (just embedding-data)

words : Box Char

value · line 16

The words, one a line, and the sentences as word numbers, each ended by 0.

words ← ᵗl̲ines ⎕N̲GET "data/words.txt"

ids : Int

value · line 17
ids ← f̲loor n̲umbers ⎕N̲GET "data/ids.txt"
Used in: C

v : Int

value · line 18
v ← t̲ally words

The embeddings

C : Int

value · line 22

How often each word occurs within 2 words of each other, then PPMI.

C ← (v, 2) ᵉᵐc̲ooccur ids
Used in: P

P : Float

value · line 23
P ← ᵉᵐp̲pmi C
Used in: E

E : Float

value · line 25

64 numbers a word, each row unit length.

E ← ᵉᵐu̲nit 64 ᵉᵐp̲roject P
Used in: mu, xyz, ᵘn̲ear, Z

mu : Float

value · line 27

The 3 directions that spread the words most.

(mu, V, ev) ← 3 ᵐˡp̲ca E
Used in: mu, xyz, Z

V : Float

value · line 27

The 3 directions that spread the words most.

(mu, V, ev) ← 3 ᵐˡp̲ca E
Used in: mu, xyz

ev : Float

value · line 27

The 3 directions that spread the words most.

(mu, V, ev) ← 3 ᵐˡp̲ca E

xyz : Float

value · line 28
xyz ← (mu, V, ev) ᵐˡp̲roject E

What it shows

ᵘr̲ound : Float -> Float

function · line 32
ᵘr̲ound ← { a → (f̲loat f̲loor 0.5 + 1000.0 × a) ÷ 1000.0 }

ᵘn̲umber : Char -> Int

function · line 34

A word's number, and its 5 nearest words by cosine in 64 dimensions.

ᵘn̲umber ← { w → f̲irst w̲here '{ x → w m̲atch d̲isclose x } e̲ach words }
Used in: ᵘn̲ear

ᵘn̲ear : Char -> Char

function · line 35
ᵘn̲ear ← { w → w c̲at ": " c̲at " " ᵗj̲oin '{ j → d̲isclose j s̲elect words } m̲ap (E, 5) ᵉᵐn̲earest ᵘn̲umber w }

Z : Float

value · line 42

The share of the words' variance the 3 directions keep.

Z ← E − (s̲hape E) r̲eshape mu