librarylibs/Sample/src/Sample.xtl
Sample: choosing the next token from scores: greedy, temperature, top-k, drawing by given uniform numbers
Import it with an alias of your choice: "sm:" u_se< "Sample". Put libs/Sample/src on XETAL_PATH ("just path"); the reference is libs/Sample/docs. Names with l: are exported; h: names are private to this file.
Scores (logits) are a row per position, a column per token; tokens count from 1. The library draws no chance itself: a draw takes uniform numbers in 0 .. 1 from the caller (r_oll!, seeded, or any other source), so the same numbers give the same tokens.
ʰs̲pread : a -> b -> a
A value per row spread across that row's columns (of an n x m matrix).
ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }
Choosing
ˡg̲reedy : Num a => a -> Int
g̲reedy s: the most likely token of each row.
"sm:" u̲se< "Sample"
ˢᵐg̲reedy 2 3 r̲eshape 0.1 2.0 0.5 3.0 0.0 1.0 2 1
ˡg̲reedy ← { s → ⁿⁿa̲rgmax s }
ˡp̲robs : Num a => a -> a -> Float
t p̲robs s: the probabilities at temperature t: softmax of the
scores over t (below 1 sharper, above 1 flatter).
"sm:" u̲se< "Sample"
1.0 ˢᵐp̲robs 0.0 0.6931471805599453 0.3333333333333333 0.6666666666666666
ˡp̲robs ← { t s → ⁿⁿs̲oftmax s ÷ t }
ˡt̲op : Int -> Float -> Float
k t̲op p: only each row's k largest probabilities kept (the rest
0), then scaled to sum to 1 again. On a tie at the k-th, all tied
are kept.
"sm:" u̲se< "Sample"
2 ˢᵐt̲op 0.1 0.5 0.15 0.25 0.0 0.6666666666666666 0.0 0.3333333333333333
ˡt̲op ← { k p → m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p rows ← m r̲eshape p ⍝│ Each row's k-th largest: its largest taken and shifted below the ⍝│ rest, k times (X_eTaL has no grade along a row yet). The shift is ⍝│ finite, 1 + the range of the values, so it runs in 32-bit floats ⍝│ too (a huge constant is infinity there, and 0 times it NaN). (_, kth) ← k '{ (q, t) → top ← 'm̲ax r̲/₂ q (q − (1.0 + ('m̲ax r̲/ r̲avel q) − 'm̲in r̲/ r̲avel q) × f̲loat q ≥ top ʰs̲pread q, top) } p̲ower (rows, (1 t̲ake m) r̲eshape 0.0) kept ← rows × f̲loat rows ≥ kth ʰs̲pread rows (s̲hape p) r̲eshape kept ÷ ('+ r̲/₂ kept) ʰs̲pread kept }
ˡd̲raw : (Num a, Num b, Truthy b) => a -> a -> b
u d̲raw p: a token for each row of probabilities p, drawn by the
uniform number u (one per row): the first token where the running
sum of the row passes u.
"sm:" u̲se< "Sample"
0.1 0.7 ˢᵐd̲raw 2 3 r̲eshape 0.2 0.3 0.5 0.2 0.3 0.5 1 3
ˡd̲raw ← { u p → m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p c ← '+ s̲\₂ m r̲eshape p 1 + '+ r̲/₂ c < u 'l̲eft t̲able o̲ffsets 2 s̲elect m }