sourcelibs/Sample/src/Sample.xtl
1⍝│ Sample: choosing the next token from scores: greedy, temperature, top-k, drawing by given uniform numbers
2⍝│
3⍝│ Import it with an alias of your choice: "sm:" u_se< "Sample".
4⍝│ Put libs/Sample/src on XETAL_PATH ("just path"); the reference is libs/Sample/docs.
5⍝│ Names with l: are exported; h: names are private to this file.
6⍝│
7⍝│ Scores (logits) are a row per position, a column per token; tokens
8⍝│ count from 1. The library draws no chance itself: a draw takes
9⍝│ uniform numbers in 0 .. 1 from the caller (r_oll!, seeded, or any
10⍝│ other source), so the same numbers give the same tokens.
11
12"nn:" u̲se< "NN"
13
14⍝│ A value per row spread across that row's columns (of an n x m matrix).
15ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }
16
17⍝━━ Choosing
18
19⍝│ g̲reedy s: the most likely token of each row.
20⍝│ >> "sm:" u_se< "Sample"
21⍝│ >> sm:g_reedy 2 3 r_eshape 0.1 2.0 0.5 3.0 0.0 1.0
22⍝│ 2 1
23ˡg̲reedy ← { s → ⁿⁿa̲rgmax s }
24
25⍝│ t p̲robs s: the probabilities at temperature t: softmax of the
26⍝│ scores over t (below 1 sharper, above 1 flatter).
27⍝│ >> "sm:" u_se< "Sample"
28⍝│ >> 1.0 sm:p_robs 0.0 0.6931471805599453
29⍝│ 0.3333333333333333 0.6666666666666666
30ˡp̲robs ← { t s → ⁿⁿs̲oftmax s ÷ t }
31
32⍝│ k t̲op p: only each row's k largest probabilities kept (the rest
33⍝│ 0), then scaled to sum to 1 again. On a tie at the k-th, all tied
34⍝│ are kept.
35⍝│ >> "sm:" u_se< "Sample"
36⍝│ >> 2 sm:t_op 0.1 0.5 0.15 0.25
37⍝│ 0.0 0.6666666666666666 0.0 0.3333333333333333
38ˡt̲op ← { k p →
39 m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p
40 rows ← m r̲eshape p
41 ⍝│ Each row's k-th largest: its largest taken and shifted below the
42 ⍝│ rest, k times (X_eTaL has no grade along a row yet). The shift is
43 ⍝│ finite, 1 + the range of the values, so it runs in 32-bit floats
44 ⍝│ too (a huge constant is infinity there, and 0 times it NaN).
45 (_, kth) ← k '{ (q, t) →
46 top ← 'm̲ax r̲/₂ q
47 (q − (1.0 + ('m̲ax r̲/ r̲avel q) − 'm̲in r̲/ r̲avel q) × f̲loat q ≥ top ʰs̲pread q, top)
48 } p̲ower (rows, (1 t̲ake m) r̲eshape 0.0)
49 kept ← rows × f̲loat rows ≥ kth ʰs̲pread rows
50 (s̲hape p) r̲eshape kept ÷ ('+ r̲/₂ kept) ʰs̲pread kept
51}
52
53⍝│ u d̲raw p: a token for each row of probabilities p, drawn by the
54⍝│ uniform number u (one per row): the first token where the running
55⍝│ sum of the row passes u.
56⍝│ >> "sm:" u_se< "Sample"
57⍝│ >> 0.1 0.7 sm:d_raw 2 3 r_eshape 0.2 0.3 0.5 0.2 0.3 0.5
58⍝│ 1 3
59ˡd̲raw ← { u p →
60 m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p
61 c ← '+ s̲\₂ m r̲eshape p
62 1 + '+ r̲/₂ c < u 'l̲eft t̲able o̲ffsets 2 s̲elect m
63}