librarylibs/Sample/src/Sample.xtl

Sample: choosing the next token from scores: greedy, temperature, top-k, drawing by given uniform numbers

Import it with an alias of your choice: "sm:" u_se< "Sample". Put libs/Sample/src on XETAL_PATH ("just path"); the reference is libs/Sample/docs. Names with l: are exported; h: names are private to this file.

Scores (logits) are a row per position, a column per token; tokens count from 1. The library draws no chance itself: a draw takes uniform numbers in 0 .. 1 from the caller (r_oll!, seeded, or any other source), so the same numbers give the same tokens.

source · imports nn: libs/NN/src/NN.xtl

ʰs̲pread : a -> b -> a

function (private) · line 15

A value per row spread across that row's columns (of an n x m matrix).

ʰs̲pread ← { v m → v 'l̲eft t̲able o̲ffsets 2 s̲elect s̲hape m }
Used in: ˡt̲op

Choosing

ˡg̲reedy : Num a => a -> Int

function · line 23

g̲reedy s: the most likely token of each row.

      "sm:" u̲se< "Sample"
      ˢᵐg̲reedy 2 3 r̲eshape 0.1 2.0 0.5 3.0 0.0 1.0
2 1
ˡg̲reedy ← { s → ⁿⁿa̲rgmax s }

ˡp̲robs : Num a => a -> a -> Float

function · line 30

t p̲robs s: the probabilities at temperature t: softmax of the scores over t (below 1 sharper, above 1 flatter).

      "sm:" u̲se< "Sample"
      1.0 ˢᵐp̲robs 0.0 0.6931471805599453
0.3333333333333333 0.6666666666666666
ˡp̲robs ← { t s → ⁿⁿs̲oftmax s ÷ t }

ˡt̲op : Int -> Float -> Float

function · line 38

k t̲op p: only each row's k largest probabilities kept (the rest 0), then scaled to sum to 1 again. On a tie at the k-th, all tied are kept.

      "sm:" u̲se< "Sample"
      2 ˢᵐt̲op 0.1 0.5 0.15 0.25
0.0 0.6666666666666666 0.0 0.3333333333333333
ˡt̲op ← { k p →
  m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p
  rows ← m r̲eshape p
  ⍝│ Each row's k-th largest: its largest taken and shifted below the
  ⍝│ rest, k times (X_eTaL has no grade along a row yet). The shift is
  ⍝│ finite, 1 + the range of the values, so it runs in 32-bit floats
  ⍝│ too (a huge constant is infinity there, and 0 times it NaN).
  (_, kth) ← k '{ (q, t) →
    top ← 'm̲ax r̲/₂ q
    (q − (1.0 + ('m̲ax r̲/ r̲avel q) − 'm̲in r̲/ r̲avel q) × f̲loat q ≥ top ʰs̲pread q, top)
  } p̲ower (rows, (1 t̲ake m) r̲eshape 0.0)
  kept ← rows × f̲loat rows ≥ kth ʰs̲pread rows
  (s̲hape p) r̲eshape kept ÷ ('+ r̲/₂ kept) ʰs̲pread kept
}

ˡd̲raw : (Num a, Num b, Truthy b) => a -> a -> b

function · line 59

u d̲raw p: a token for each row of probabilities p, drawn by the uniform number u (one per row): the first token where the running sum of the row passes u.

      "sm:" u̲se< "Sample"
      0.1 0.7 ˢᵐd̲raw 2 3 r̲eshape 0.2 0.3 0.5 0.2 0.3 0.5
1 3
ˡd̲raw ← { u p →
  m ← ((t̲ally r̲avel p) d̲iv -1 t̲ake s̲hape p) c̲at -1 t̲ake s̲hape p
  c ← '+ s̲\₂ m r̲eshape p
  1 + '+ r̲/₂ c < u 'l̲eft t̲able o̲ffsets 2 s̲elect m
}