programdemos/ternary-net/ternary-net.xtl
A 1.58-bit network: a tiny classifier (which of three spiral arms a point is on, 2 -> 16 -> 16 -> 3) run with its weights stored four ways: 32-bit and 16-bit floats (simulated by rounding), 8-bit integers, and ternary -1 / 0 / +1 (log2 3 = 1.58 bits each), where a layer is additions and subtractions of its inputs and one scale. The same points go through all four; the measures compare them.
g : Int
The map's side (g by g points) and the ternary threshold (a weight is kept as +1 or -1 when its size passes t times the layer's mean).
g ← 12
The weights, read from data/ (just ternary-train)
fp : Float
fp: trained in full precision; qa: fine-tuned for ternary weights. Each is 17 x 3 x 16: for each row (16 inputs, padded with 0, then the bias), layer and output (padded with 0).
fp ← 17 3 16 r̲eshape n̲umbers ⎕N̲GET "data/fp.txt"
nums : Float
The test points, one per line: x, y and the arm (1, 2 or 3).
nums ← n̲umbers ⎕N̲GET "data/test.txt"
The inputs: points padded to 16 numbers
e1 : Float
Each point (x, y) becomes x e1 + y e2: a row of 16 with the rest 0, so every layer is the same 16 wide.
e1 ← 16 t̲ake 1.0
c : Float
The map: g by g points over -1.1 .. 1.1, row by row, y downwards.
c ← -1.1 + 2.2 × (0.5 + f̲loat o̲ffsets g) ÷ f̲loat g
The network: three layers of one array
ᵘl̲ayer : Num a => a -> a -> a
A layer plane p (17 x 16): 16 rows of weights, then the biases.
ᵘl̲ayer ← { x p → (x '+ '× i̲nner 16 t̲ake p) + (o̲ffsets t̲ally x) 'r̲ight t̲able f̲irst -1 t̲ake p }
ᵘn̲et : Float -> Float -> Float
ᵘn̲et ← { x k → h1 ← ᵘr̲elu x ᵘl̲ayer 1 s̲elect₂ k h2 ← ᵘr̲elu h1 ᵘl̲ayer 2 s̲elect₂ k 3 t̲ake₂ h2 ᵘl̲ayer 3 s̲elect₂ k }
The formats: the weights rounded four ways
w : Float
w is 16 x 3 x 16 (input, layer, output); a value per layer is spread back over its layer; the biases stay as they are.
w ← 16 t̲ake m
s8 : Float
Floats: Quant's f_loats keeps b bits after the leading one (23 for FP32, 10 FP16). INT8: each layer's largest weight becomes 127 steps, and Quant's s_nap rounds every weight to its layer's step.
s8 ← ᵘl̲ayers ('m̲ax r̲/₁₃ a̲bs w) ÷ 127.0
q : Float
Ternary: +1 or -1 where the size passes t times the layer's mean size, else 0 (Quant's c_ut); the scale a is the mean size of the kept weights.
q ← (t × ᵘl̲ayers ('+ r̲/₁₃ a̲bs w) ÷ nw) qz:c̲ut w
The maps: every map point through each format
The measures
ᵘa̲rm : Num a => a -> Float
The arm each point is given (the largest of its three outputs).
ᵘa̲rm ← { y → '+ r̲/₂ (f̲loat y = ('m̲ax r̲/₂ y) 'l̲eft t̲able o̲ffsets 3) × (o̲ffsets t̲ally y) 'r̲ight t̲able 1.0 2.0 3.0 }
ᵘm̲ean : Float -> Float
ᵘm̲ean ← { x → ('+ r̲/ r̲avel x) ÷ f̲loat t̲ally r̲avel x }
ᵘm̲easures : Float -> Float -> Float
For a format's weights k and its map y: accuracy on the test points, agreement with FP32 over the map, mean output error.
ᵘm̲easures ← { k y → acc ← ᵘm̲ean f̲loat pl = ᵘa̲rm test ᵘn̲et k agree ← ᵘm̲ean f̲loat (ᵘa̲rm y) = ᵘa̲rm y32 err ← ᵘm̲ean a̲bs y − y32 acc c̲at agree c̲at err }