programdemos/ternary-net/ternary-net.xtl
A 1.58-bit network: a tiny classifier (which of three spiral arms a point is on, 2 -> 16 -> 16 -> 3) run with its weights stored four ways: 32-bit and 16-bit floats (simulated by rounding), 8-bit integers, and ternary -1 / 0 / +1 (log2 3 = 1.58 bits each), where a layer is additions and subtractions of its inputs and one scale. The same points go through all four; the measures compare them.
g : Int
The map's side (g by g points) and the ternary threshold (a weight is kept as +1 or -1 when its size passes t times the layer's mean).
g ← 12
The weights, read from data/ (just ternary-train)
fp : Float
fp: trained in full precision; qa: fine-tuned for ternary weights. Each is 17 x 3 x 16: for each row (16 inputs, padded with 0, then the bias), layer and output (padded with 0).
fp ← 17 3 16 r̲eshape n̲umbers ⎕N̲GET "data/fp.txt"
tp : Float
The test points, one per line: x, y and the arm (1, 2 or 3).
tp ← n̲umbers ⎕N̲GET "data/test.txt"
The inputs: points padded to 16 numbers
e1 : Float
Each point (x, y) becomes x e1 + y e2: a row of 16 with the rest 0, so every layer is the same 16 wide.
e1 ← 16 t̲ake 1.0
c : Float
The map: g by g points over -1.1 .. 1.1, row by row, y downwards.
c ← -1.1 + 2.2 × (0.5 + f̲loat o̲ffsets g) ÷ f̲loat g
The network: three layers of one array
ᵘl̲ayer : Num a => a -> a -> a
A layer plane p (17 x 16): 16 rows of weights, then the biases.
ᵘl̲ayer ← { x p → (x '+ '× i̲nner 16 t̲ake p) + (o̲ffsets t̲ally x) 'r̲ight t̲able f̲irst -1 t̲ake p }
ᵘn̲et : Float -> Float -> Float
ᵘn̲et ← { x k → h ← ᵘr̲elu x ᵘl̲ayer 1 s̲elect₂ k h ← ᵘr̲elu h ᵘl̲ayer 2 s̲elect₂ k 3 t̲ake₂ h ᵘl̲ayer 3 s̲elect₂ k }
The formats: the weights rounded four ways
w : Float
w is 16 x 3 x 16 (input, layer, output); a value per layer is spread back over its layer; the biases stay as they are.
w ← 16 t̲ake m
ᵘf̲loats : Float -> Float -> Float
Floats: keep b bits after the leading one (23 for FP32, 10 FP16).
ᵘf̲loats ← { b x → s ← 2.0 ^ (f̲loat f̲loor (l̲og 0.000000001 m̲ax a̲bs x) ÷ l̲og 2.0) − b s × f̲loat f̲loor 0.5 + x ÷ s }
ᵘi̲nt8 : Float -> Float
INT8: each layer's largest weight becomes 127.
ᵘi̲nt8 ← { x → s ← ᵘl̲ayers ('m̲ax r̲/₁₃ a̲bs x) ÷ 127.0 s × f̲loat f̲loor 0.5 + x ÷ s }
ᵘt̲ern : Float -> Float -> Float
Ternary: +1 or -1 where the size passes t times the layer's mean size, else 0; the scale a is the mean size of the kept weights.
ᵘt̲ern ← { k x → s ← k × ᵘl̲ayers ('+ r̲/₁₃ a̲bs x) ÷ nw (f̲loat x > s) − f̲loat x < n̲eg s }
Additions only: a ternary layer for one input h
ᵘa̲dds : Float -> Float -> Float
Sum the inputs where the weight is +1, subtract those where it is -1, then one multiply by the layer's scale.
ᵘa̲dds ← { h k → ('+ r̲/ (h 'l̲eft t̲able o̲ffsets 16) × f̲loat k = 1.0) − '+ r̲/ (h 'l̲eft t̲able o̲ffsets 16) × f̲loat k = -1.0 }
The maps: every map point through each format
The measures
ᵘa̲rm : Num a => a -> Float
The arm each point is given (the largest of its three outputs).
ᵘa̲rm ← { y → '+ r̲/₂ (f̲loat y = ('m̲ax r̲/₂ y) 'l̲eft t̲able o̲ffsets 3) × (o̲ffsets t̲ally y) 'r̲ight t̲able 1.0 2.0 3.0 }
ᵘm̲ean : Float -> Float
ᵘm̲ean ← { x → ('+ r̲/ r̲avel x) ÷ f̲loat t̲ally r̲avel x }
ᵘm̲easures : Float -> Float -> Float
For a format's weights k and its map y: accuracy on the test points, agreement with FP32 over the map, mean output error.
ᵘm̲easures ← { k y → acc ← ᵘm̲ean f̲loat pl = ᵘa̲rm test ᵘn̲et k agree ← ᵘm̲ean f̲loat (ᵘa̲rm y) = ᵘa̲rm y32 err ← ᵘm̲ean a̲bs y − y32 acc c̲at agree c̲at err }