undefined | Better HN

0 pointsricardobeat2y ago0 comments

As someone else pointed out here, you can store 5 ternary values in 1 byte, 3^5 == 243.

0 comments

That’s still not 1 bit, and that would basically destroy whatever perf advantage you might hope to get if you want to keep the model in memory in that format rather than unpack it on load.

marty18852y ago

Not fully, 8 bits has 256 values. It's easy to keep a look up table in the L1 cache of any CPU and constant cache of any GPU. For ASICs and FPGAs, it's a simple 256-value LUT. It's not ideal, yes, but not a deal breaker. Epically considering LLMs are memory bound. GGML dequantizes weights on-the-fly and still gets near linear scaling on GPUs.

j / k navigate · click thread line to collapse

0 comments

ein0p2y ago

That’s still not 1 bit, and that would basically destroy whatever perf advantage you might hope to get if you want to keep the model in memory in that format rather than unpack it on load.

marty18852y ago

j / k navigate · click thread line to collapse