Addition is all you need for energy-efficient language models (opens in new tab)

What you're describing is called fixed point arithmetic, a super cool technique I wish more programmers knew about.

Proper finance related code should use it, but in my experience in that industry it doesn't seem very common unless you're running mainframes.

Funnily enough, I've seen a lot more fixed point arithmetic in software rasterizers than anywhere else. FreeType, GDI, WPF, WARP (D3D11 reference rasterizer) all use it heavily.

I recall playing with FRACTINT, which was a fractal generator that existed before floating point coprocessors were common, that used fixed point math to calculate and display fractals. That was back when fractals were super cool and everyone wanted to be in the business of fractals, and all the Nobel Prizes were given out to fractal researchers.

Ozaki has been doing fp64 matrix-multiplication using int8 tensor cores

https://arxiv.org/html/2306.11975v4

Interesting AF.

AFAIK this is still the best way to handle money/financial numbers.

That particular trick is known as fixed point arithmetic (not to be confused with a fixed point of a function)

this is still true for many embedded projects. like pi pico (2040) uses a table.

Sure, FRACTINT is called FRACTINT because it uses fixed-point ("integer") math. And fixed-point math is still standard in Forth; you can do your example in GForth like this:

    : organize; gforth
    Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
    Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
    Type `bye' to exit
    : %* d>s 10 m*/ ;  : %. <# # [char] . hold #s #> type ;  ok
    1.6 4.1 %* %. 6.5 ok

Note that the correct answer is 6.56, so the result 6.5 is incorrectly rounded. Here's how this works.

(If you're not familiar with Forth, Forth's syntax is that words are separated by spaces. "ok" is the prompt, ":" defines a subroutine terminated with ";", and you use RPN, passing parameters and receiving results on a stack.)

In standard Forth, putting a decimal point in a number makes it a double-precision number, occupying two cells on the stack, and in most Forths the number of digits after the decimal point is stored (until the next number) in the non-standardized variable dpl, decimal point location. Here I've just decided that all my numbers are going to have one decimal place. This means that after a multiplication I need to divide by 10, so I define a subroutine called %* to do this operation. (Addition and subtraction can use the standard d+ and d- subroutines; I didn't implement division, but it would need to pre-multiply the dividend by the scale factor 10.)

"%*" is defined in terms of the standard subroutine m*/, which multiplies a double-precision number by a single-precision number and divides the result by a divisor, and the standard subroutine d>s, which converts a double-precision number to a single-precision number. (There's probably a better way to do %*. I'm no Forth expert.)

I also need to define a way to print out such numbers, so I define a subroutine called "%.", using Forth's so-called "pictured numeric output", which prints out an unsigned double-precision number inserting a decimal point in the right place with "hold", after printing out the least significant digit. (In PNO we write the format backwards, starting from the least significant digit.) The call to "type" types out the formatted number from the hold space used by PNO.

Then I invoked %* on 1.6 and 4.1 and %. on its result, and it printed out 6.5 before giving me the "ok" prompt.

If you want to adapt this to use two decimal places:

    : %* d>s 100 m*/ ;  : %. <# # # [char] . hold #s #> type ; redefined %*  redefined %.   ok
    1.60 4.10 %* %. 6.56 ok

Note, however, that a fixed-point multiplication still involves a multiplication, requiring potentially many additions, not just an addition. The paper, which I haven't read yet, is about how to approximate a floating-point multiplication by using an addition, presumably because in multiplication you add the mantissas, or maybe using a table of logarithms.

Forth's approach to decimal numbers was a clever hack for the 01970s and 01980s on sub-MIPS machines with 8-bit and 16-bit ALUs, where you didn't want to be invoking 32-bit arithmetic casually, and you didn't have floating-point hardware. Probably on 32-bit machines it was already the wrong approach (a double-precision number on a 32-bit Forth is 64 bits, which is about 19 decimal digits) and clearly it is on 64-bit machines, where you don't even get out of the first 64-bit word until that many digits:

    0 1 %. 184467440737095516.16 ok

GForth and other modern standard Forths do support floating-point, but for backward compatibility, they treat input with decimal points as double-precision integers.

Pre-fill (even in the single batch case) and multi-batch decoding are still compute dominated. The oft repeated trope of "decoder only transformer inference is bottle-necked on memory bandwidth" is only strictly true in the single batch decoding case, because you're mostly doing vector matrix mults when the batch size is one.

Its worse than that: the energy gains are when comparing computations made with fp32, but for fp8 the multipliers are really tiny and the adder/shifters represent a largest part of the operators (energy-wise and area-wise) and this paper will only have small gains.

On fp8, the estimated gate count of fp8 multipliers is 296 vs. 157 with their technique, so the power gain on the multipliers will be much lower (50% would be a more reasonable estimation), but again for fp8 the additions in the dot products are a large part of the operations.

Overall, its really disingenuous to claim 80% power gain and small drop in accuracy, when the power gain is only for fp32 operations and the small drop in accuracy is only for fp8 operators. They don't analyze the accuracy drop in fp32, and don't present the power saved for fp8 dot product.

I'm also sure that fp8 is small enough that multiplication can really be done in a much simpler circuit than larger fp formats. Even smaller formats like fp4 would be able to just use a lookup table, and that makes them more like sort-of-standardized quantization schemes.

Sounds like the awesome architecture for transformers would be colocation of memory and compute.

That is true for single user/light inference only. For training and batch inference you can get compute bound fast enough.

Maybe this technique can be used for training then since that is a lot more compute intensive?

Imagine if you had a systolic array large enough that all the weights would only have to be loaded once at startup. Eliminating the memory-compute bottleneck of the von Neumann architecture could make this quite a bit more efficient.

Bro... they are NOT lightweight on compute!

GradIEEEnt half decent: The hidden power of imprecise lines [video] - https://news.ycombinator.com/item?id=36806970 - July 2023 (9 comments)

GradIEEEnt half decent - https://news.ycombinator.com/item?id=35780921 - May 2023 (32 comments)

I had hoped that they would reference this in their paper as some kind of "supporting previous exploration" but no, alas.

Log space is nice, multiplication can be replaced by addition.

This part is easy and anyone can implement hardware to do this. The tricky bit is always the staying in log space while doing accumulations, especially ones across a large range.

Yes, this is logarithmic number systems at work.

The paper has an odd feel about it to me too. Doing a gate estimation as a text explanation without a diagram makes it too easy to miss some required part. It wouldn't need to be a full gate level explanation but blocks labeled 'adder'.

Seeing the name de Vries in the first paragraph didn't help my sense of confidence either.

It actually is about 0.5 bits less efficient per weight in terms of precision/range, something the paper never highlights.

This could still be useful for battery constrained devices, right?

Redefining degrees to be 2pi = 256 was a pretty clever trick.

Paper?

https://research.nvidia.com/publication/2022-12_lns-madam-lo...

We will need a paper titled '"Considered Harmful" Articles is All You Need' to complete that cycle.

The reign of Threadripper!

Coal (and even wood!) powered cars actually existed long before Ford, but didn't take off because they were too heavy and unwieldly. The Model T was the result of a century of optimization.

https://en.wikipedia.org/wiki/Nicolas-Joseph_Cugnot

Also, making neural networks faster/cheaper is a big part of how they advance.

We've known about neural architectures since the 70s, but we couldn't build them big enough to be actually useful until the advent of the GPU.

Similarly, the LLM breakthrough was because someone decided it was worth spending millions of dollars to train one. Efficiency improvements lower that barrier for all future development (or alternatively, allow us to build even bigger models for the same cost.)

Cheaper compute is basically a prerequisite to making better models. You can get some improvements on the margins by making algorithms better with current hardware, but not an order of magnitude improvement.

When there is an order of magnitude improvement in hardware, the AI labs will figure out an algorithm to best take advantage of it.

The optimizations described could easily work on other models, not just transformers. Following your analogy, this is optimizing plumbing, pistons and valves on steam engines, it could be useful for whatever follows.

You're also welcome to contribute. There are many people doing many things at once in this space, I don't think experiments like this are a problem at all.

What if working well means making them efficient enough to run more 'neurons' on our current hardware?

I have to disagree. Nvidia spent a lot of effort on researching improved numerical representations. You can see a summary in this talk:

https://www.youtube.com/watch?v=gofI47kfD28

A lot of their work was published but went by unnoticed. But in fact the majority of their performance increase in new architecture is resulting from this work.

Reading between the lines, it seems that they came to the conclusion that a 4 bit representation with a group exponent ("FP4") is the most efficient representation of weights for inference. Reducing the number of bits in weights has the biggest impact on LLMs inference, since they are mostly memory bound. At these low bit numbers, the impact of using multiplication or other approaches is not really significiant anymore.

(multiplying a 4 bit wight with a larger activation is effectively 4 additions, barely more than what the paper proposes)

"Good enough" for what? We're in the middle of an AI arms race. Why do you believe people would choose to run the same LLMs on cheaper equipment instead of using the greater efficiency to train and run even larger LLMs?

Given LLM performance seems to scale with their size, this would result in more powerful models, which would grow the applicability, use and importance of AI, which would in turn grow the use and importance of Nvidia's hardware.

So this theory doesn't really stack up for me.

It's still a massively parallel problem suited to GPUs, whether it's float or int, or addition or multiplication doesn't really matter.

If an addition-only LLM performed better, nvidia would probably still be the market leader.

Next gen nvidia chips would have more adders and fewer multipliers.

Google & Apple already run custom chips, Meta and MS are deploying their own soon too. Your theory is that none of them have researched non-matrix-multiplication solutions before investing billions?

I’d estimate that fraction of Nvidia’s dominance that’s dependent on their distinctive advantages in kernel primitives (add vs. multiply) would be a rounding error in FP8.

The CUDA tooling and ecosystem, VLSI architecture, organizational prowess… all matter at multiple orders of magnitude more.

NVidia GPUs support integer operations specifically for use with deep learning models.

So let me get this straight. Universities don’t want to show that Nvidia gpus are obsolete, so they can receive a steady stream of obsolete gpus? For what possible reason, that doesn’t make sense.

no matter how fast cpu, network and browser has become, websites are still slow. we will run out of data to train much earlier than people will stop inventing even larger models.

Alternatively, other people fund LLM research

> I have a pet theory

You mean you have a conspiracy theory.

Why wouldn't other companies that buy Nvidia GPU fund these researches? It would greatly cut their cost.