Kaist develops next-generation ultra-low power LLM accelerator (opens in new tab)

(en.yna.co.kr)

97 pointsreadline_prompt2y ago38 comments

38 comments

Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/

They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/

moffkalast2y ago

It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.

jiggawatts2y ago

Or to put it another way: they’ve made a compute substrate with the correct ratios of processing power to memory capacity.

NVIDIA GPUs were optimised for different workloads, such as 3D rendering, that have different optimal ratios.

This “supercomputer” isn’t brute force or wasteful because it allows more requests per second. By having each response get processed faster it can pipeline more of them through per unit time and unit silicon area.

2 more replies

ben-schaaf2y ago

> pure L3 cache with a whopping 230 MB per card

Just to put these numbers in perspective a desktop 8 core 7800x3d has 96MB of L3 cache, and the top-end 96-core Epyc 9684X has 1.15GB of L3.

LoganDark2y ago

They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo.

I imagine Mixtral by itself would only take something like 200-300 LPUs

1 more reply

hackerlight2y ago

What's the cost per inference relative to H100? Isn't that the number to care about?

2 more replies

imtringued2y ago

Grayskull has 96 MB SRAM and people call it overpriced at $600 to $800. It is far more plausible that their chip costs are somewhere around $500.

moralestapia2y ago

Nothing wrong with that, though.

moffkalast2y ago

> The 4.5-mm-square chip, developed using Korean tech giant Samsung Electronics Co.'s 28 nanometer process, has 625 times less power consumption compared with global AI chip giant Nvidia's A-100 GPU, which requires 250 watts of power to process LLMs, the ministry explained.

>processes GPT-2 with an ultra-low power consumption of 400 milliwatts and a high speed of 0.4 seconds

Not sure what's the point on comparing the two, an A100 will get you a lot more speed than 2.5 tokens/sec. GPT 2 is just a 1.5B param model, a Pi 4 would get you more tokens per second with just CPU inference.

Still, I'm sure there's improvements to be made and the direction is fantastic to see, especially after Coral TPUs have proven completely useless for LLM and whisper acceleration. Hopefully it ends up as something vaguely affordable.

dloss2y ago

Which of the model requirements of Coral TPUs [1] are the most problematic for LLMs?

[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...

semisight2y ago

Guessing as to what the GP meant--coral TPUs max out around 8M parameters, IIRC. That's a few orders of magnitude less than the smallest LLM model.

moffkalast2y ago

The part where they have like 3 bytes of memory so you switch from extremely high latency of RAM to laughably sluggish latency of USB serial. I think there's also no support below 8 bit quants, which you'd really need.

zachbee2y ago

Neuromorphic computing is cool, but not new tech. However, using a neuromorphic spiking architecture to run LLMs seems new. Unfortunately, there doesn't seem to be a paper associated with this work, so there's no deeper information on what exactly they're doing.

imtringued2y ago

I heavily doubt that they are running LLMs on this.

zachbee2y ago

The article says they ran GPT-2! Which isn't particularly large, but replicating a large language model with a spiking neural network seems like novel work at least.

sroussey2y ago

Is their work similar to Rain.ai then?

dartos2y ago

> New structure mimics the layout of neurons and synapses

What does that mean, practically?

How can you mimic that layout in silicon?

zachbee2y ago

Neuromorphic computing basically uses individual "neurons", represented with either analog or digital circuits, which communicate using asynchronous pulses called "spikes". Unlike the human brain, neuromorphic chips are 2D, but we can replicate a good amount of neural dynamics in silicon.

It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures.

https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/Ben...

p1esk2y ago

This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.

colinator2y ago

Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part.

I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?

1 more reply

imtringued2y ago

Spiking neural networks are not software, they are usually built directly into silicon chips because they are using pulse timing to encode information instead of multiple bits. The problem is that training them is difficult because they operate over time, not that they don't work. As of now, scaling training infrastructure is more important than theoretical power efficiency.

PeterStuer2y ago

Quick shoutout to https://youtube.com/@TechTechPotato for those interested in keeping tabs on the AI hardware space. There is much more going on in this area than you would think if you only follow general media.

bglazer2y ago

The article says 400 milliwatt power draw.

Wolfram Alpha says thats roughly equivalent to cell phone power draw when sleeping.

pavelstoev2y ago

We build software acceleration for LLM, effectively running smaller llama2 models at the same performance on several L4's as on 1xA100.

j / k navigate · click thread line to collapse

38 comments

geuis2y ago

Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/

They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/

moffkalast2y ago

jiggawatts2y ago

Or to put it another way: they’ve made a compute substrate with the correct ratios of processing power to memory capacity.

NVIDIA GPUs were optimised for different workloads, such as 3D rendering, that have different optimal ratios.

2 more replies

ben-schaaf2y ago

> pure L3 cache with a whopping 230 MB per card

Just to put these numbers in perspective a desktop 8 core 7800x3d has 96MB of L3 cache, and the top-end 96-core Epyc 9684X has 1.15GB of L3.

LoganDark2y ago

They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo.

I imagine Mixtral by itself would only take something like 200-300 LPUs

1 more reply

hackerlight2y ago

What's the cost per inference relative to H100? Isn't that the number to care about?

2 more replies

imtringued2y ago

Grayskull has 96 MB SRAM and people call it overpriced at $600 to $800. It is far more plausible that their chip costs are somewhere around $500.

moralestapia2y ago

Nothing wrong with that, though.

moffkalast2y ago

>processes GPT-2 with an ultra-low power consumption of 400 milliwatts and a high speed of 0.4 seconds

dloss2y ago

Which of the model requirements of Coral TPUs [1] are the most problematic for LLMs?

[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...

semisight2y ago

Guessing as to what the GP meant--coral TPUs max out around 8M parameters, IIRC. That's a few orders of magnitude less than the smallest LLM model.

moffkalast2y ago

zachbee2y ago

imtringued2y ago

I heavily doubt that they are running LLMs on this.

zachbee2y ago

The article says they ran GPT-2! Which isn't particularly large, but replicating a large language model with a spiking neural network seems like novel work at least.

sroussey2y ago

Is their work similar to Rain.ai then?

dartos2y ago

> New structure mimics the layout of neurons and synapses

What does that mean, practically?

How can you mimic that layout in silicon?

zachbee2y ago

https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/Ben...

p1esk2y ago

This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.

colinator2y ago

Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part.

1 more reply

imtringued2y ago

PeterStuer2y ago

bglazer2y ago

The article says 400 milliwatt power draw.

Wolfram Alpha says thats roughly equivalent to cell phone power draw when sleeping.

pavelstoev2y ago

We build software acceleration for LLM, effectively running smaller llama2 models at the same performance on several L4's as on 1xA100.

j / k navigate · click thread line to collapse