undefined | Better HN

0 pointswkat42421y ago0 comments

I think it's mostly the memory bandwidth though that makes the GPUs so fast with LLMs. My card does about 1TB/s. CPU RAM won't come near that. I'm sure a lot of optimisations can be had but I think GPUs will still be significantly ahead.

Macs are so good at it because Apple solder the memory on top of the SoC for a really wide and low latency connection.

0 comments

simpaticoder1y ago

This is a good and valid comment. It is difficult to predict the future, but I would be curious what the best case theoretical performance of an LLM on a typical x86 or ARM system with DDR4 or DDR5 RAM. My uneducated guess is that it can be very good, perhaps 50% the speed of a specialized GPU/RAM device. In practical terms, the CPU approach is required for very large contexts, up to as large as the lifetime of all interactions you have with your LLM.

j / k navigate · click thread line to collapse

0 pointswkat42421y ago0 comments

Macs are so good at it because Apple solder the memory on top of the SoC for a really wide and low latency connection.

0 comments

simpaticoder1y ago

j / k navigate · click thread line to collapse