undefined | Better HN

0 pointskamranjon6mo ago0 comments

Don’t all of the M series chips contain neural cores?

0 comments

Yes, they do. They're called Neural Engine, aka NPUs. They aren't being used for local LLMs on Macs because they are optimized for power efficiency running much smaller AI models.

Meanwhile, the GPU is powerful enough for LLMs but has been lacking matrix multiplication acceleration. This changes that.

astrange6mo ago

The neural engine is used for the built-in LLM that does text summaries etc., just not third party LLMs.

And there's an official port of Stable Diffusion to it: https://github.com/apple/ml-stable-diffusion

mrheosuper6mo ago

I thought 1 of the reason we do ML on GPU is fast Matrix multiplication ?

So the new engine is accelerator for matmul accelerator ?

wtallis6mo ago

From a compute perspective, GPUs are mostly about fast vector arithmetic, with which you can implement decently fast matrix multiplication. But starting with NVIDIA's Volta architecture at the end of 2017, GPUs have been gaining dedicated hardware units for matrix multiplication. The main purpose of augmenting GPU architectures with matrix multiplication hardware is for machine learning. They aren't directly useful for 3D graphics rendering, but their inclusion in consumer GPUs has been justified by adding ML-based post-processing and upscaling like NVIDIA's various iterations of DLSS.

cchance6mo ago

These are different these are built into the GPU Cores

j / k navigate · click thread line to collapse