undefined | Better HN

0 pointswpm1y ago0 comments

AFAIK there is no general purpose, "do this on the ANE" API. You have to be using specific higher level APIs like CoreML or VisionKit in order for it to end up on the ANE.

0 comments

bt1a1y ago

This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

sanxiyn1y ago

It works quite well as long as you don't care about battery.

j / k navigate · click thread line to collapse

0 pointswpm1y ago0 comments

AFAIK there is no general purpose, "do this on the ANE" API. You have to be using specific higher level APIs like CoreML or VisionKit in order for it to end up on the ANE.

0 comments

bt1a1y ago

This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

sanxiyn1y ago

It works quite well as long as you don't care about battery.

j / k navigate · click thread line to collapse