undefined | Better HN

0 pointsivan_gammel1mo ago0 comments

When output is good enough, other considerations become more important. Most people on this planet cannot afford even an AI subscription, and cost of tokens is prohibitive to many low margin businesses. Privacy and personalization matter too, data sovereignty is a hot topic. Besides, we already see how focus has shifted to orchestration, which can be done on CPU and is cheap - software optimizations may compensate hardware deficiencies, so it’s not going to be frozen. I think the market for local hardware inference is bigger than for clouds, and it’s going to repeat Android vs iOS story.

0 comments

bigyabai1mo ago

This is the same justification that was used to ship the (now almost entirely defunct) NPUs on Apple and Android devices alike.

The A18 iPhone chip has 15b transistors for the GPU and CPU; the Taalas ASIC has 53b transistors dedicated to inference alone. If it's anything like NPUs, almost all vendors will bypass the baked-in silicon to use GPU acceleration past a certain point. It makes much more sense to ship a CUDA-style flexible GPGPU architecture.

ivan_gammelOP1mo ago

Why are you thinking about phones specifically? Most heavy users are on laptops and workstations. On smartphones there might be a few more innovations necessary (low latency AI computing on the edge?)

bigyabai1mo ago

Many laptops and workstations also fell for the NPU meme, which in retrospect was a mistake compared to reworking your GPU architecture. Those NPUs are all dark silicon now, just like these Taalas chips will be in 12-24 months.

Dedicated inference ASICs are a dead end. You can't reprogram them, you can't finetune them, and they won't keep any of their resale value. Outside cruise missiles it's hard to imagine where such a disposable technology would be desirable.

1 more reply

wmf1mo ago

Taalas is more expensive than NPUs not less. You have GPU/NPU at home; just use it.

ivan_gammelOP1mo ago

I feel weird defending Taalas here, but this argument is quite strange: of course it is more expensive now. It is irrelevant - all innovations are expensive at early stage. The question is, what this technology will cost tomorrow? Can it do for consumers what NPUs could not, offering good UX and quality of inference for reasonable price?

wmf1mo ago

It will always be more expensive.

1 more reply

j / k navigate · click thread line to collapse

0 pointsivan_gammel1mo ago0 comments

0 comments

bigyabai1mo ago

This is the same justification that was used to ship the (now almost entirely defunct) NPUs on Apple and Android devices alike.

ivan_gammelOP1mo ago

bigyabai1mo ago

1 more reply

wmf1mo ago

Taalas is more expensive than NPUs not less. You have GPU/NPU at home; just use it.

ivan_gammelOP1mo ago

wmf1mo ago

It will always be more expensive.

1 more reply

j / k navigate · click thread line to collapse