undefined | Better HN

0 pointsoblio8mo ago0 comments

What hardware advancement? There's hardly any these days... Especially not for this kind of computing.

0 comments

Have you heard of TPUs?

Sort of a hardware advancement. I'd say it's more of a sidegrade between different types of well-established processor. Take out a couple cores, put in some extra wide matrix units with accumulators, watch the neural nets fly.

But I want to point out that going from CPU to TPU is basically the opposite of a Moore's law improvement.

oblioOP8mo ago

Yeah, I'm a regular Joe. How do I get one and how much does it cost?

Dylan168078mo ago

If your goal is "a TPU" then you buy a mac or anything labeled Copilot+. You'll need about $600. RAM is likely to be your main limit.

(A mid to high end GPU can get similar or better performance but it's a lot harder to get more RAM.)

2 more replies

haiku20778mo ago

Specifically, I upgraded my mac and ported my software, which ran on Windows/Linux, to macos and Metal. Literally >100x faster in benchmarks, and overall user workflows became fast enough I had to "spend" the performance elsewhere or else the responses became so fast they were kind of creepy. Have a bunch of _very_ happy users running the software 24/7 on Mac Minis now.

oblioOP8mo ago

The thing is, these kinds of optimizations happen all the time. Some of them can be as simple as using a hashmap instead of some home-baked data structure. So what you're describing is not necessarily some LLM specific improvement (though in your case it is, we can't generalize to every migration of a feature to an LLM).

And nothing I've seen about recent GPUs or TPUs, from ANY maker (Nvidia, AMD, Google, Amazon, etc) say anything about general speedups of 100x. Heck, if you go across multiple generations of what are still these very new types of hardware categories, for example for Amazon's Inferentia/Trainium, even their claims (which are quite bold), would probably put the most recent generations at best at 10x the first generations. And as we all know, all vendors exaggerate the performance of their products.

j / k navigate · click thread line to collapse

0 comments

Sebguer8mo ago

Have you heard of TPUs?

Dylan168078mo ago

But I want to point out that going from CPU to TPU is basically the opposite of a Moore's law improvement.

oblioOP8mo ago

Yeah, I'm a regular Joe. How do I get one and how much does it cost?

Dylan168078mo ago

If your goal is "a TPU" then you buy a mac or anything labeled Copilot+. You'll need about $600. RAM is likely to be your main limit.

(A mid to high end GPU can get similar or better performance but it's a lot harder to get more RAM.)

2 more replies

haiku20778mo ago

oblioOP8mo ago

j / k navigate · click thread line to collapse