undefined | Better HN

0 pointsKeyframe6mo ago0 comments

That's AMD Ryzen AI Max+ 395, right? Lots of those boxes popping up recently, but isn't that dog slow? And I can't believe I'm saying this - but maybe RAM filled-up mac might be a better option?

0 comments

loudmax6mo ago

A GMKtec or a Framework desktop with a Strix Halo/AI Max CPU is about the cheapest way to run a model that needs to fit into about 120GB of memory. Macs have twice the memory bandwidth of these units, so will run significantly faster, but they're also much more expensive. Technically, you could run these models on any desktop PC with 128GB of RAM, but that's a whole different level of "dog slow." It really depends on how much you're prepared to pay to run these bigger models locally.

whitehexagon6mo ago

from the .de website I see 2000eur for the 128GB, But looking at the shipping info, it sounds like it might still be shipped from .cn: ´Please ensure you can handle customs clearance and taxes yourself.´

Also it is Windows 11 which is a big No from me.

But if this is the start of the local big model capable hardware it looks quite hopeful. A 2nd hand M2 128GB studio (which I can use Asahi on) is currently ~3600eur

https://es.wallapop.com/item/mac-studio-m2-ultra-1tb-ssd-128...

ricardobeat6mo ago

Yes, but the mac costs 3-4x more. You can get one of these 395 systems with 96GB for ~1k.

KeyframeOP6mo ago

When I was looking it was more like 1.6k euros, but still great price. Mac studio with M4 Max 16/40/16 with 128GB is double that. That's all within a range of "affordable". Now, if it's at least twice the speed, I don't see a reason not to. Even though my religion is against buying a mac as well.

edit, just took a look at amazon. GMKtec EVO-X2 AI, which is the AMD Ryzen AI Max+ 395 with 128GB of RAM is 3k euros. Mac M4 Max with 16 cores and 128 gigs is 4.4k euros. Damn, Europe. If you go with M4 Max with 14 cores, but still 16 cores of "Neural engine"... ah, you can't get 128 GB of RAM then. Classic Apple :)

edit2: look at gmktec site itself. machine is 2k euros there. Damn, amazon.

mhast6mo ago

I have a M4 Max with 128 GB memory. Even on that machine I would not consider 70B+ models to be useable. Once you go below 20 tokens/s it becomes more like having a pen pal than an AI assistant.

MoE models can still be pretty fast. As are smaller models.

(This is mostly a warning for anyone who is enamored by the idea of running these things locally to make sure to test it before you spend a lot of money.)

Currently I'd probably say the Nvidia RTX pro 6000 is a Challenger if you want local models. It "only" has 96 GB of RAM, but it's very fast (1800 GB/s). If you can fit the model on it and it's good enough for your use case then it's probably worth it even at $10k.

1 more reply

te00066mo ago

Note that the small print on the GMKtec site says that prices do not include customs and VAT. Which seem to amount to 19% in the EU. So, almost 2,4K€.

1 more reply

A4ET8a8uTh0_v26mo ago

I will admit that the price difference was a big value differentiator for me since speed is not a priority ( playing with big models at a reasonable price is ).

buyucu6mo ago

I'm not buying a Mac. Period.

j / k navigate · click thread line to collapse

0 comments

loudmax6mo ago

whitehexagon6mo ago

Also it is Windows 11 which is a big No from me.

But if this is the start of the local big model capable hardware it looks quite hopeful. A 2nd hand M2 128GB studio (which I can use Asahi on) is currently ~3600eur

https://es.wallapop.com/item/mac-studio-m2-ultra-1tb-ssd-128...

ricardobeat6mo ago

Yes, but the mac costs 3-4x more. You can get one of these 395 systems with 96GB for ~1k.

KeyframeOP6mo ago

edit2: look at gmktec site itself. machine is 2k euros there. Damn, amazon.

mhast6mo ago

I have a M4 Max with 128 GB memory. Even on that machine I would not consider 70B+ models to be useable. Once you go below 20 tokens/s it becomes more like having a pen pal than an AI assistant.

MoE models can still be pretty fast. As are smaller models.

(This is mostly a warning for anyone who is enamored by the idea of running these things locally to make sure to test it before you spend a lot of money.)

1 more reply

te00066mo ago

Note that the small print on the GMKtec site says that prices do not include customs and VAT. Which seem to amount to 19% in the EU. So, almost 2,4K€.

1 more reply

A4ET8a8uTh0_v26mo ago

I will admit that the price difference was a big value differentiator for me since speed is not a priority ( playing with big models at a reasonable price is ).

buyucu6mo ago

I'm not buying a Mac. Period.

j / k navigate · click thread line to collapse