undefined | Better HN

0 pointsGalanwe22h ago0 comments

The 5090 is crap for inference. Unless you like dummy models, sure they will run at light speed. All the rage is MoE with 500B-1T weights nowadays.

0 comments

zozbot23420h ago

MoE is fine. You can put the shared weights on the 5090 (will fit handily even for the largest models) and expert weights on CPU, possibly with weights offload from storage.

j / k navigate · click thread line to collapse

0 comments

zozbot23420h ago

MoE is fine. You can put the shared weights on the 5090 (will fit handily even for the largest models) and expert weights on CPU, possibly with weights offload from storage.

j / k navigate · click thread line to collapse