Ask HN: You have 50k USD and want build an inference rig without GPUs. How?

3 points4k1y ago1 comments

This is more like a thought experiment and I am hoping to learn the other developments in the LLM inference space that are not strictly GPUs.

Conditions:

1. You want a solution for LLM inference and LLM inference only. You don't care about any other general or special purpose computing

2. The solution can use any kind of hardware you want

3. Your only goal is to maximize the (inference speed) X (model size) for 70b+ models

4. You're allowed to build this with tech mostly likely available by end of 2025.

How do you do it?

1 comments

j / k navigate · click thread line to collapse

3 points4k1y ago1 comments

This is more like a thought experiment and I am hoping to learn the other developments in the LLM inference space that are not strictly GPUs.

Conditions:

1. You want a solution for LLM inference and LLM inference only. You don't care about any other general or special purpose computing

2. The solution can use any kind of hardware you want

3. Your only goal is to maximize the (inference speed) X (model size) for 70b+ models

4. You're allowed to build this with tech mostly likely available by end of 2025.

How do you do it?