GPU Memory Math for LLMs: Formula That Tells You What Fits on Your GPU (opens in new tab)

(theahmadosman.substack.com)

12 pointsXMasterrrr5d ago2 comments

2 comments

This isn't very useful.

V of context is not equal across models.

Also, huggingface tells you how big the model is for the exact one you have in your hand, why the weird guesswork? Dynamic quants are not going to magically fit some formula.

metadat5d ago

This is super useful. Most of the time I go to run a model off Hugging Face on my 64GB MBP I run into issues where I drastically overestimated what it could do. :>

j / k navigate · click thread line to collapse

2 comments

DiabloD35d ago

This isn't very useful.

V of context is not equal across models.

Also, huggingface tells you how big the model is for the exact one you have in your hand, why the weird guesswork? Dynamic quants are not going to magically fit some formula.

metadat5d ago

This is super useful. Most of the time I go to run a model off Hugging Face on my 64GB MBP I run into issues where I drastically overestimated what it could do. :>

j / k navigate · click thread line to collapse