undefined | Better HN

0 pointspmontra2y ago0 comments

I wish that all these repos were more clear about the hardware requirements. Seeing that it runs on a 8 GB Raspberry, probably with abysmal performance, I'd say that it will run on my 32 GB Intel laptop on the CPU. Will it run on its Nvidia card? I remember that the rule of thumb was one GB of GPU RAM per G parameters, so I'd say that it won't run. However this has 4 bit quantization so it could have lower requirements.

Of course the main problem is that I don't know enough about the subject to reason on it on my own.

0 comments

dkjaudyeqooe2y ago

Roughly speaking I believe it's the number of parameters times the size of the parameters. So in the 4 bit case it's half a gigabyte per billion parameters.

From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.

j / k navigate · click thread line to collapse

0 pointspmontra2y ago0 comments

Of course the main problem is that I don't know enough about the subject to reason on it on my own.

0 comments

dkjaudyeqooe2y ago

Roughly speaking I believe it's the number of parameters times the size of the parameters. So in the 4 bit case it's half a gigabyte per billion parameters.

From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.

j / k navigate · click thread line to collapse