undefined | Better HN

0 pointsliuliu5y ago0 comments

They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.

0 comments

p1esk5y ago

They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.

j / k navigate · click thread line to collapse

0 pointsliuliu5y ago0 comments

They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.

p1esk5y ago

They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.

j / k navigate · click thread line to collapse