undefined | Better HN

0 pointsswyx2y ago0 comments

its a (minor, not permanent) blow to open source ai that stability, and now mistral, felt the need to switch to closed for their leading models. openwashing is a crowdpleaser but everyone goes closed when things get serious. basically only meta is the leading hope. i hope more players emerge - but i also dont have a ton of ideas on how to fund them. these efforts take serious resources.

that said i think its impt to acknowledge how much stability has shared in its research, just the other day they were on HN for Stable Video 3D, not to mention hourglass diffusion and other Stable* models. may not be the overwhelming SOTA but its real open source AI work that pushes the frontiers. you have to give them credit for that.

0 comments

credit_guy2y ago

> these efforts take serious resources

Meta just published their new optimization results [1]. According to them

  > training a 7B model on 512 GPUs to 2T tokens using this method would take just under two weeks.

In this context a GPU is an NVIDIA A100, which you can buy, if you can buy, for $10000.

And this is after an explosion of ideas that lead to unthinkable optimizations just two years ago.

If someone did train such a model 2 years ago, it would have cost hundreds of millions. Now it's 5 million. Maybe in 2 years it's going to be only $50k. Should you start a startup now and invest $5 million, an risk someone stealing the show for pennies in 2 years? If you do, I really can't see if you can afford to open source the results of your training.

[1] training a 7B model on 512 GPUs to 2T tokens using this method would take just under two weeks.

j / k navigate · click thread line to collapse