What would it take to bring an open source model like Neo to OpenAI GPT levels in terms of resources around the world donating computing power?
How many people are we talking about and how long would it take? Is it even possible to coordinate something like that.
With so many crypto mining rigs left to rust around the planet one would think this could be a good cause to put them back to work.
Benchmarks are one thing but I suspect if any were truly on par we would know about it.
Obviously it's partly compute cost, but I also suspect there's a lot of R&D that would need to be redone in the open.
How many tricks does openai's pretraining have that aren't found in some paper somewhere?
Training big models takes a lot of random reads/writes and those tend to be pretty latency sensitive. There _may_ be a way to train this BitTorrent style with donated compute, but it's hard to say how many orders of magnitude slower that would be. (Do you need 2x more compute to do it distributes? 10? 100x?)
It is an interesting question to be able to explore this space more!
Hugging Face and other collaborators led the creation of BLOOM. Given how hot the competition is, I imagine some team will take advantage of this soon...
Unless a team has been working on a competitor for a while, we're only going to see lacklustre fine-tunes of existing older models that can barely take on GPT-3.5 for the next few months.
