- The training compute of notable AI models is doubling roughly every six months.
- Training compute costs are doubling every nine months for the largest AI models.
- Training compute has scaled up faster for language than vision.
- The size of datasets used to train language models doubles approximately every eight months.
- The length of time spent training notable models is growing.
- The power required to train frontier AI models is doubling annually.
- Leading AI companies have hundreds of thousands of cutting-edge AI chips.
Those are the conclusions for each section with some explanation for the data in each section. The implications seem to be that production of these LLMs is getting ever more costly, especially in terms of time and energy, even as training and algorithms become more efficient and more effective.
Is this all supply-side with the assumption of demand, or is the demand curve already rising to match?
https://huggingface.co/MeissonFlow/Meissonic
https://github.com/SWivid/F5-TTS
The reason the trend is that compute costs are doubling is because this is an arms race and everyone in the corporate space is prioritizing bigger models over better architecture in the pursuit of a breakaway. It is not indicative of a law ala Moore's Law.
1. What about new AI chip vendors?
2. How will the price of compute change?
3. How will the demand for compute change?
4. How will the overall supply of chips change?
