Comparing Netflix (and most highly profitable computer businesses) to the world of producing AI models by training is not going to be fruitful. Netflix takes a lot of effort to operate but you can do on the small scale what Netflix does, quite directly. You can't replicate an AI model like ChatGPT-4 very easily unless you have all the data and huge compute that OpenAI does. Now, once the model has been produced, you can operate that model on the small scale with maybe less amazing results (see llama.cpp, etc) but producing the model is a scale problem like producing high quality steel. You can't escape the need for scale (without some serious technological developments first).
Netflix cheats. They send non-supercomputer boxes out to ISPs to install locally. If I could convince every ISP to install a bunch of my media servers people could watch my shows from anywhere in the US too.
I don't think compute cost has dropped by 1000x since 20 years ago. Maybe by 10 to 50x. And if you add in the demand for higher quality, the cost has probably increased. Like encoding a video for streaming 20 years ago, at that standard, may have cost roughly the same as it does today, or more, when you factor in the increases in resolution and quality.
My prediction is that training the latest model will continue to cost millions to tens of millions for a long time, and these costs may even increase dramatically if significantly more powerful models require proportional increase in training compute.
Unless of course we have some insane algorithmic breakthrough where we find an AI algorithm that blows llama2 out of the water for a small fraction of the compute.
