I don't really know anything about business, but something else I've wondered is this: if LLM scaling/progress really is exponential, and the juice is worth the squeeze, why is OpenAI investing significantly in everything that's not GPT-5? Wouldn't exponential growth imply that the opportunity cost of investing in something like Sora makes little sense?
A huge one for me is that Altman cries "safety" while pushing out everyone who actually cares about safety. Why? He desperately wants governments to build them a moat, yesterday if possible. He's not worried about the risks of AGI, he's afraid his company won't get there first because they're not making progress any more. They're rushing to productize what they have because they lost their only competitive advantage (model quality) and don't see a path towards getting it back.
I also don't thing the only way to improve LLM is by improving as zero shot inference. Did wrote any code in zero shot style that compiled and worked? It's a multistep process and probably agents and planning will be a next step for LLM.
Cheap inference help a lot in this case since you can give a task during the night to AI what you wanna do. Go to sleep then in the morning review the results. In this way AI is bruteforcing the solution by trying many different paths but that's kind of e.g. most programming works. You try many things until you don't have errors, code compiles and passes the tests.
I think this is really interesting, but I wonder if there really is enough data there to make a qualitative difference. I'm sure there's enough to make a better model, but I'm hesitant to think it would be better than an improved chatbot. What people are really waiting for is a qualitative shift, not a just an improved GPT.
> It's a multistep process and probably agents and planning will be a next step for LLM.
I agree, we definitely need a new understanding here. Right now, with the architecture we have, agents just don't seem to work. In my experience, if the LLM doesn't figure it out with a few shots, trying over and over again with different tools/functions doesn't help.
If they start scraping, training, and generating images and video, then they have lots more data to work with.
Now that would be super funny.
Civilization VII tech tree
AI singularity tech
Prerequisites: in order to research this, your world needs to have at least 100 billion college educated inhabitants.
:-)))
If I had these concerns as OpenAI, I'd be pushing hard to regulate and restrict generative image/video models, to push the end of the "low background data" era as far into the future as possible. i feel like the last thing I'd be doing is productizing those models myself!
They are! And I'm guessing maybe their perspective is if they can identify their own generative content, they can make a choice to ignore it and not cannibalize.
I don't actually think it's a bad one, but OpenAI didn't think that far ahead. They are pushing for regulation but that's mainly to screw over competing models, not to give them more data runway. Every capitalist is a temporarily embarrassed feudal aristocrat after all.
Furthermore, even if OpenAI had a perfect AI/human distinguisher oracle and could train solely on human output, that wouldn't get us superhuman reasoning or generalization performance. The training process they use is to have the machine mimic the textual output of humans. How exactly do you get a superhuman AGI[0] without having text generated by a superhuman AGI to train on?
[0] Note: I'm discounting "can write text faster than a human" as AGI here. printf in a tight loop already does that better than GPT-4o.
> And if you want to reach the goal as quickly as possible, you will try things in parallel
This is sort of the exploration-exploitation problem, right? But I think you'd agree that a company full of people who firmly believe that GPT-(n+1) will literally be AGI, and that we're on an exponential curve, will be fully in exploitation mode. In my mind, exploring methods of generating videos is not a path towards their stated goal of AGI. Instead, it's an avenue to _earn money now_. OpenAI is in a slightly awkward position: their main product (ChatGPT) is not super useful right now, and is facing increasingly viable competition.
You can spend 100% on the next generation or you can spend a small percentage to productize the previous generation to unlock revenue that can be spent on the next generation.
The latter will result in more investment into the next generation.
