> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
I'm curious whether they have continued to scale up model size/compute significantly or if they have managed to make significant innovations there.
I just skimmed the paper but seems they are also omitting details about how they actually feed the images in too, which is a shame as a curious outside observer.
Conversely, if all actors are given equal access at the same time, no such lone bad actor can be in a position to maintain a hidden advantage.
OpenAI's actions continue to be more than merely annoying.
It's not a zero-sum game where you can level the playing field and say everything's good.
Leveling the playing field won't instantly make everyone safe, but leaving it uneven certainly doesn't either.
What you are looking for is a publication known as "Industrial Society and Its Future"
> 1995 anti-technology essay by Ted Kaczynski… contends that the Industrial Revolution began a harmful process of natural destruction brought about by technology, while forcing humans to adapt to machinery, creating a sociopolitical order that suppresses human freedom and potential.
People have spilled a lot more ink than that on this subject! And most of them weren't also terrorists.
> A minority of the problems in the exams were seen by the model during training
A minority can be 49%. They do mention they tested against newly available practice exams, but those are often based on older real exam questions which may have been discussed extensively in forums that were in the training data. Now that it is for-profit ClosedAI we have to somewhat treat each claim as if it were made adversarially, assuming minority may mean 49% when it would benefit them one way and .1% when it serves their look better for sales pitch to the Microsoft board, etc.
It is unsafe to not release the source along with the service. That incentivizes competitors to sacrifice their own safety research in favor of speed to market. Instead of getting shared safe tools, we get a bunch of for profit corporations pushing their proprietary unsafe tools.
Preventing this situation was the original reason to setup OpenAI. Speed run to the dark side.
It's almost certainly a VQ-VAE-style encoding of the image itself into a sequence of tokens, as was done by DALL-E 1, CM3, Gato and a whole bunch of more recent models. It's the very obvious thing to do, and their context window is more than large enough now.
Safety has nothing to do with it. It's an easy tack on for them because of popular fear of AGI.
It's all about power over the market.
Cringe.
Let's be honest, the real reason for closeness is the former.
As a beginner in the NLP world, this may serve me a purpose which is to hide the complexity behind building such models.. numbers like xyzB parameters, 12K A100s.. are scary, so I still can dream of building one system one day. This story [0] and this one [1] hide some extremely complex edge cases that a beginner will never though of or had the courage to start if he knew what is the real cost.
We may, however, still be able to infer some details [probably in the future] knowing how Microsoft had re-arranged its infrastructure to welcome OpenAI training [2]
_________________
[0]. https://www.construct.net/en/blogs/ashleys-blog-2/simple-sof...
[1]. https://prog21.dadgum.com/29.html
[2]. https://www.theverge.com/2023/3/13/23637675/microsoft-chatgp...
