People don't need the gigantic amount of input data that ChatGPT needs to learn. However I'm not sure what exactly "this" is you and GP are referring to, and it may be possible to improve existing ideas so that it works with less input data.
You are not storing it all, but you are finding patterns and correlations in it from the day you were born. This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning. It’s fast, but it depends on a deep base of prior knowledge.
And even if I did, most of it would not be of the same category as what LLMs are learning; information about how a fabric feels or the sound of an ambulance doesn't teach me anything about programming or other things GPT can do. So when comparing the inputs, it makes no sense to count all the information that our senses are getting, to all the input for an LLM.
>This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning
Not really comparable
I think you discount general knowledge acquired non-verbally too easily. The sound of an ambulance and how it moves through space and how it correlates with visual information represents an astronomical amount of information that can be generalized from handsomely. All these data streams have to be connected, they correlate and intertwine in fantastically complex ways.
I think the sound of an ambulance does teach you things that eventually help you “program”. You have witnessed similar (non-)events thousands if not millions of times. Each time it was accompanied with shitloads of sensory data from all modalities, both external and internal.
The base of general patterns you start out from once you are ready for language is staggering.
Again not saying LLMs work like that, because they do not. All I mean to do is put their information requirements in perspective. We ingest a lot more than a bunch of books.
Sure they do. Humans rely on tons of audio and video before they can even read (or walk).
For reading, the same applies. Our brains are equipped with many of the foundational aspects required for reading, and we only _learn_ a part what is necessary for the skill of reading.
Unlike computer models, brains are no tabula rasa. So we don't need the same input as computer models to learn.
The training data isn't the input. It's a part of the algorithm.
The common perception has been that children aren't exposed to enough data to arrive at their grammatical language skills, implying there's some proto language built in. Comparative analysis of languages has looked for what aspects are truly universal but there's actually not a lot of concrete true universals to ascribe to our genetic innate language.
But if it is genetic that doesn't really mean it's fundamentally different than ChatGPT, it just took a different and drastically longer training period and then transfer learning when children learn their mother tongue.
It doesn't necessarily mean it's fundamentally different, but it doesn't mean it is comparable either. Geoff Hinton doesn't think the brain does backpropagation. Training a neural net uses backpropagation. So if Hinton is correct then saying "it just took a longer training period" while brains doesn't learn like our current neural nets is glossing over a lot of things.
