undefined | Better HN

0 pointsadamsmith1432y ago0 comments

Actually OP is clearly, ironically, parroting the stochastic parrot idea that LLMs are incapable of anything other than basic token prediction and dismissing any of their other emergent abilities.

0 comments

srslack2y ago

It can't generalize and adapt outside of its corpus, not in a way that's correct anyhow, and there's nothing "emergent." They are incapable of anything other than token prediction on its corpus and context. it just produces really good predictions. Funny how everyone keeps citing that Microsoft paper, when Microsoft is who is lobbying for this regulatory capture, and it's already been shown that such emergence on the tasks they chose when you scale up was a "mirage."

adamsmith143OP2y ago

This is demonstrably wrong. It can clearly generate unique text not from it's training corpus and can successfully answer logic based questions that were also not in it's training corpus.

Another paper not from Msft showing emergent task capabilities across a variety of LLMs as scale increases.

https://arxiv.org/pdf/2206.07682.pdf

You can hem and haw all you want but the reality is these models have internal representations of the world that can be probed via prompts. They are not stochastic parrots no matter how much you shout in the wind that they are.

comp_throw72y ago

Yes, and neither could GPT-3, which is why we don't observe any differences between GPT-3 and GPT-4. Right?

Tell me: how does this claim _constrain my expectations_ about what this (or future) models can do? Is there a specific thing that you predicted in advance that GPT-4 would be unable to do, which ended up being a correct prediction? Is there a specific thing you want to predict in advance of the next generation, that it will be unable to do?

woeirua2y ago

Spoiler alert: they're actually both LLMs arguing with one another.

jazzyjackson2y ago

yea but that's a boring critique and not the point they were making - whether or not LLMs reason or parrot has no relevance to whether Mr Altman should be the one building the moat.

j / k navigate · click thread line to collapse