That's an utterly bizarre notion. The answer in question never existed before.
By your definition humans never produce anything new either, because we always also extrapolate on patterns from our previous knowledge.
> it fails spectacularly for exotic programming languages.
My experience is that it not just succeeds for "exotic" languages, but for languages that didn't exist prior to the prompt.
In other words, they can code at least simple programs even with zero-shot by explaining semantics of a language without giving them even a single example of programs in that language.
Did you even read the comment you replied to above?
To quote myself: "Invent a programming language that does not exist."
I've had this work both for "from scratch" descriptions of languages by providing grammars, and for "combine feature A from language X, and feature B from language Y". In the latter case you might have at least an argument. In the former case you do not.
Most humans struggle with tasks like this - you're setting a bar for LLMs most humans would fail to meet.
If you tell someone that multiplying a number by 2 is adding the number to itself, then if this person knows addition, you can't be surprised if it tells you that 9*2 is 18. A small leap in discovery is when the person can extract the pattern and gives you 5*3 is 5+5+5. A much bigger leap is when the person discovers exponent.
But if you take the time to explain each concept....
Yes, but it didn't exist during training. Nothing in the training data would provide pre-existing content for the model to produce from, so the output would necessarily be new.
> But if you take the time to explain each concept....
Based on the argument you presented, nothing a human does is new, because it is all based on our pre-exististing learned rules of language, reasoning, and other subjects.
See the problem here? You're creating a bar for LLMs that nobody would reasonably assign to humans - not least because if you do, then "accusing" LLMs of the same does not distinguish them from humans in any way.
If that is the bar you wish to use, then for there to be any point to this discussion, you will need to give a definition of what it means to create something new that we can objectively measure that a human can meet that you believe an LLM can't even in theory meet, otherwise the goalpost will keep being moved when an LLM example can be shown to be possible.
