undefined | Better HN

0 pointsvidarh10mo ago0 comments

> LLMs can only give you code that somebody has wrote before. This is inherent.

This is trivial to prove to be false.

Invent a programming language that does not exist. Describe its semantics to an LLM. Ask it to write a program to solve a problem in that language. It will not always work, but it will work often enough to demonstrate that they are very much capable of writing code that has never been written before.

The first time I tried this was with GPT3.5, and I had it write code in an unholy combination of Ruby and INTERCAL, and it had no problems doing that.

Similarly giving it a grammar of a hypothetical language, and asking it to generate valid text in a language that has not existed before also works reasonably well.

This notion that LLMs only spit out things that has been written before might have been reasonable to believe a few years ago, but it hasn't been a reasonable position to hold for a long time at this point.

0 comments

shrewduser10mo ago

This doesn't surprise me, i find LLM's are really good at interpolating and translating. so if i made up a language and gave it the rules and asked it to translate i wouldn't expect it to be bad at it.

vidarhOP10mo ago

It shouldn't surprise anyone, but it is clear evidence against the claim I replied to, and clearly a lot of people still hold on to this irrational assumption that they can't produce anything new.

skydhash10mo ago

They're not producing anything new... If you give it the answer before asking the question, no wonder it can answer. Prompting is to find resonance in the patterns extracted from the training data, which is why it fails spectacularly for exotic programming languages.

1 more reply

j / k navigate · click thread line to collapse