undefined | Better HN

0 pointsqsort2y ago0 comments

The thing is that security is binary. One input out of a billion causes bad behavior and you're fucked, exactly like eval, execvpe, sql injections and all their relatives.

The point isn't that you can't use LLM output, it's that you should always consider LLM output as potentially hostile. You can somewhat mitigate this by pairing a LLM with a deterministic system that only allows a predictable subset of behavior, but it's a tricky problem to remove completely.

0 comments

brookst2y ago

> you should always consider LLM output as potentially hostile

Sure, agreed. How is that different from human output?

qsortOP2y ago

"Human output" isn't automated nor connected to your production systems. Would you let any random user run arbitrary SQL against your production DB?

MacsHeadroom2y ago

Not a random user, but an employee called or emailed by a random social engineer yes. Notably, most real "hacking" is social engineering and LLM prompt exploitation seems more like an extension of SE than technical hacking.

1 more reply

unusualmonkey2y ago

>Human output" isn't automated nor connected to your production systems.

Err... what?

How do you think businesses work?

j / k navigate · click thread line to collapse