I like to quip that people need to imagine LLM security as at-least-as-bad as javascript code in someone else's web-browser: A determined person can make them emit whatever they want (as noted above) and also none of the data that went into them is reliably secret either.

As an analogy, it still needs some work through, since it doesn't adequately alarm people about the risks of covertly poisonous data even with an honest user.

And even that’s imperfect if you miss an integration with an externally visible effect- for example an agent with web search can exfiltrate info via visiting specific urls with that log visitors- I’ve POC’d this with claude in the browser, although I only got a few bits out since you need to get N pages ranked on google to exfiltrate log (N!) bits

I suspect people routinely paste urls to documentation into claude code, which it will fetch, or maybe tell it to do web searches. It has those tools built in.

You're right, but if all it can do is create and display output, then how will any of the oligarchs betting the farm on LLMs actually fulfill the breathless promises they've made?