undefined | Better HN

0 pointsroywiggins26d ago0 comments

"With your permission, Copilot may execute code or bash commands in this folder." could be interpreted either way I suppose, but the actual question is "do you trust the files in this folder" and not "do you trust Copilot to execute any bash commands it wants without further permissions prompts".

The risk isn't solely that there might be a prompt injection, Copilot could just discover `env sh` doesn't need a user prompt and just start using that spontaneously and bypassing user confirmation. If you haven't started Copilot in yolo mode that would be very surprising and risky.

If it usually asks for user confirmation before running bash commands then there should, ideally, not be a secret yolo mode that the agent can just start using without asking. That's obviously a bad idea!

"Actually copilot is always secretly in yolo mode, that's working as designed" seems like a pretty serious violation of expectations. Why even have any user confirmations at all?

0 comments

binsquare26d ago

If the user is working in a folder where copilot can discover a malicious `env sh` to run, the user should not give permission to trust the files in the folder.

I think it's a valid observation that we can bypass the coding AI's user prompting gate with the right prompt. That is a valid limitation of LLM supported agentic workflows today.

But that's not what this article claims. The article claims that there was no user approval and no user interaction beyond initial query and that the copilot is downloading + executing malware.

I'm saying this is sensationalized and not a novel technical vulnerability write up.

The author explicitly gave approval for copilot to trust "untrusted repository". Crafted a file which had instructions to do a curl command despite the warnings on copilot start up. It is not operating secretly in yolo mode.

If the claim of the article is "Copilot doesn't gate tool calls with env", I'd have a different response. But I also have to mention, you can tune approved tool calls.

j / k navigate · click thread line to collapse