However, in your example, I don't see how the agent decides what to do and how to do it. So it is unclear for me how the main agent is protected. That is, what is preventing the quarantined LLM to act on the malicious instructions instead, ignoring the documentation update, causing the agent to act on those?
That is, what is preventing the quarantined LLM to make the agent think it should generate a bug report with all the API keys in it?
Anyway, I do think having a secondary quarantined LLM seems like a good idea for agentic systems. In general, having a second LLM review the primary LLM in seems to identify a lot of problematic issues and leads to significantly better results.
The main LLM does have access to the tools or sensitive data, but doesn't have direct access to untrusted data (quarantine LLM is restricted at the controller level to respond only with integer digits, and only to legitimate questions from the main llm)
I know LLama folks have a special Guard model for example, which I imagine is for such tasks.
So my ignorant questions are this:
Do these MCP endpoints not run such guard models, and if so why not?
If they do, how come they don't stop such blatant attacks that seemingly even an old local model like Gemma 2 can sniff out?
Joey here from Archestra. Good question. I recently was evaluating what you mention, against the latest/"smartest" models from the big LLM providers, and I was able to trick all of them.
Take a look at https://www.archestra.ai/blog/what-is-a-prompt-injection which has all the details on how I did this.
It’s really a cat-and-mouse game, where for each new model version, new jailbreaks and injections are found
