1. What happened to all the data Copilot trained on that was confidential? How is that data separated and deleted from the model’s training? How can we be sure it’s gone?
2. This issue was found; unfortunately without a much better security posture from Microsoft, we have no way of knowing what issues are currently lurking that are as bad as —- if not worse than —- what happened here.
There’s a serious fundamental flaw in the thinking and misguided incentives that led to “sprinkle AI everywhere”, and instead of taking a step back and rethinking that approach, we’re going to get pieced together fixes and still be left with the foundational problem that everyone’s data is just one prompt injection away from being taken; whether it’s labeled as “secure” or not.
I'd add (3) - a DLP policy is apparently ineffective at its purpose: monitoring data sharing between machines. (https://learn.microsoft.com/en-us/purview/dlp-learn-about-dl...).
Directly from the DLP feature page:
> DLP, with collection policies, monitors and protects against oversharing to Unmanaged cloud apps by targeting data transmitted on your network and in Microsoft Edge for Business. Create policies that target Inline web traffic (preview) and Network activity (preview) to cover locations like:
> OpenAI ChatGPT—for Edge for Business and Network options > Google Gemini—for Edge for Business and Network options > DeepSeek—for Edge for Business and Network options > Microsoft Copilot—for Edge for Business and Network options > Over 34,000 cloud apps in the Microsoft Defender for Cloud Apps cloud app catalog—Network option only
/Offtopic
Yes, MSFT's DLP/software malfunctioned, but getting users to MANUALLY classify things as confidential is already an uphill battle. These are for the rare subset of people that are aware of and compliant with NDAs/Confidentiality Agreements!
1) you can't be sure it's gone. It's even questionable if data can be removed (longer discussion needed). These are compression machines, so the very act of training is compressing that information. The question really becomes how well that information is compressed or embedded into the model. On one hand, the models (typically) aren't invertible so the information is less likely to be compressed lodslessly. On the other hand, the models aren't invertible, so reversing them is probabilistic and they are harder to analyze in this sense.
2) as you may gather from 1) there's almost certainly more issues like this. There are many unknown unknowns waiting to be discovered. Personally this is why I'm very upset the field is so product focused and that a large portion regards theory as pointless. Theory does two things for us because it builds a deeper and more nuanced understanding. Theory advancing allows us to develop faster as we can iterate on paper rather than through experimentation. This allows us to better search the solution space and even understand our understanding. This also leads to better safety of models as it is necessary to understand them to understand where they fail and how to prevent those failures. Experimentation alone is incredibly naïve. It is like proving the correctness of your programs through testing (see the issues with TDD). Tests are great but they are bounds, not proofs. They can suggest safety, give you some level of confidence in safety, but they cannot guarantee it. We all know that the deeper understanding of your code the better tests you can write, and this is the same thing here. That theory is reducing your unknown unknowns and even before strong proofs are made we can get wider coverage in our testing.
I think we're so excited right now we're blinding ourselves. If we're cutting off or reducing fundamental research then we are killing the pipeline of development. Theory is the foundation that engineering sits on top of. But what worries me is that there's so many unknown unknowns and everyone is eagerly saying "we're just need 'good enough'" or "what's the minimum viable product". These are useful tools/questions but they have limits and it gets dangerous when putting out the minimum at scale
They have significant experience in this. Microsoft software since the 2014, for the most part, is also paraphrased from other people's code they find laying around online.
It depends. E.g. OpenAI says: "By default, we do not train on any inputs or outputs from our products for business users, including ChatGPT Team, ChatGPT Enterprise, and the API."[0]
[0] https://openai.com/policies/how-your-data-is-used-to-improve...
If their models ever spit out obviously confidential information belonging to their paying customers they'll lose those paying customers to their competitors - and probably face significant legal costs as well.
Your random confidential corporate email really isn't that valuable for training. I'd argue it's more like toxic waste that should be avoided at all costs.
That was pretty funny and explains a lot.
I wish I could do more :(
Instead I always break things when I paraphrase code without the GeniusParaphrasingTool
Microsoft releasing overly ambitious features with disastrous consequences.
Apple releasing features so unambitious it's hard to remember they're there.
Big tech is reaping what they've sown in a very satisfying way.
Half of the time it's open user hostility and blatant incompetence. The other half it's just the incompetence. Ambition doesn't enter the picture at all.
How is having Copilot breach trust and privacy an “advisory”? Am I missing something?
Unfortunately "Advisory" is a report written about a security incident, like an official statement about the bug, it's impact, and how to fix it -- which differs from the english meaning... it's not meant to mean to "advise" people or to "take something" under "advisory" (which, is a very soft statement typically).
An advisory gives notice and/or warns about something, and may give recommendations on possible actions (but doesn’t have to).
So, yes, technically, it's de-facto advisory to publish this information, but assigning "advisory" as a severity tag here is questionable.
What's the actual action needed here by a security team? None. You can hate it or not care but the end of the day there's no remediation or imminent harm, just a potential issue with DLP policies. Don't make it look like a 0-day that they actually have to deal with.
The basic distinction in the infosec industry is that advisories are what you publish to tell customers that you had a bug in your product that might have exposed them or their data to attacks and you want them to take some specific action (e.g., upgrade a package, review logs); while an incident report is what you publish when you know that the damage happened, it involved your infrastructure, and you want to share some details about happened and how you're going to prevent it from happening again.
Because the latter invites a lot more public attention and regulatory scrutiny, a company like Microsoft will go out of their way to stick to advisories whenever possible (or just keep incidents under wraps). It might have happened at some points in their history, but off the top of my head, I don't recall Microsoft ever publishing a first-party security incident report.
I assume that whatever that is processed by AI service are generally retained for product improvements (training).
100 nasty bugs in the code
100 bugs in the code
Take one down
Patch it around
-127 nasty bugs in the codeTrusted operating system Mandatory Access Control where art thou?
You guys need to read the actual manifestos these AI leaders have written. And if not them, then read the propagandist stories they have others write like The Overstory by Richard Powers which is an arrogant pile of trash that culminates in the moral:
humans are horrible and obsolete and all should die and leave the earth for our new AI child
Which is of course, horseshit. They just want most people to die off, not all. And certainly not themselves.
They don't care about your confidential information, or anything else about you.
Just my whacky conspiracy theory of the day!