We collected the evidences and filed police report. The bill is paid through a distributor, anything we ask about the reduction of payment, the distributor just passes it on to Microsoft. I feel if we don't find a way to talk to Microsoft, we will just end up paying the whole thing.
Many of you might think we screwed up, we pay up, but I think it's more like a stolen credit card situation, we can negotiate with the bank. How do I go about this?
Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions?
Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.
1. OP <--> Partner (confidential agreement) <--> Microsoft (MPA)[1]
2. OP <--> Microsoft (MCA)[2]
3. OP <--> Microsoft (MOSA)[3]
The different types of agreements have different limitations of liability clauses. What OP wrote indicates a "partner" is involved and if this is the case, Microsoft have essentially shifted liability for fraud and billing non-payments onto the "partner"[4], who would then either wear the cost or try to shift this liability to the OP. It's not that straightforward though as any of the three parties could have a share of liability, and the "partner" would be very unlikely to want to get in a dispute with Microsoft as this would impact their other business. Liabilities are possibly also impacted by default spending limits and caps that are imposed by Microsoft on different services[5].
Allowing a $200,000 bill for one month (a 200x increase) has the appearance of being very poor financial management from the "partner" as they're potentially going to be stuck with unsecured $200,000+ liabilities from their customers if the customers became insolvent. I suppose it is possible the OP and "partner" have a bank guarantee in place to cover at least $200,000 but I'd hazard a guess they may just try to rely on an insurance policy instead to cover these rare events.
[1] Microsoft Partner Agreement (MPA): https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE...
[2] Microsoft Customer Agreement (MCA): https://www.microsoft.com/licensing/docs/customeragreement
[3] Microsoft Online Subscription Agreement (MOSA): https://azure.microsoft.com/en-au/support/legal/subscription...
[4] https://learn.microsoft.com/en-us/partner-center/non-payment...
[5] https://azure.microsoft.com/en-us/support/legal/offer-detail...
There is some risk that they will terminate your account.
You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.
The type of company posting about getting hacked like this is probably using the root / admin accounts to do most things. Their lowest hanging fruit and biggest wins would probably MFA, then SSO.
However, IAM is generally powerful enough to allow you to configure what an account can do.
So best practice, you also want to think about how you're going to lose credentials.
- Sharing passwords across services - leak of your .dotfiles, either by having your laptop pwned, or uploading your .dotfiles to a public repo as a backup or something. - Accidently pasting into the wrong window or something.
SSO & MFA defeats all of these with exception that your sts token will be signed for 1h in those .dotfiles when you auth yourself. I'm not sure what happens if you remove it from the token from the device, but the device itself being compromised would allow someone to piggy back your session.
Ergo, you move to least privilege access, and then if your laptop, server, or ci/cd runner gets hijacked, then it's only able to do whatever it was allowed to do in the first place.
The last part is you need to detect the misuse.
When you have least privileged access, and a pretty locked down account, the hope is when a session is hijacked, the attacker will attempt to use the credentials and get an access denied. This should allow you to detect and remediate the reason for leak.
Obviously this turns your cloud install into a lot more work, and you still also need to look at maintaining and patching the actual services so they're not compromised in the first place.
On the one side, if you got hacked to that degree - root account, MFA, e-mail, etc - you really fucked up, to be very blunt.
On the other, it's down to Microsoft to provide good security and protections - e.g. spending limits, with a "contact us" and mandatory waiting period if you're about to go e.g. 10x over what you normally pay.
Banks (at least here) offer you a kind of insurance; if you get hacked, they can lock your account and return you your money. Their reasoning there is that they messed up and didn't make it obivious enough that you're about to, for example, send X amount of money away. (this is one reason why cryptocurrencies will never replace banks)
How did the account get compromised? What was the nature of the attack (e.g. cryptocurrency mining, expensive egress traffic for file hosting, etc.)?
Every (consumer) credit card I've seen requires you to take reasonable steps to keep the cards secure to be eligible for fraud protection (e.g. changing the PIN if compromised, not lending it to people, alerting the issuer ASAP in case of suspected fraud, etc.). I do not use Azure but I would imagine that it works the same way - that is, if you fail to follow basic security precautions (enabling MFA, not using shared accounts or passwords that have been known to be compromised in a leak, etc.) you'll probably end up stuck with the bill. Hopefully you had things reasonably well secured.
Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature.
If you're using a credit card to pay (though can't see a credit card having a 200k limit, even business) you might want to see if they can help (though it's not the credit card itself that was stolen, so it's unlikely they'd cover you). Otherwise, I'd imagine you're SOL unless you have some other insurance you can rely on.
Beacuse the public indignation directed at cloud companies who don't always eat the costs in these situations vastly outweighs the cost of simply eating these costs, at least for cloud companies at the top tier of economies of scale (AWS, GCP, Azure, etc)
If AWS didn't always eat costs like this, startups might think twice before using AWS, etc, etc.
"Goodwill" has value to a corporation. Taking a hard line against legitimate mistakes that anyone (yes, anyone) can make costs them goodwill, and costs them customers.
And beyond that, while accidental/fraudulent usage doesn't cost them $0, the services are marked up to the point that they probably doesn't really lose that much by forgiving the charges.
Microsoft might, but are unlikely, to help you out.
Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t.
Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new business. You may need to go through an lawyer or administrator depending on bankruptcy laws where you are.
I’ve taken a client through this, after a similar situation - they ended up with a vast bill to a supplier brought about by someone else using their credentials, and the supplier not being willing to budge. It cost about a week of time and about $2k in legal fees.
I’ve also been on the receiving end, where I presented a legitimate invoice and rather than pay the client reincorporated and kept the IP - which sucks, but Microsoft will be insured against insolvencies, so I wouldn’t feel bad about it. You’re just allowing their insurer to help everyone out.
Is that true? I have no experience with Microsoft, but I've heard quite a few stories of Amazon crediting AWS accounts when customers write in to say their account was compromised. Or even cases when the customer themselves screwed up some permissions in a way that ended up costing an arm and a leg. Hard to believe this practice would be unique to AWS.
It’s still worth trying, as a first resort, of course, but it isn’t something I’d count on.
Apart from that you seem to be limited to monitoring individual resources, or using some external service that enunerates everything
Disclaimer: I work on AWS, although nowhere near billing
It's amazing what I can have a pre-paid account for a VPS hosting in Nicaragua, yet Amazon doesn't have this as an option.
/rant
Once you pay it, you lose all leverage. You're much less likely to ever get any money back.
Probably consult with a lawyer.
Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In my experience, companies are pretty willing to forgive fraudulent charges if you don't have an unusual history of them.
Allowing that is a slippery slope for a cloud host. If people can simply say "oh, someone used our credentials to do that thing that cost a lot of money" as a get-out-of-bill card..
If they were legitimately hacked, as in, the intruder did NOT simply obtain their access credentials, but actually bypassed the security system itself (hacking into the actual azure host, or exploiting a technical glitch in the azure login system) then, of course they should forgive the bill (and apologize to their customers)..
Don't rely on the distributor/vendor, they act very slowly.
You're a customer of Azure, you can contact them by any mean, the fact you pay through a distributor doesn't change that relationship.
So I would open a Azure support, and also will try to find Azure team on Twitter/Hacker News etc and contact them politely for help.
There is no way you would have to pay this bill. They will sort out something or even waived it if it's the first time.
What we did to recover the cost was to contact the account manager for our region at the time. So, maybe you could have better luck trying to find the particular person in linkedin. Or, have you tried opened a ticket from Azure console?
Nonetheless, I hope after everything has been settled down, you won't fire anyone (and treat it as learning opportunity)
So maybe just file a support ticket, or have your distributor file a ticket for you?
We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.
Why did you fire the employee (?) so quickly? Did he have a history of negligence and/or incompetence on the job and this was the last straw that broke the camel's back?
This would've been much easier if someone stole your credit card and bought things with it (the CC company would help with the chargeback).
Why would AWS be any different?
It was pretty much entirely our fault, and we were still able to get those charges forgiven when we owned up to the error and asked nicely.
So I'd at least recommend asking Azure politely first.
