February 28th DDoS Incident Report (opens in new tab)

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

AS much as I hate chatops as a fad, it makes it really easy for others to also see what happened and follow along.

"Hey Jim, Did you go adjust that thing?"

vs.

"10:15am [JIM] /chatbot adjust x to y"

When you have infrastructure as code, you get infrastructure by coders.

(I’ll be here all week)

I wonder if it's time for providers like Amazon to provide configs by default that block all ports besides TCP 22, 80 and 443. You want to do other stuff? Configure a firewall. Don't know how? Hire somebody who does. This scenario with cheap insecure things being put out on the internet repeats again and again. IoT, PaaS, etc.

Google's "Beyond Corp" initiative [1] discourages trusted networks and VPNs in favor of secure services on public networks. By trusting the network to provide a level of security, you are more likely to be vulnerable to escalation attacks by bad actors that are able to access your private networks. You're also more likely to encourage legitimate users to set up workarounds that result in secure network breaches. Typically they use an Identity aware proxy in front of the service, but services can have a public view as well.

To answer your second question, I work for an open source non-profit software company, and we run some of our jenkins servers, which do continuous integration builds, publicly available so that community contributors and users can see build failures. Google has a number of open source projects that probably have similar goals.

1. https://cloud.google.com/beyondcorp/

Many open-source applications (especially Java-based) use a public-facing Jenkins server for running and distributing nightly and PR builds. Nowadays, this is usually handled by hosted CI (Travis or GitLab), but there are still some who prefer good old Jenkins.

That's good, but they should go a step further and stop listening on all addresses by default.

That sounds like a negative externality that ISPs get to be lazy about and save money on by shoving the burden onto the cloudflares of the world. It’s really hard to dispose of hazardous waste when manufacturing things, but we force manufacturers to pay for the negative externalities. We should probably start thinking about the internet in the same way we think about the environment.

Note everyone seems to agree:

https://www.internetsociety.org/blog/2014/07/anti-spoofing-b...

If I understand the article's point, essentially, carriers pay for the egress traffic that causes DDoSes, that cost and the cost of the generated ill-will outweighs that of filtering, whose price has fallen and continues to fall.

Personally, I think that if the article author is correct, then I wonder if this is one of those high-level long-term decisions that companies appear absolutely incapable of making. (In my experience, short-term gains are way overvalued at the cost of long-term loss, generally, especially when it is hard to directly determine the costs/benefits involved.)

Let's rephrase the question - Is there any reason consumer ISP's don't follow BCP38?

There is almost no reason whatsoever for clients to spoof their public IP address. Obviously, there are reasons to SNAT at the carrier level for load balance or routing purposes.

How could source IP spoofing help with DDoS mitigation?

Please define better. If (say) an IPFS item is cached by 3-5 nodes each of which is on a private home internet connection it is trivial to DDOS each of those and thus effectively censor the content. On the other hand, it is decidedly nontrivial to DDOS Cloudflare.

There are, fundamentally, two different kinds of attacks:

- Volumetric attacks like this one, mostly reflection.

- Application level attacks like SYN floods or protocol-specific attacks.

Defending against both costs a LOT of money.

Volumetric attack are dealt with at the network edge using rate limits and router ACLs. They're really easy to identify and block, but the point is that you need more bandwidth than the attacker in order to successfully do so. With attacks in the terabits-per-second range, this gets expensive.

Application-level attacks are harder to execute since there's no amplification and you need more bandwidth to pull it off, but they're much harder to block, too. They exhaust the server software's capacity by mimicking a real client. Common examples are SYN or HTTP floods.

When you get hit by a DDoS attack, you have two choices:

- Filter the attack and block the offending traffic without affecting legitimate requests. This is hard, and most companies can't do this. They need to have someone like Akamai on the retainer and dynamically reroute traffic like GitHub did.

- Declare bankruptcy and announce a blackhole route to your upstream providers (taking down the host in question, but protecting the rest of your network).

When you host custom applications that can't be scaled out or cached, DDoS mitigation is especially hard since you cannot just throw more servers at it like CloudFlare does.

Most services we host use proprietary binary UDP protocols, which is unfortunate, since UDP is easy to spoof and even experienced DDoS mitigation companies have trouble filtering it. Our customers get hit by DDoS attacks 24/7, so blackholing is not an option.

We had to build our own line-rate filtering appliances in order to handle the ever-increasing number of application-level DDoS attacks, by reverse engineering the binary protocols and building custom filtering and flow tracking.

All of this costs a huge amount of money, and most ISPs simply lack the resources to do this.

Happy to answer questions, but I'm going home right now, so it may take a few hours :-)

(Nitrado is a leading hosting provider specializing on online gaming, both for businesses/studios and regular customers, so we're dealing with DDoS attacks on a regular basis. We got hit with the same memcached attacks than GitHub and CloudFlare, and it was the largest attack in our company history. Ping me if you want to talk.)

ISPs absolutely could, but having worked near this space previously, it really isn't as easy as it sounds, both the detection and the mitigation, and ISPs are not particularly equipped to handle it themselves right now. There's a lot of money to be made there, though.

Well, we can't even get ISPs to filter outbound traffic. There's still no real incentive for them to do that.

Well, that's too bad. Bad actors ruined it for you.

We make things more annoying for VPN traffic because it's 99% bad actors. Every time someone is up to no good on our services, they're behind Tor/VPN.

It's simple cost/benefit analysis. If you think a business should bend to every single whim someone might have, then you haven't built much of one.

Making someone run Javascript so they can click on a captcha? Worth the loss of a few pennies because someone's angry about it on HN.

You need to conveniently ignore why people use Cloudflare to say that Cloudflare is breaking the internet. Ideally, nobody would have to use it, but that isn't reality.

"... you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage..."

Does anyone have an example of this webpage?

Unless I am engaging in e-commerce, I do not run a browser JavaScript engine. I rarely if ever encounter a webpage that truly "requires" one. GitHub certainly does not require JavaScript for me to use it via www.

I done believe you need to enable these options in cloudflare, these are optional, I believe it may be different over tor, where you have to do captcha. Certainly a couple.of years ago, I didn't have any of these convenience features enabled, as they were optional.

I assume this explains the apparent hugeness of the attack:

> The vulnerability via misconfiguration described in the post is somewhat unique amongst that class of attacks because the amplification factor is up to 51,000, meaning that for each byte sent by the attacker, up to 51KB is sent toward the target.

memcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...

Probably to get press. If you have a giant botnet and get press for it, it's probably easier to get clients.

It's in bit/sec not byte/sec so 8x lower.

This is honestly why I have no problem with things like brickerbot.

I consider it community service.

The GitHub post says the amplification factor is 51000, while the Wired article says it's 50. So which is it?

Maybe in the milliseconds between packet swarms, or immediately before or after? Just seems like a lot of resources to pour into an attack that was defeated in a few minutes. To what end?