1. dig @1.1.1.1 jgc.org
2. nc -v 104.22.11.223 80
3. curl -v https://jgc.org/cdn-cgi/trace
4. curl -v https://jgc.org/
Hmm #1 was fast so network is routing OK. Hmm #2 was fast so TCP is OK. Hmm #3 was fast so I know (because I worked on that code) that this code path is good. Hmm #4 is slow so that means component X is slow but still working.Of course, in parallel I'm in a conference call with about 40 other people who have actual access to monitoring and systems and other things who can see exactly where things are.
But I was damn close with four commands and gave me confidence in what people were saying. But, I have to say, Cloudflare's internal distributed tracing system is pretty cool because I got sent a trace and you could see right where the slowdown was.