I thought I might've introduced a regression in the application layer, but the pattern I saw was strange. The latency would only spike on weekdays from around 8am-6pm PST i.e. peak hours. Note that the site is still a WIP and has 0 real-user traffic.
After some digging I've confirmed the issue isn't happening at the application layer, but rather at the network/Cloudflare level. When I check `mydomain.com/cdn-cgi/trace` during peak hours, the Cloudflare data centre processing the request is very far from both me and the origin server (e.g. LHR London). And then it changes to something closer during off-hours. I've also confirmed latencies returns to normal when I disable Cloudflare proxying.
Some Cloudflare community members have described that the issue is actually a business problem with ISP-Cloudflare peering agreements rather than a technical problem: https://community.cloudflare.com/t/very-slow-server-response-time/611853/3 . But they say the fix is to go from a free plan to a pro plan, and I've tried that but the latency still persists...
I'm not really sure where to go from here. Has anyone gone through anything similar before? Am I missing anything obvious? I would prefer not to remove Cloudflare, but if the user experience is at the whims of Cloudflare's ISP agreements (which seemed to be fine a month ago!), then I'm seriously considering to just handle security on my own.
