1
So I inherited this old project at work that's basically been put on ice. It had its own AWS account where we managed all things related to the project. Aside from occasionally verifying that the service was working as expected, basically nothing was required from me.
About a month ago, someone in accounting killed the credit card that was paying for all the services on this account unbeknownst to me. Obviously, this was a big screw up by us, but whatever. When stuff like this happens usually, it's not too big of a deal because SaaS providers usually will send notifications that the charge failed to process. Not so with AWS. We received no notification - nothing.
So it just so happens that at the exact time our payment failed to process, our domain name came up for renewal on Route 53. Since our payment credentials were invalid, our domain failed to renew. So all DNS requests to the domain, including all emails, just get dropped now since DNS can't resolve.
Eventually, it came to our attention that the service was down and the payment credentials were invalid. No big deal - it's a temporary interruption of service on a tool that hardly anyone uses.
Here's where the kafkaesque scenario unfolds. When I go to login at AWS with the root account, I'm presented with a MFA challenge to enter the code emailed to my account. However, since our DNS has expired, I can't receive the email. I can't login to AWS because our email is down due to DNS. But I can't fix our email because I can't login to AWS.
Once I figure out what's happening, I reach out to support. A low level outsourced offshore IT worker answers the phone who literally doesn't even understand what DNS is. She requests my "the name registered to my account" which I do. But apparently the name I provide is incorrect. No one in our company knows what the name might be, since it's actually an admin account, not an account owned by an actual named person. I get that she's trying to keep out scammers, etc, but she just keeps repeating ad infintum the same thing. I ask to be transferred to her manager, upon which she transfers me to the Amazon consumer division and the CSR has zero idea of what I'm talking about.
Next I try to wade through an impenetrable maze of various forms and contact numbers. After submitting a bunch of forms, I finally get someone on the line who seems to be an American and understands the concept of DNS. However, she can't help me, and tells me to fill out yet more forms. I fill those out and provide all the details of the case in the form. I get a stock response saying they can't help me along with links to the same forms I was originally directed to.
And that's where I'm at. As far as I can tell, there's no way out of the Kafkaesque hamster wheel I find myself in. If anyone has some ideas for how I can find a way out of this situation, I'd love to hear it.
So anyhow, be warned ya'll.
For the life of me I cannot figure out how to speed up the scraping process. For example, when I scrape it locally I can only get like a maximum of like 300kb/second no matter how much I try to parallelize requests, even though I have 200mbps of bandwidth. It's just annoying for our marketing team to have such a long delay in between publishing changes and seeing it deployed live.
Am I getting hit with some sort of Cloudfront rate limiting by IP address? Is there some socket limit at a real low level I'm hitting on both my local mac and the linux box I do the scraping on?
What are the best ways I can speed things up?