> double++

I'd suggest to ++double the cost. Compare:

++double: spoken as "triple" -> team says that double++ was a joke, we can obviously only double the cost -> embarrassingly you quickly agree -> team laughs -> team approves doubling -> you double the cost -> team goes out for beers -> everyone is happy

double++: spoken as "double" -> team quickly agrees and signs off -> you consequently triple the cost per c precedence rules -> manager goes ballistic -> you blithely recount the history of c precedence in a long monotone style -> job returns EINVAL -> beers = 0

And likely far more than double the cost since you have to use the criminally-priced outbound bandwidth to keep everything in sync.

Shouldn't be double in the long term. Think of the second cloud as a cold standby. Depends on the system. Periodic replication of data layer (object storage/database) and CICD configured to be able to build services and VMs on multiple clouds. Have automatic tests weekly/monthly that represent end-to-end functionality, have scaled tests semi-annually.

This is all very, very hand-wavey. And if one says "golly gee, all our config is too cloud specific to do multi-cloud" then you've figured out why cloud blows and that there is no inherent reason not to have API standards for certain mature cloud services like serverless functions, VMs and networks.

Edit to add- ink ow how grossly simplified this is, and that most places have massively complex systems.

And data egress fees just to get the clone set up, right? This doesn’t seem feasible as a macrostrategy. Maybe for a small number of critical services.

If you use something like cockroachdb you can have a multi-master cluster and use regional-by-row tables to locate data close to users. It'll fail over fine to other regions if needed.

I totally agree with you. Where I work, we self-host almost everything. Exceptions are we use a CDN for one area where we want lower latency, and we use BigQuery when we need to parse a few billion datapoints into something usable.

It's amazing how few problems we have. Honestly, I don't think we have to worry about configuration issues as often as people who rely on the cloud.

We have our own data center with servers. The upfront costs are high, but it was worth it in our use-case

Not GP, but my company also self-hosts. We rent rackspace in a colo. We used to keep my team's research server in the back closet before we went full-remote.

Somewhat inevitable for any company as they get larger. Easy to move fast and break things when you have 1 user and no revenue. Very different story when much of US commerce runs you on.

For folks who came of age in the late 00's, seeing companies once thought of as disruptors and innovators become the old stalwarts post-pandemic/ZIRP has been quite an experience.

Maybe those who have been around longer have seen this before, but its the first time for me.

If you bring something down in a real way, you can forget about someone trusting you with a big project in the future. You basically need to switch orgs

Curious. When did AWS hit “Day Two”, or what year was your 2nd tour of duty?

I've never heard tour of duty being used outside of the military, is it really that bad over at AWS it has to be called that?

They have been pushing me and company extremely hard to vet their various AI-related offerings. When we decide to look into whatever service it is, we come away underwhelmed. It seems like their biggest selling point so far is “we’ll give it to you free for several months”. Not great.

Fascinating, thanks for sharing this.

I found this summary:

https://fortune.com/2025/07/31/amazon-aws-ai-andy-jassy-earn...

And the transcript (there’s an annoying modal obscuring a bit of the page, but it’s still readable):

https://seekingalpha.com/article/4807281-amazon-com-inc-amzn...

(search for the word “tough”)

I have a joke from 15 years ago, where I compared my friend who flaked out all the time as "having less availability than US-EAST-1".

This is not a new issue caused by improper investment, it's always been this way.

It's easy to say this, but in the real world, most of the critical path is heavily-dependent on third party integrations. User auth, storage, logging, etc. Even if you're somewhat-resilient against failures (i.e. you can live without logging and your app doesn't hard fail), it's still potentially going to cripple your service. And even if your entire app is resilient and doesn't fail, there are still bound to be tons of integrations that will limit functionality, or make the app appear broken in some way to users.

The reason third-party things are in the critical path is because most of the time, they are still more reliable than self-hosting everything; because they're cheaper than anything you can engineer in-house; because no app is an island.

It's been decades since I worked on something that was completely isolated from external integrations. We do the best we can with redundancy, fault tolerance, auto-recovery, and balance that with cost and engineering time.

If you think this is bad, take a look at the uptime of complicated systems that are 100% self-hosted. Without a Fortune 500 level IT staff, you can't beat AWS's uptime.

With the exception of Amazon, anyone in this situation already has a third-party product in their critical path - AWS itself.

> Why would a third-party be in your product's critical path?

i bet only 1-2% of AI startups are running their own models and the rest are just bouncing off OpenAI, Azure, or some other API.

Not necessarily our critical path but today circleci was affected greatly which also affected our capacity to deploy. Luckily it was a Monday morning therefore we didn’t even have to deploy an hot fix.

That's nearly every ai start-up done for

No man is an island, entire of itself

* IAM / Okta * Cloud VPN services * Cloud Office (GSuite, Office365)

Good luck naming a large company, bank, even utility that doesn't have some kind of dependency like this somewhere, even if they have mostly on-prem services.