In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.
Doing this kind of fast iteration has its risks, but it has its benefits too. We de-risk it, in part, by having extensive CI, which is why it was so important to us that the CI is fast & reliable.
Delivering larger, less-frequent updates has its own risks. You're not practising your release process as frequently, so it's a much bigger event. You're pushing many, many more changes in one go, so there's a lot more surface area for something to go wrong, and rolling it back is a much bigger job. And dropping many/bigger changes to the user experience is much more noticeable.
Again, this isn't the right process for everyone, but it works for us and its how we've managed to build a product that delivers value to our users.
Like, is there nothing in between? Like once a week, once every other week?
If you would have to estimate, wouldn't be there less bugs if you would deploy less fast (and use this time for validation)?
I've worked on products across the spectrum, Enterprise software that releases twice a year, smaller stuff that ships once a month, and now a SaaS that releases many times a day. I don't have the data to compare, but I genuinely don't believe the rate of bugs was materially different between any of them.
We make faster changes, but the changes are much smaller, and so the surface area to test is smaller. The more changes you make, the more time you need to validate. If you make a weeks worth of changes, you're going to need an appropriately longer validation cycle than if you make 1 change. It scales, and in my experience, many products with slower release cycles aren't appropriately scaling up the validation time to match.
That said, this doesn't necessarily mean we're writing code and 30 minutes later its in production...there's still an iterative dev cycle with lots of validation happening...but if something is ready to go, ship it!
Very interesting. I agree different strokes for different folks, you guys seem to be on the extreme end of CI/CD.
Have you done any sort of analysis you could share on what it costs to release up to 20 times per day?
Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.
I would think constant releasing would also make debugging prod issues pure hell.
