undefined | Better HN

0 pointswgerard7y ago0 comments

Hmm, I don't see how that changes with longer/stricter deploy processes - unless you have some of the tooling around that makes CD very possible in the first place (automated checks/etc.)

I've certainly worked in places with very long and strict deploy processes that managed to mangle production data frequently. Even worse, because the deploy process was so strict and long the bad code managed to stay on production for much longer than 10 minutes (the deploy time mentioned in the article).

There's some vague notion out there that long deploy process == safe, but there's very little evidence to suggest that's the case. If anything, it seems much more dangerous because larger changesets are going out all at once.

0 comments

toomuchtodo7y ago

It goes back to my original comment above; if you have the proper tooling (tests executed that must pass prior to deploy given a green light, green/blue deploys, canaries, automated datastore snapshots/point in time recovery, granular control of the deployment process), I think continuous deployment provides a great deal of value above what you've invested into the process. But that investment is critical if you've bought into CD. Otherwise, it's "deploy and pray".

wgerardOP7y ago

Sure, and I guess my point is: If you haven't invested in those things, waterfall-esque deploy processes are just as bad and perhaps even worse because there's more chance for confounding changes to cause a nasty error.

The only reason waterfall-esque deploy processes work without those things is because companies often waste tons of people-hours on testing things out in the staging environment (which requires time, obviously).

vlovich1237y ago

The thing you're missing is that you're amortizing the cost. Yeah, it's typically prohibitive to run the manual testing on every CL. However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway. Automated tests don't necessarily solve this problem either. 1) Some automated tests can be time-consuming & so require batching of CLs to run too 2) it's impossible to predict if you are going to catch all issues via automated testing 3) there's always things it's easier to test for manually.

When it comes to data integrity, I would think you need a structured mechanism (at least for larger teams that have a high cost for failure) for rolling back any given CL either by tracking writes, making sure to have a plan in place to recover from any given CL (e.g. nuking the data doesn't break things), being able to undo the bad writes, or just reverting to a snapshot. Without being careful here CD-style development feels like lighting up a cigarette beside an O2 tank. Now for web development this is fine since it's not touching any databases directly. More generally it feels like a trickier thing to attempt everywhere across the industry.

1 more reply

j / k navigate · click thread line to collapse