undefined | Better HN

0 pointsscubbo3y ago0 comments

Sort-of a nitpick, but for stateless services that's _precisely_ what you want to happen!

0 comments

The clients and network have state. If you kill every service, and set everything up from scratch, your users are probably seeing an error message. Their connections all died and the retries failed while you were spinning up the new version of things.

scubboOP3y ago

Right, which is why you set up the new versions of the service _first_ and cutover to them before tearing down.

taeric3y ago

Not when your infrastructure includes your data store, though. Or if it includes such things as your application endpoint. You may want to stand up the new endpoints and then drain over traffic, as an example. Not take an outage for a deployment.

scubboOP3y ago

Right, yes - sorry, I should have clarified that I didn't mean "tear down then set up", but rather "set up (new), cut over, then tear down (old)". But still - the service should be "created from scratch"!

tremon3y ago

Not when your infrastructure includes your data store, though

Then the service is no longer stateless.

taeric3y ago

I mean, you aren't wrong. But I am challenged to think of a service that doesn't store state somewhere.

And you ignore the other situation. Even a fully "stateless" service that literally just responds to things probably doesn't want an outage just to deploy a new version.

tremon3y ago

That's true. The way we solve it is by splitting our infrastructure deployments in two parts: a volatile part and a persistent part; the volatile part is versioned by buildnumber, so each deployment is side-by-side with the previous one. The drain/switchover is a manual process for now as we have a user acceptance step in the middle.

It gives us more confidence when deploying the stateless part, but sadly it also means that the persistent deployments don't get as much exercise as the stateless part and are somewhat more prone to bitrot.

j / k navigate · click thread line to collapse

0 comments

nitwit0053y ago

scubboOP3y ago

Right, which is why you set up the new versions of the service _first_ and cutover to them before tearing down.

taeric3y ago

scubboOP3y ago

tremon3y ago

Not when your infrastructure includes your data store, though

Then the service is no longer stateless.

taeric3y ago

I mean, you aren't wrong. But I am challenged to think of a service that doesn't store state somewhere.

And you ignore the other situation. Even a fully "stateless" service that literally just responds to things probably doesn't want an outage just to deploy a new version.

tremon3y ago

j / k navigate · click thread line to collapse