undefined | Better HN

0 pointswalshemj6y ago0 comments

In my experience ops call the developers when something breaks overnight in any case.

0 comments

In our organisation, developers have need nor clearance to access live customer data. This means they stay out of production and we have to deal with breakage.

Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick.

Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.

walshemjOP6y ago

So you have some one every shift who is familiar with every part of a complex system and the requisite programming languages.

And reverting part way through a monthy telco billing run might not be the best idea.

And in my case they also looked after online services and other systems.

peterwwillis6y ago

Preventing calling the devs requires dev to front-load ops. Ops needs triage documents, remediation strategies, architectural and workflow diagrams, dependency charts, distributed tracing, intelligent logging, custom application metrics, and tests they can run during incidents to isolate causes.

The devs can do all of this on their own, teach ops how to use it, and then they'd only be called when it was a code issue. But as a dev, you probably don't know all of the above, so ops has to go to dev and be like, "hey y'all, if you don't want to be called, this is what we need."

And this is what DevOps is intended to fix: get everyone in a room, talk about problems, find solutions among everyone. If your org isn't doing this, you can start the change.

j / k navigate · click thread line to collapse