undefined | Better HN

0 pointsphkahler5y ago0 comments

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change.

This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

0 comments

jacquesm5y ago

> Normal testing was bypassed - per management request after a small code change.

That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

cortesoft5y ago

It isn't learned because 99% of the time, it works fine and nothing bad happens.

We are very bad at avoiding these sorts of rare, catastrophic events.

eru5y ago

That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.

smaudet5y ago

I disagree, it's in part a people problem - more draconian test suites just make developers more inclined to cheat and they tend to write tests which are not valid or just get the tool passing...

It's more important to visually model and test than to enforce some arbitrary set of rules that don't apply universally - then you have at least the visual impetus of 'this is wrong' or 'I need to test this right'.

A lot of time is spent visually testing UIs and yet these same people struggle with testing the code that matters...

monkpit5y ago

Until a manager is told about how hard the automation makes it to accomplish their goal...

1 more reply

edoceo5y ago

Managers who have been bitten still make this choice

coryrc5y ago

Or most of what we do isn't really important so it doesn't matter if it breaks every once in a while.

pacificmint5y ago

Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event.

It’s been many years since I read it, but I recall it being a very interesting read.

tinco5y ago

For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme.

The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.

TheSpiceIsLife5y ago

Do management typical typically study Computer Science?

_heimdall5y ago

Unfortunately most people become a manager by bring a stellar independent contributor. People management and engineering are very different skills, I'm always impressed when I see someone make that jump smoothly.

I always wanted companies to hire people managers as its own career path. An engineer can be an excellent technical lead or architect, but it can feel like you started over once you're responsible for the employees, their growth, and their career path.

1 more reply

tinco5y ago

Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.

1 more reply

_heimdall5y ago

As a former AT&T contractor, albeit from years later, this checks out. Sat in a "red jeopardy" meeting once because a certain higher-up couldn't access the AT&T branded security system at one of his many houses.

The build that broke it was rushed out and never fully tested, adding a fairly useless feature for said higher-up that improved the UX for users with multiple houses on their account.

j / k navigate · click thread line to collapse

0 comments

jacquesm5y ago

> Normal testing was bypassed - per management request after a small code change.

That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

cortesoft5y ago

It isn't learned because 99% of the time, it works fine and nothing bad happens.

We are very bad at avoiding these sorts of rare, catastrophic events.

eru5y ago

That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.

smaudet5y ago

I disagree, it's in part a people problem - more draconian test suites just make developers more inclined to cheat and they tend to write tests which are not valid or just get the tool passing...

A lot of time is spent visually testing UIs and yet these same people struggle with testing the code that matters...

monkpit5y ago

Until a manager is told about how hard the automation makes it to accomplish their goal...

1 more reply

edoceo5y ago

Managers who have been bitten still make this choice

coryrc5y ago

Or most of what we do isn't really important so it doesn't matter if it breaks every once in a while.

pacificmint5y ago

Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event.

It’s been many years since I read it, but I recall it being a very interesting read.

tinco5y ago

The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.

TheSpiceIsLife5y ago

Do management typical typically study Computer Science?

_heimdall5y ago

1 more reply

tinco5y ago

Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.

1 more reply

_heimdall5y ago

The build that broke it was rushed out and never fully tested, adding a fairly useless feature for said higher-up that improved the UX for users with multiple houses on their account.

j / k navigate · click thread line to collapse