You want two pairs of eyeballs on everything as you write it? Okay. Pay two guys.
You want complete test coverage? Okay. Can take as long as writing the code itself. Maybe more.
You want full documentation? Okay. Guy can't spend as much time on coding.
You want new APIs to be included? Okay. Guy's got to read docs and try samples. Less time.
What I've seen is you always end up with someone noticing that you could just spend all your time coding, and you would still produce an output that was ok. At least for small projects, or the start of large ones. It's as if there's a true productivity that's much lower than the imagined, and whoever is setting priorities doesn't notice it. He only sees code that's been written. He doesn't see technical debt. He doesn't see when a test catches a budding error. He doesn't see what that second pair of eyes is for. He doesn't know why you're always reading about some new thing.
Writing docs and unit tests now is cheaper than fixing bugs later.
I guess you're right that it is about budget because we're all trying to build a spaceship when we're only given some duct tape and a shoebox.
I've been explicitly told in one project to avoid quality and just make something cheap and fast. The project has been in that mode for the last year. Funny how cheap and fast actually means more expensive and slower.
On another project, the manager created different columns in our Kanban board suggesting that we must write unit tests and must have code reviewed and QA'd. Okay Mr Manager, now tell me where in your schedule and budget you're going to fit those key activities when you've accepted a fixed-time, fixed-cost project.
Underestimation is rampant in the industry and I really like this saying in order to justify the necessary work, "If you have time to do it over, you have time to do it right."
In these cases, some sort of unwillingness or inability to pause and be thoughtful, driven by a rush to meet a budget or deadline, cost a lot of money and time. Perhaps, as you suggest, it's the great difficulty of the lay-person or the manager judging the quality of software (or engineering I guess), and naturally selecting for the "cheapest" quality that seems to work, and very often hitting "too cheap".
What is truly criminal is not giving the developers at least a couple refactor iterations to fix all the monkey patch bandaiding put into making that happen. It's a mix of both management's naivete about how bad the underlying code is and their greed of wanting to cash in on the present victory at the expense of future technical debt (that hopefully someone else will have to pay).
I think at least some of this responsibility is on development. They should negotiate and secure the repair iterations for post launch in exchange for making 11th hour design compromises.
Spending time writing documentation does not mean less time is spent coding, it means less time is spent coding that day, but then that is offset by time gained when someone later has to modify it and can finish sooner, thereby increasing the time available for coding other features.
This article struck a nerve with me because I've been at it 11 years, and feel much the same. The one lesson that has stuck with me most is the quality vs speed paradox: focusing on quality makes you go fast, focusing on speed makes you go slow. The reason is that if you cut corners you are continually wasting time fixing yesterday's cut corners, instead of implementing today's feature. You then feel pressure to rush through the new features to make up for lost time, causing more bugs to stack up, and... In extreme cases the project reaches a virtual standstill, spending all resources fixing past mistakes. Usually that's around the time it gets canceled, no lessons learned, freeing up everyone to do it over on the next project.
So, the issue is not one of budget, but of accounting. You have to account for the future cost of debts taken out on the codebase, and when you do you eventually conclude that the numbers tell you not to take out that debt. Convincing a manager of that however ... well, let me just say that it depends on the manager.
you get the bonuses every quarter.
Exactly. But when managers don't see it that way, you have the utilization paradox: people are constantly working but constantly behind.
>So, the issue is not one of budget, but of accounting.
Yes and no. The two are intertwined. If accounting and estimation were correct, the budget would be correct on average. Instead, the budget is on average less that what it's supposed to be, because people do not account for everything.
A project manager can only get you so far, especially if the team dynamic is rotten or even sub-par.
Management is never really the answer to me, even saying that as a coder who splits time as a manager. At the end of the day, if your team doesn't have the right dynamic -- soft skills or otherwise -- I don't think they've got a great chance of success regardless of how much budget or process you can throw at them.
One thing to note is when the guy calling the shots is not a programmer. It's simply hard to see all the things programmers have to do, other than typing out code. It's also hard to see why some guy wrote a bug if you're never the guy who is responsible for it.
Have a look at sports teams. The coaches tend to be ex players, even if at a very low level.
The other thing about manager types is things never go wrong for them in a way that is purely them. When I worked in marketing, I'd never get stuck on a PowerPoint slide for days. Problems were always organizational, and someone else was always involved. Sales guys aren't selling fast enough. Fab guys are stuck with their new process. PR people have the wrong message.
It's very easy as the organizer to just think all the problems are due to other people.
If you work at a place where the client isn't willing to pay for good quality software you will never learn how to make software the right way , simply because there is no time to do it the right way.
This also boils down to the ego thing, many junior developers are proud to give short estimates to show off how quick and good they are at their work. But those short estimates are always always always just the estimated time to get a quick prototype working, where things are configured as you go in the debugger, not something that is robust and will work together with the rest of the application in a customer deployed environment. Estimation is really where you can see the difference in an experienced engineer and a non experienced one.
I work as devops and my boss always says that good operation/support people are hard to find. One can be a super hero in programming, but really bad at customer service or supporting production. Similarly some ops are very horrible at coding, but they are great system administrators.
With that, I have had product owners and product managers who are really excellent at managing team and able to cope the lack of technical skill by absorbing the technical knowledge from daily standup and eventually able to work with the team to prioritize technical challenge. For example this one product owner works in big data and he couldn't ssh without me showing him, but he could go over the pipelines just enough to make me feel embarrass. Of course, if you are on a project long enough you should know how things work in general.
On the contrary I have had really senior technical people leading teams and eventually got fired for their inability to lead.
The truth is that individual engineer skill is the only good bet, and only if within a culture that works.
d--b, you're hellbanned for some reason. FYI.
While everyone understands that maintaining a physical infrastructure like a highway involves a large amount of energy (create an alternative road to redirect the traffic, fix the road, destroy the alternative road), it is not true of software. Because what happens in the code is not obvious, clients and managers have no idea of the effort that would be required to change an existing piece of software without disrupting the current flow of operations.
Let's take a more practical example: The boss asks you to create a datatable in a database to store client information. And according to the first specs, each client has one address, so you go on and create a table that contains client id / client name / client address. Everything works fine, great! Now, you do your demo, and suddenly the boss tells you: hey I've got 2 addresses, can I have my 2 addresses stored in the system? You have then 3 choices:
1. Tell your boss: no there's only 1 address in the system 2. Tell your boss: ok I can put 2 addresses, but then it means I have to split the user table and create an indirection. So I have to rewrite the whole address read/write layer. 3. Tell your boss: sure I can do it. What I'll do is take my first design, add 3 or 4 'additional' addresses to my table (who's got 3 or 4 addresses anyway?) and I'll just have some minimal changes to make to my code.
In most cases, if you're the developer, you'll opt for option 3. You may think in the back of your head that this is disgusting and that you'll change it later, but any of the other two options will make you look pretty bad.
Later on, the boss asks you, hey you know what would be great, is that we can lookup users by their addresses. And now you find yourself having to deal with these additional addresses columns all over the place, and here starts your ball of mud.
I think you see my point. The problem is not a question of money, it is very much a problem of understanding the underlying structure of a project.
Are we doomed to fail? Maybe.
So I think the problem isn't money; the real problem is there just isn't enough software! We have a software deficit. With software being a critical part of nearly every business in the world, there is huge demand but massively insufficient supply. In my industry, there are only two big software providers and a handful of smaller ones. And it's a huge industry in both money and sheer size. But they literally cannot buy anything but big balls of mud.
People are paying big money and small money for whatever software they can get. And that natural consequence is that performance, reliability, and security are hit or miss.
They might not get it, of course -- the money is an (often ineffective) stand-in for institutional/organizational change, AKA "throwing money at the problem."
It's simply easier to get people to spend gobs money than to change their comfortable short-term habits. (Evidence: Wasted gym memberships everywhere.)
The lack of discernment of the buyer is creating all this software waste. This is one of the reasons that MS Excel and Access run most of the world's financial data. The format is open enough that it basically imbues the user partial programming ability without having to worry about things like new line feeds and uptime.
Painfully true. There are techniques for writing better software. They work. They take longer and cost more. They are not widely used outside aerospace.
It's unfortunate, and it's sad to say I'm often part of it at some level (but I fight it dammit!).
I asked him, as delicately as I could, what had happened.
This project we were working on, he told me, had been going on for about half a year, and during that time there had been a meeting every week, where everybody involved in the project had an opportunity to get together and review and discuss the progress of the project.
So there was this guy, who had only attended the very first of these meetings, without saying a single word. And the week before the thing is supposed to go live, he shows up again, with a long list of changes he wants to make to the application.
Needless to say, that programmer, and several other people in the room, told that guy how he had six freaking months to tell the programmers about his requirements, and the he could not possibly expect them to make all these changes a week before the application went live.
So that guy goes to the highest-ranking manager involved with the project, manages to pull some political strings, so that manager goes to this programmer's boss, that boss goes to the programmer and tells him, "I know how much it sucks, but you have to do this. I share your pain, but I, too, am powerless to refuse this request."
When building material things, it seems, houses or ships or airplanes or railraods or whatever, people do realize that you cannot tell, e.g. the construction company that you actually want a house with a circular outline rather than the rectangular one the company has been building for the last, what?, three months. Managers at car companies, I guess, do not storm into the engineers' offices a week before production of a new car starts to tell them the vehicle not only needs to be small and fuel efficient, but also needs to be able to work in antarctic climate and run on carbon dioxide instead of gasoline.
With other products - at least, that is the impression I get - people have an intuitive or explicit understanding of the limitations of the things they want to have built, and they also seem to basically understand that you cannot make drastic changes at the last minute. With software, which does exist in the tangible way that, say, a car our house does, people seem to have a much harder time understanding those limitations. I am not sure this is the entire problem with software development, but it is a big part.
Maybe not as much as people think. There are machine-assisted mechanisms for building mathematically perfect software. See Curry-Howard oriented languages like Coq and Agda. It's certainly harder to write perfectly, provably correct programs, but maybe not as hard as most people think.
There is also a lot of middle ground in the form of relatively powerful (but not quite Agda-powerful) type systems.
As for type systems, mission-critical software has generally made good practice with them (Ada, proof checkers, etc.) More so than consumer and enterprise circles. Yet a ton of catastrophic bugs have been the result of errors outside the scope of type checking, and as a counterpoint, our telephony is pretty robust with dynamically typed Erlang switches. It seems that mechanisms for building self-healing and concurrent systems are often put aside and equivocated solely with strong type systems. From what I recall, Erlang's signature feature of hot code loading actually conflicts with static typing.
Being wrong — rather, being incorrect — is an extremely humbling experience. The catastrophically incorrect, which is when software crashes, money is lost, or the absolute worst, data is stolen, is the kind of thing that makes you question your career choice. It makes you want to curl up into a ball and weep at how completely fucking stupid you were when you’ve found the problem.
Just had to quote this - I urge people who stopped reading or down't want to read the article to read this anyway. I've read a lot on developping but I think it's the first time I saw someone putting it like this. And oh boy, is he right. At least for me. Every bug report (well, the ones which points to something I obviously fucked up) hurts. What hurts even more is the dreaded reopened because I fucked up again. Especially because sometimes that means the whole set of classes surrounding the bug are just textbook examples of all code smells in the universe. And the only thing that can be done about it is the nearly impossible write good code 100% of the time.
Of course they try to write good code, follow good practices, write tests etc but there's just going to be bugs, period. So don't beat yourself up. Just fix the bug, learn whatever you can from it so hopefully you won't do it again and or if it calls for it adjust your build infrastructure or testing infrastructure so you're more likely to catch them in the future.
The baggage associated with bugs is so prevalent that we can't talk about them without a real sense of shame. "Hopefully you won't do it again".
Bugs are just part of the cycle. I completely agree with your assessment that people who can realize that are happier and more productive because they can plan for the probable case.
Bugs in a tracker don't hurt, bugs not in a tracker hurt.
So it really is safe to assume that everything you do is wrong. It is nearly impossible to tell the difference between right and wrong when you take action. And wrong is so much more likely than right.
> ...just after the first successful moon landing [...] I knew that each Apollo flight required some 40,000 new lines of code. [...] I was duly impressed that they got so many lines of code correct. So when I met Joel, I said, "how did you do that? [...] Getting that software right." [...] He said that in one of the calculations of the orbit of the lunar module the moon had been defined as repelling instead of attracting. They had discovered that error by accident. Imagine, by accident five days before the shot.
Sound familiar to anyone?
Some 16 years experience myself, the clarity of "the problem of time" has become the foremost factor of bad software that I knowingly write. I imagine NASA has the same problems that we do.
I've been through about 5 cycles that all occur in the same form: 1) We put the under-performing developers on maintaining "legacy" systems and the seniors begin on the next one. 2) Time is purposely estimated shorter than viable, corners get cut in the software, prototypes are put into production once again, and a new legacy system is born. 3) The good programmers leave, some juniors become seniors and others are hired, and the cycle repeats.
There are many additional mismanagement mistakes that accumulate, but limited time always seems to be at the center of the problem for me. I whole-heartily believe that really great software can be authored by just two or three high quality engineers over the span of a few to many years with a clear vision.
Author starts by mentioning huge, complicated and ambitious projects and pointing out how seemingly insignificant mistake makes all difference between "it works!" and "it doesn't work!", which sure will sound familiar to what we experience every day, but on different scale. With this impression he moves on to talking about modern software in the industry and at home, and how we fail to write correct code and why.
But in reality, nobody (except the programmer himself, maybe, if he fails to see the bigger picture) gives a fuck about correct code. Glitches and errors are ok, if they don't make software unusable and prevent us from achieving some global business-goal. It's ok for landing-site to have a CSS bug, that makes some button look weird in certain situation, unless it really annoys the customer and we lose sale — the chances are that nobody even notices, and it may be fixed next week or even never. It's ok to put your business-critical back-office software on production and find out that it contains some really nasty bug, which you'll fix in a hurry in the next 20 minutes. It's ok for Ubisoft to spend millions of dollars on production and then ship game with bugs, which will be fixed with the next patch-release. It's ok to have stupid over-complicated networking protocol, that might have been 5 times more efficient, but still is usable and allows people to do something they couldn't have done before.
What is not ok is to spend millions of dollars on development and then never ship the product. It is not ok to let your client to find a solution which works for him, but is cheaper. In a word, whatever actually hurts your business is not ok. And the truth is that cheap solution that is "somewhat ok" is usually better, than the expensive "good solution".
I think it's entirely unjustified. What makes modern software engineering difficult is that it's one of the few engineering disciplines where not only are you making a Boeing 747 fly, and all the parts were made by different people in different locations, but you're also replacing the engine in flight!
Also the specs on the parts are kind of spotty. They weren't agreed upon by a single group of people, and they certainly weren't decided by you. Some of the parts are decades old. There are parts in there that are older than most of us!
Often, the parts decide to change how they work. They don't ask you whether they should. Sometimes, also, the parts decide to disappear. It's up to you to figure that out.
In fact, there aren't really specs on how the parts go together, come to think of it. None of the people who are building this are talking to each other. You could say that most of it is kind of emergent. There are no top-down quality controls.
There are layers upon layers of abstraction, and no one knows all of it. No one even knows most of it. In fact, most people don't care about any of it other than a cubic centimeter of their own few layers.
And it works - by golly it works. And a person with no training can absentmindedly navigate all this by flicking their finger about while walking down the street and drinking their coffee. IMHO, this is a miracle in proportions that are indescribable. The moon landing really was nothing compared to this. So yes, there's some overhead involved in getting that to work. :-)
I used to believe that, but I have slowly moved to the other side.
Things are complicated. There is no doubt about that. But, landing on the moon was a fight against nature (gravity, air, distance, etc). Current programming is a fight against stuff someone else dreamed up and no one ever fixed, because "LOL, that's old school" or "You're just doin' it wrong".
Most of the points about parts not being specified correctly, not working as expected, or disappearing happens all the time in other industries. (That's one reason hardware kickstarters fail so often.)
Which is why you can go to the moon with a decent graphics calculator, but you definitely can't have a self-driving car with much less then a modern supercomputer.
Could you elaborate that, using a mainstream example, please?
> Also the specs on the parts are kind of spotty. ...
Are you referring to the moving parts of the Internet itself?
> There are layers upon layers of abstraction, and no one knows all of it. No one even knows most of it. In fact, most people don't care about any of it other than a cubic centimeter of their own few layers.
This is an invented problem. Scaling a one-off to work for the masses does not require `layers upon layers of abstraction'. Most of those layers are poor engineering backed by a lot money that is thrown at hardware and code maintenance.
One of the realisations that I had over years of `fixing crises', was that increasing numbers of `start up' programmers produce write-only code. Testing and debugging them is painful and expensive, even in the medium-term. Such code bases demonstrate a clear lack of basic engineering discipline. Nonetheless, such programmers are hailed as heroes, because they managed to cobble up a nice-looking mess in a week or two. Then, enormous amounts of effort are poured into making the mess work tolerably. It also consumes phenomenal time.
`Moon landing' projects cannot countdown and blast-off with a great `UX', and then start figuring out how an inertial navigation system should work, over the next week-end.
So, yes that page that `takes so long to load' is indeed `a miracle in proportions that are indescribable'. Only, in a mostly pathetic way!
But if you start with the idea that you're going to write solid code on principle then you're just throwing good money after bad.
Again, building a one-off is easier than a consumer-ready working system deployed across millions of locations that is owned by no central authority and has to run continually. That's why we had SHRDLU (https://www.youtube.com/watch?v=QAJz4YKUwqw and http://hci.stanford.edu/winograd/shrdlu/) and The Mother of All Demos (https://www.youtube.com/watch?v=yJDv-zdhzMY) years ahead of their time. That was my point.
So most developers are just trying to make a 747 taxi to the end of the runway. Or even just sit there without its wings falling off.
We'll get it to fly in the next version.
... writing good code (that others, including yourself 1 year from now, can maintain easily) that works well ...
Otherwise, I really do like reading articles like this as touchstones for engineering and development.
[1] http://blog.codinghorror.com/coding-for-violent-psychopaths/
I lifted all of the comments from stackoverflow, too: http://stackoverflow.com/questions/184618/what-is-the-best-c...
I'm curious where the egotistical portions were, though. I wanted to at least prefix some of the more "thou shalt" sections toward the end with a disclaimer that they're personal opinions on how I operate.
Glad you liked it, though!
[1] http://blog.codinghorror.com/the-noble-art-of-maintenance-pr...
I pine for a golden, future age of programming where documentation is as he describes.
> This image brought to you by preventable catastrophe, budget overruns, political wrangling, infighting, incalculable complexity, and against it’s best effort, humanity.
Beautiful caption. The dismal history of NASA in one sentence.
There is plenty to Monday-morning quarterback, but "dismal" is not the word I would ever use.
What's so inherently wrong about software? Why is it so damn difficult to write software that doesn't explode if you don't hold your tongue at the right angle?
Hubris mostly.
> It contained a defect of effectively point zero zero zero zero one inches. that’s 0.000001.
If you can boil your software down to parts that can afford latency (even if they need throughput), you can pass messages between them. That is possibly and probably less efficient, but your macro pieces that pass well defined messages between each other can exist without adding to the complexity of the other pieces.
One interesting issue I see is with specific documentation rather than 'self-documenting code': Considering docs don't compile, I have seen (in my 13 years ;) that docs tend to drift apart from the actual code and become more of a PITA than the code itself. At least once I have caught myself down a wrong path because I trusted the docs.
I don't have anything against docs though - write a bunch myself: I am very circumspect against trusting it completely though.
1. I'm not a TDD purist but I do think a red/green TDD cycle does strongly encourage all of the following things to happen in a more natural, flowing way: "Tell yourself what you’re going to do, and then implement that sentence. When you’re finished, take that sentence and write a test, then just move on." Coding before testing isn't the issue as much as the potential for needing to change code just to be more testable (an efficiency issue).
2. No sale on the idea of documenting every piece of code; I truly feel like doc comments muddy the contents of the file, plus resulting documentation is less likely to be read than code and more likely to fall out of sync with the contents of the code. Completely separate documentation is hard to keep updated. It's often something to maintain with often no tangible benefits unless you are churning developers like crazy. Certainly it might be reasonable to document high-level interactions for complete view of the system but documentation at lower levels seems an increasingly bad value proposition to me. In a perfect world, sure... but in real life there are deadlines and tests+code almost always serve as enough of a developer guide to understand clearly. If not, I'd rather spend time refactoring instead of writing docs. For the few new developers brought on, reviewing the system in a paired programming manner is a good way to get to know them and ramp them up at the same time. Parts of the system that are difficult for them to comprehend might warrant refactoring attention.
3. "If something is performed more than three times in a 4 hour span" It's difficult to make a wholesale suggestion on things like that. If said task is 1 second clicking the "get mail" button and automation is auto-checking every n minutes, automation can be a net loss in terms of productivity. I think the XKCD[1] chart is probably a better measure ;) Anyhow it's definitely better to not waste loads of time doing things you do often, and the more often you do something the more likely it is that automating the task would pay off; just don't do it with an utter lack of discretion.
Some typos in the post: - Stray period before "I just call it by what it was: A big ball of mud." - "Simple isn't sexy." should maybe be "Complex isn't sexy" or "Simple is sexy" - "internet" (proper noun) - "explaining just how the interact with" s/the/they - "english" (proper noun)
Very thought-provoking and fun read (lots of laughs)- I really look forward to future posts!
Maybe 5K words without any sort of lists or clickbait titles is just too much for the average reader.
It's not cause the tone is accusatory, or because it rambles, or compares mission critical life-or-death Apollo software with the unimportant web and native applications we write now. Yep, it's cause people are too dumb to look at that many words without a beer commercial dropped in the middle of it.
Saying "people who disagree with my opinions are too stupid to understand the subtlety of the point" is the rhetorical equivalent of denial in the Kubler-Ross sense of the word.
According to a copy+paste of the text into http://sarahktyler.com/code/sample.php, the article as a whole reaches Flesch-Kindaid grade level 7.
