This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects.
I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity 2.0 with 100% more pull. And nobody will ever tell you that your bridge will now get a shopping mall in the middle of it where people can purchase products of their favorite brands.
Software starts to break down when it has to be taken above initial design constraints and when there is not enough time to rewrite subsystems (or all of it) but instead when you have to make the abstractions leaky and compromise.
But back to qmail:
qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail. Instead everybody is running "qmail" which is qmail plus some patches. Sometimes home-grown, sometimes taken from third parties.
But more often than not they are unmaintained and very far removed from the high quality standards of the underlying software.
This is the downside. Yes. You have a bug-free core that totally meets its designers (limited) use-case, but in reality nobody is actually running that.
Having the correct architecture, and being able to rewrite the subsystems piecemeal means it is possible for users to experiment with new features organically[1]. That's part of why some qmail installations have features not available in any other mail server even today.
Or to put another way, software purity and homogeny isn't a "good thing", but a trade off: You get to share risk with everyone else who chose like you did, but you're also stuck with the same features and risk everyone else has.
I'd choose "feature-poor download, but highest-feature in production" over "a-few-more-features download and limited-ability-to-upgrade" any day.
[1]: If you're curious, some of the experiments I did are briefly mentioned here: https://news.ycombinator.com/item?id=16166530
Our web frontend for example worked on top of a slightly modified Qmail POP3 server that relied on encoding some message state in the filename, so we need only scan the Maildir without opening the files to be able to e.g. get message read state, and various flags. We also added caching of metadata etc. The changes were tiny and self-contained, and added one extra non-standard command to the pop3 server to retrieve a message list with much more data. The design of Qmail let us start with just changing the pop3 server, and later some tweaks to local delivery, and know we could test those programs in isolation.
The flexibility of Qmail even got us to use it as a messaging middleware later coupled with tinydns - all the routing and retry logic made it very convenient and extremely simple to troubleshoot.
I personally consider this backscatter issue a design bug in qmail.
We certainly should have switched mail servers, but qmail was deeply ingrained in our home-grown hosting automation system, and it would have been a big deal to change.
There were patches to fix the problem, along with offering useful features, but for whatever reason we went with exim (exim 3.x from Debian).
Instead everybody is running "qmail" which is qmail plus some patches."
I ran and continue to run qmail without any patches.
The above quoted statements thus cannot be true.
But maybe "nobody" and "everybody" are figures of speech?
Also, how are you dealing with backscatter?
are you sure? How can you be sure that your custom patches didn't affect the security of the core product? qmail wasn't designed to be extensible. It had no plugin interface.
Of course it's possible that you didn't make a mistake back then.
Just as it's possible that I didn't make a mistake when I was 18 and wrote a patch to Cyrus imapd to allow authenticating against an SQL database.
But TBH, when I look back at the code I wrote back then, at least in my case, I'm quite sure I f'ed up in various ways.
Thankfully, I never shared these patches with other people.
I had Qmail running circa 1998 and I'm pretty sure it was vanilla, no patches. But after all this time I could be wrong.
Anyway, shortly thereafter I discovered Postfix and that was the end of my relationship with Qmail.
(1996) https://www.fastcompany.com/28121/they-write-right-stuff
> It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
> The process isn’t even rocket science. Its standard practice in almost every engineering discipline except software engineering.
The problem is consequences. We had centuries of people dying in bridge collapses before we got our shit together and started prioritizing safety in civil engineering (i.e. engineers and managers going to prison if they don't).
The same will be true for software. As more people get harmed by thrown together software (e.g. mass panic in Hawaii, state psychological exploitation on social media), we'll start regulating it like other engineering fields.
As a former chemical engineer, I welcome this transition, but I realize it will likely also take centuries of hard lessons.
Come to SE Asia and I’ll show you plenty of civil engineering projects with similarly poor planning and execution.
I can't find it now, but one of its points of comparison was: if a nut isn't tightened enough on a bridge, you can tighten it a bit more -- turn the nut 3 or maybe 6 millimeters more. In software engineering, if you tighten the nut only 3 millimeters instead of 6, one of the bridge's endpoints now ends in a different dimension.
I think there is a valid point to it. I'm always amazed in housing construction how things are done "approximately". A wall should be vertically straight... give or take a few centimeters. That kind of approximation doesn't work in software engineering. It has a whole other type of complexity to it.
For that matter, let me know when your engineers have built something as complex as the AWS infrastructure, by which I mean, every single service they offer, by information content (something like Kolmogorov complexity). There are some disciplines like power management that certainly have some very large and impressive things that I don't wish to diminish, but the idea that software engineering needs to go learn from all the other engineers is an idea stuck in a very 1970s view of what a "large project" looks like.
I am abundantly confident that if you took the amount of engineering done to build something like the Golden Gate bridge today, then took all those man-hours and saw how much software you could get for the same amount, that you'd be surprised how small the result is. Not that it wouldn't be a good chunk of hours to play with, and it would be much larger than many phone apps or web sites, but if you stack it up to something like a browser or a usable OS kernel I'm pretty sure you'd find that the very visually-imposing bridge is actually orders of magnitude, plural, simpler. I say this not because I don't respect civil engineers, but because I also respect just how quickly big computer projects can chew through the man-centuries. No bridges would get built if each one individually required engineering effort commensurate to a new web browser.
It's past time for programmers to get over their inferiority complex. A sober look at what the programming world does vs the other engineering fields shows that all things considered, we're doing pretty well. Still a huge amount of room for improvement, but we're not doing so badly that we need to go running off to other completely different disciplines to drag in irrelevant, if not actively harmful, practices that are unconnected to the problems we face.
Especially if you're building something with more than one story, and you expect people to actually pay for it.
(The most common one is breakoff studs on the bolts, which actually speeds thing up because the wrench can grab the nut and the stud on one side of the connection and torque away till it pops)
Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run?
I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
The difference is that I don't then promise a municipality that I can build them a bridge that can take their citizens across a river, even in case of a 100 year event. Or an early warning system that can save their citizens from nuclear holocaust. This is the difference in maturity between software "engineering" and real engineering disciplines.
You're free to experiment with your own resources but as soon as you make a promise to the public or your customers you should be required to meet your promises in all circumstances.
After all, we've witnessed move-fast-and-break-things social networks (Facebook, Twitter, Reddit) that had no desire to influence society in a big way get hijacked by state actors to achieve political goals.
We're also witnessing toasters having web servers pre-installed on them with no thought into what happens if the company that pushes patches (if they do at all) goes bankrupt, thus creating vast oceans of botnets that the rest of us now have to deal with.
Again, I think there's a long road ahead of hard lessons when we don't take our own power in software seriously.
A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.
One big difference between those and programming is that there (outside of military, physical security, and few other exceptions) there are no active adversaries for your project. You design a X that will serve a defined purpose. You design for some parameters (known wind speeds or local natural hazards).
You don't normally design with someone actively trying to destroy your building in mind. You don't design thinking how to avoid the current issues and to isolate the impact of potential future attackers. Your product doesn't have a "well funded attackers will spend undefined amount of time trying to make your project fail" requirement.
You do, actually. When designing a building, one considers many "attacks" it can be subject to, depending on the geography and demography of the site: what are the precipitation patterns like, what is the situation WRT sysmical activity, how is the soil, what is beneath, what is the crime rate in the area, what kind of animals live there, where does the wind tend to come from, how much sunlight is received, etc. etc., about all of these questions are in one way or another is relevant to security of a building, and you have to consider each and every one of them, and design accordingly. You don't design against destruction because that's hardly possible, there isn't much you can do to mitigate someone bombing a building or attacking with heavy machines. Think of that as analogous to using the ME in an Intel CPU.
The article makes an unfair comparison; if the solution to the problem is like the one below (extract from the article), the change, reasonably, will never happen in commercial programs (the ones "with 5000 errors").
> Take the upgrade of the software to permit the shuttle to navigate with Global Positioning Satellites, a change that involves just 1.5% of the program, or 6,366 lines of code. The specs for that one change run 2,500 pages, a volume thicker than a phone book. The specs for the current program fill 30 volumes and run 40,000 pages.
- Tacoma Narrows bridge: https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge
- Thalidomide: https://en.wikipedia.org/wiki/Thalidomide
- Hyatt Regency walkway collapse: https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse
- Challenger disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste...
- Columbia disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste...
- Vioxx: https://en.wikipedia.org/wiki/Rofecoxib
...and plenty more. The only thing that separates software engineering with the other engineering disciplines is that there is a structure (usually in the form of code/spec enforcement) for internalizing and learning from disasters when they happen.
And it’s the dominant image of the software development world: Gen-Xers sporting T-shirts and distracted looks, squeezing too much heroic code writing into too little time; rollerblades and mountain bikes tucked in corners; pizza boxes and Starbucks cups discarded in conference rooms; dueling tunes from Smashing Pumpkins, Alanis Morrisette and the Fugees. Its the world made famous, romantic, even inevitable by stories out of Sun Microsystems, Microsoft, and Netscape.
It's incredible how much software development has changed over the last 21 years. Change Gen-Xers to Millennials, update the playlist, and replace Sun Microsystems, Microsoft, and Netscape with Apple, Facebook and Google.
I don't do a car inspection at a guy that just happens to know some stuff about mechanics.
The leads to a race to the bottom for which both the software vendors and consumers are responsible. Everybody wants the latest stuff as soon as possible, and wants to play less. If you release software only when everything's "right", the market will have moved on.
... unless we the people demand legislation that protects us from software "vendors" selling us their bugs.
I think it's interesting that people can see a car that doesn't work, and a house that's falling apart, but they can't really see software. Can you imagine someone saying "my house is leaking" and the landlord saying, with all seriousness: "Have you tried turning it off and on?"
Of course, we have legislature for those things, so why not software?
I.e. we rarely hear about plane crashes caused by software, or power plants accidentally blowing up due to an integer underflow.
Civil engineers make solid bridges, but most software is not the golden gate, it's a shed thrown together by some guy who's not an engineer nor a professional mason.
We could make cars that don't kill people.
We could make software that doesn't break.
I recall setting up qmail "toasters" on FreeBSD to do virtual hosting. Maybe I was just too much of a "n00b" but I remember it being a big PITA to get all the services to play well together. There was this hip new outfit named Yahoo! that was using it for their new webmail service, though -- as opposed to sendmail, which pretty much every MTA on the Internet used at the time (and I was proficient enough with sendmail that I would edit my sendmail.cf by hand; pffft, who needs m4!?) -- so I assumed it was certainly capable of handling my volume of mail. (I wasn't running authoritative DNS servers at the time or I probably would've used djbdns over BIND as well.)
qmail, unfortunately, never did become too popular (relatively speaking, of course) and that's really a shame, because, as the quote in the article says:
> "We need invulnerable software systems, and we need them today, ..."
While that was certainly true then, it's even more true now.
On a side note, I'm surprised that the "qmail security guarantee" [0,1] wasn't mentioned in the article:
> "In March 1997, I took the unusual step of publicly offering $500 to the first person to publish a verifiable security hole in the latest version of qmail: for example, a way for a user to exploit qmail to take over another account. My offer still stands. Nobody has found any security holes in qmail. I hereby increase the offer to $1000."
[0]: https://cr.yp.to/qmail/guarantee.html
[1]: https://cr.yp.to/qmail/qmailsec-20071101.pdf (PDF)
At one point, it was the second most popular MTA on the Internet. What pray tell would "too popular" look like?
> I remember it being a big PITA to get all the services to play well together.
When you were thinking about qmail correctly, it was an absolute pleasure to get everything to work together. Promise. Yet whilst the documentation was correct, it probably wasn't very good from the perspective of helping people think about it correctly. André Oppermann[1] (and perhaps Dave Sill[2]) did a much better job of this, so when they came available I would usually have pointed people there and see what kinds of questions they still had.
* https://cr.yp.to/maildisasters/postfix.19981221
* https://cr.yp.to/maildisasters/postfix.html
* https://groups.google.com/forum/#!msg/mailing.postfix.users/...
Software like qmail and the dev file system at the time really rubbed a lot of people the wrong way because of the drastic design changes they push. I'm glad that particular dev file system died as it had a lot of weirdly named nodes and a devfs daemon that had to run to create symbolic links to all the known names.
If it wasn't for all that, we might well all be using qmail-based mail servers today, as qmail was really ahead of its time in so many ways.
It was kind of like the Amiga of mail servers, back in the day. It could have easily dominated the market, but it wound up a mere historical curiosity.
That setup contributed to security.
It also made qmail very easy to extend: I operated a medium-sized mail service a while ago and qmail's pluggability meant I could add features that didn't exist in other mail servers (like postfix or exim) without forking the entire project.
* I had SMTP AUTH when the only other mail server to support it was Netscape; before RFC2554 was written.
* My qmail-popup proxied to another machine so it didn't need root (making me immune to the Guninski vulnerability) so my users only needed to know about a single hostname regardless of where their mail was stored, and without needing to use something icky like NFS.
* I had a web interface with auto-enrolled client certificates for authentication and confidentiality for SMTP, POP, and IMAP.
* My qmail-remote recognised certain suggestions our IP was being blacklisted and would retry immediately with a new IP.
* My qmail-remote recognised certain greylisting error messages and rescheduled retry for that time.
* I had multiple mail queues based on the number of retries the message had seen.
And so on. I didn't start out to make those features, they just grew over a decade or so organically. At no point would I have forked postfix or exim to add any of those features because once you fork it you own it unless you can get your changes upstream. I had shit to do, so the real alternative was simply buy more servers and/or pay for commercial software.
I wish the model had caught on, because it's a superior way to develop software. I didn't understand why though until fifteen years later...
> …and unwillingness to share and collaborate on its development…
Dan absolutely collaborated, and I certainly was using betas back in 1996.
If my memory/anecdote isn't enough: There are a number of explicit points of evidence in the changelog distributed with qmail.
What he doesn't do is let people save face when they say something incredibly stupid and then try to backpedal when it's obvious how wrong they are. This bruised more than a few egos, and contributed to a campaign to actively smear his name and discredit his software.
> It could have easily dominated the market…
I think if Dan had let the peanut gallery have their way, we probably wouldn't have gotten postfix, but then qmail wouldn't have been qmail except in name. What's the value in the qmail brand if it isn't secure anymore?
Yes, they have been feature locked for a long time. But let's not pretend that collaboration in 1996 was anything like it is today. And let's not pretend that there haven't been a plethora of security issues in all kinds of software that simply haven't befallen Bernsteins' software as a direct result of how he managed the projects.
I had the distinct impression that "dominating the market" was never a thought in his mind. He wrote the software he wanted to write and shared it. It was good software. It's still being used by many people.
The man isn't infallable. He isn't always right. But whenever I paid attention to him, he was right a lot more frequently than he was wrong.
The fact that he wasn't driven by dollars or market dominance was a good thing.
I, for one, admire the man and his software.
Can you elaborate?
The licensing stuff, I'm not super clear on. If memory serves, DJB did not want to worry about bugs introduced by patches he'd never seen or approved, added by package maintainers for OSes he didn't use.
As for the somewhat weird ecosystem (daemontools), I think it wasn't that it wasn't good enough, it was just that people always find reasons to be dissatisfied. I can't even keep track of the latest reinvention of whatever people are using instead of daemontools, but I bet it's a hundred times as complex, and much less reliable.
* http://jdebp.eu./FGA/law-licence-free-softwares.html
Daemontools ironically wasn't weird at all, as evidenced by the fact that over the past almost two decades the daemontools world got us changes to softwares, all of those do not daemonize options that have appeared in that time as well as things like removing mysql_safe and other Poor Man's Dæmon Supervisors, that made it a lot easier to use those softwares with other service managers.
* http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Promp...
Some of the things that people use instead of daemontools are not much less reliable. (-:
In his example, yes, you could change the DNS responses, but you still could not escalate to a higher lever where you can potentially modify stored user data. That is a success in practice.
The only protection from this is to make the code that does this as small as possible so that us human beings can convince ourselves that it is correct and that the risk of a bug that someone can control is zero (or as close to zero as to make no odds).
When Jim Reid wants to pat himself on the back because "at least they didn't get root on my nameserver box", he misses the point: gethostbyname()'s spec doesn't say "it may or may not return. if it returns it could return anything. don't trust it, don't even use it!" They say gethostbyname() return a structure describing the address of the named Internet host, so people expect that and depend on that. Something that "suddenly" violates that gets in the news[1]. Fortunately, nobody remembers what Jim said so the BBC doesn't ask him for a comment.
Anyway.
"Minimizing privilege" doesn't solve that problem because the DNS server needs the privilege to respond to DNS requests.
It might be easier to think about a better example. Let's talk about zlib.
A program that needs to decompress some text is not concerned with the contents of the compressed text, only the uncompressed text. Resource limits on our program exist to keep some things from getting out of control[2], but what about bugs?
If we could run zlib's decompress() with the permission only to decompress text, then the worst-case impact would either spin the cpu or be equivalent to "getting out of control". What do we need to do that?
• No creating file descriptors can be done with setrlimit() except for the dynamic linker is going to open a shittonne of files. We need to know what the minimum number of files are, and decompress can't ever change that without changing our program anyway.
• No accessing files or the network could be done with a setuid wrapper and iptables. At least on Linux. Most programmers don't do this, and most sysadmins only do what they're told, so in practice this doesn't happen.
• Sandboxing! Google published some clever user-level sandboxing that works on Linux to whitelist each syscall. This "verifier" could do it as long as it's smaller than decompress()!
That sandboxing one is tricky: A tiny inflate routine takes around 500 lines of C done the normal way, but how big is our sandbox? Probably a lot bigger.
• Ask the operating system for help! This is what DJB suggests. Ask for a disablefiles and a disablenetwork system call. OpenBSD is implementing this with their pledge[3] system call.
There's not a portable and satisfying solution here yet, but you can see they all cluster around reducing the privileges of the untrusted program.
Now, what's to prevent decompress from lying? What if someone can produce a content stream that causes a future decompress run from producing invalid results. Maybe something really sneaky[4]. What possible protection could we have?
As you can see, in this case so long as decompress is supposed to produce "text", there's nothing we can do to make sure it produces the "correct text".
That's why DJB doesn't want to focus on the "untrusted" aspect, and instead on trying to solve the problem that we have to solve anyway: How do we write software that is correct?
[1]: http://news.bbc.co.uk/2/hi/technology/7496735.stm
As soon as you build non-trivial systems, you have to contain error propagation with POLP, although you are striving to build simple and secure systems.
Taking the DNS Helper example, one could imagine a function-like DNS Helper which has the capability only to return a value. This would make libresolv just a bug, not a security hole, because the attacker would only pervert their own request.
As others have pointed out, one thing left out of the paper is not updating the software. qmail doesn't support SPF or other security extensions, which makes it useless these days without patches.
