I self-hosted for well over 20 years, I did not throw the towel and I do not plan to. Self-hosting is a sign of pride. Neither my government nor my Prime Minister nor even my Ministry of Interior or Foreign Ministry can host their own email.
Last time I checked, only State Security self-hosted.
I was probably lucky, but I rarely had delivery problems. The last one was a couple years ago with Microsoft swallowing my emails and it was due to the combination of a fairly old exim and a TLS certificate verification quirk at *.protection.outlook.com. I found a fix in the form of a configuration option somewhere on SO.
In all fairness, there is very little maintenance involved, and whenever I have to do maintenance work, I take the opportunity to learn something new. Like this year, I decided to finally replace my aging Debian jessie setup by Arch Linux, and I rewrote all cron jobs as systemd timers.
I must admit that when I send a really important email, I check the mail server log if it went off without errors, but this does not bother me as checking logs manually once in a while is a good thing anyway.
Lastly, a piece of advice: treat self-hosting like a hobby and learn to enjoy it.
Oh and the very last thing: the person who designed Exim configuration for Debian deserves a special place in hell for all the hours wasted. If you set up Exim on Debian, just figure out how to use the upstream exim config and adapt it to your needs.
Man, I wish I had 1% of the motivation I had 20 years ago to do something like this, before all the full time job, wife and child.
I really don’t want to live in a world where only two or three companies run email for the entire world, and this is my little act of resistance.
why Microsoft is so crappy?
I think when we’re building something with “good UX” the major point of “does this remove agency from users” is somehow missing from the picture. When everything runs on some kind of system, it’s not extraordinary to expect people to know how it works and maybe be able to do it themselves.
Otherwise, fast forward a decade of simplifications, and we can’t even install an app without someone on the other side of the world approving the “transaction”.
This is why I have stepped away from a lot of my self hosting. I have turned my attention/time elsewhere. Apparently though the time/money balance is shifting a bit again, so it may be worth it to go back.
My biggest hesitance to self hosting email specifically is dealing with spam. What does that look like these days and do you have any pointers to share?
Postfix can easily be configured to reject incoming emails from senders without a reverse DNS mapping for their IP address, which makes it reject a lot of spam.
For spammers with reverse mapping greylisting still works fine, they almost never retry.
Certain commercial spammers (hello China :-0) use software which can be filtered with a just one rule matching their sending software, which is "nice" enough to display its name in their mail headers.
And last but not least spamassassin / rspamd work fine to filter whatever comes through.
In the end I get less than 10 spam emails per week. And these go into a separate mailbox filtered by good old procmail, based on spamassassin's ratings. I check the spam inbox maybe once a week for false positives and more often than not the box is empty.
Overall the mail server is very low maintenance. I had to add SPF and DMARC a couple years ago (DKIM isn't necessary) and integrate TLS with letsencrypt (just a few lines in a config file), and sometimes a Debian upgrade requires reviewing the configuration (several years apart as well). There's really not that much to do.
The biggest issue is getting an IP address which is not in the banned lists. IP reputation is key along with SPF and do not send spam!
In the UK a "business" static IP address is sometimes/usually/probably/might be OK. If you are unfortunate then it is already in the lists and you can check that out at point of sign up.
You might look into IPv6 too. I managed to do the Hurricane Electric IPv6 email thing on my home connection for a laugh. That was a few years ago. It seems I need to do something more to get to Guru status.
Being able to check the server log can be very useful. E.g. to tell someone that their mail was delivered to a served using their domain name, with that IP-address at that time.
1. Because I couldn't ensure consistent backup and restore with regular monitoring,
2. no disaster recovery plan and in doing so it'd be more expensive than going through another email provider,
3. not always on top of security (my friend that I colo'd with also ran an email server and his system was struck with ransomware (with no backup [except a copy of email via thick client] or DR); I seemed to get away unscathed because I was using FreeBSD which generally less of a target).
I agree that it is little maintenance, but once you're off the happy path, it can be a huge pain in the arse and devastating.
email has easily one of the best responses to failure modes ever and its ancient!
Most smtp daemons will put outbound emails in a queue and run the queue. If the other end is unavailable then it will generally retry on a schedule with some sort of increasing period and then give up after a week or so.
You can easily define multiple inbound relays via your MX records which predate SRV and generic TXT and are supported everywhere.
I've run a lot of other people's email, including my own vanity domains for decades. It really isn't rocket science.
Google and MS and Co really don't screw you around if you follow the rules and that largely involves only SPF being compulsory and the rest (DKIM n that) are nice to have. If you do send spam then you will be crucified and rightly so.
Email is not a critical (its important) service because of course you have several other means of communication starting off with the SIP n RTP server you also run ... 8)
This is exactly why I only trust myself to do it. I almost lost my gmail account a couple of times in the past, and every time it was quite stressful. Since then, I use gmail as a backup email provider, than is, pretty much never.
Due to the way mail servers work, you have a couple of days to sort out your troubles before you will start missing emails. At worst, you can always buy Google for Work or some other SaaS and point your MX servers there.
Backup is always a hard problem, but I got to live with Hetzer Clould backing up my VMs, Hetzher Backup boxes as restic backup targets and a tiny Celeron server in the laundry closet for local backups.
No delivery problems if you set up everything correctly. It's not luck, just the same reason why well maintained car runs smoother than something that's seen last maintenance 100,000 miles ago.
Worse, GitHub (back in 2016 and 2018) would mark a recipient as "unavailable" after a single bounce, refusing to send any more notifications to that address. They since improved the situation and their support was actually very helpful and responsive here, but it's pretty clear that modern SMTP senders have an expectation that recipients will be "always online" that didn't exist when the protocol was invented.
I have never had anyone claim that their mail has not been delivered to me, and I get a lot of mail.
Retry is built in to the spec, and if you’re really worried you can put a second “receive” SMTP server on the internet with a lower priority, and have it backhaul with LMTP.
———
Email was designed in a time where hosts were not perpetually connected to each other.
I have Postfix logs showing things like "this address is receiving a high rate of email" which are later accepted.
Q: If your server(s) is/are offline for a few hours, why would you "lose messages"?
I've just checked my own email server -> "up 219 days"
Honestly, compared with the stuff we do all day, this is not hard...
They said...
>> Email is supposed to be resilient to down time (retries, trying each MX record, etc.) but I found that large mail providers tend to just bounce and walk away.
I take that to mean that if your server isn't availble to receive the mail at the time it is first offered, it won't be retried later. That wasn't the case (for most mail) when I gave up on self hosting 10 years ago, but it's plausible.
Host your own mail. I get 99% deliverability with 0 repuation since i do dkim and spf correct.
Don't be distracted by the "complexity" - if you config right it's totally doable.
Gives you actual private caldav too btw
Your anecdote of success doesn't matter to the others that correctly configured DKIM/SPF and still don't get their emails delivered to Gmail/Outlook/Yahoo/etc. E.g. : https://news.ycombinator.com/item?id=32715437
One of the reasons for hard-to-diagnose sending failures is that Gmail/Outlook have "extra invisible rules" that override correct DKIM/SPF settings because spammers and phishers also have correct DKIM/SPF. So they use extra heuristics such as "ip reputation" etc.
And even after one gets it working, e.g. "submit some form" to Microsoft and wait a few days to get things unblocked... the deliverability may break again because of another "invisible heuristic".
EDIT to reply: >No, that's because your relay overwrites part of the header which makes dkim strict break. Change to relaxed or don't modify the header on your relay.
Delivery reliability can still break without using a relay.
In fact, this unreliability of 100% self-hosting at home is why some self-hosters split it into a hybrid setup and add an external relay for outgoing SMTP and only keep self-hosting for receiving email.
The fact is, big email providers have all the leverage and you will have to play their game ($$$) in order for your email to work everywhere.
It happened to me and that made me realize it's not worth the hassle. Good luck
Use Mail-in-a-box to get started [1]. You can literally set it up in a couple of hours by following the instructions and everything should just work.
After a few years, you can think about switching your personal correspondence to your new email.
I've looked (and tried) a few other projects in the past, but Stalwart was the easiest to setup, and I haven't had any issues with it so far.
This looks really nice, especially also for saas projects.
They all look about the same from a newb's perspective.
That said - receiving account sign-up emails is the absolute biggest pain in the backside with Mailinabox! The greylisting anti-spam feature relies on bouncing unknown senders and waiting for a retry. The trouble is, many legit sites just don't bother retrying. So email verification for new accounts and 2FA-type stuff often takes ages to come through, if at all. MIAB stubbornly has no easy, mail user-facing way to temporarily disable spam filtering and it's a real PITA at times.
I see that the only way to disable greylisting is to configure the underlying tool [1]. But it also means that SPAM will increase a lot.
[1] https://discourse.mailinabox.email/t/how-to-turn-off-edit-gr...
Email delivery and receiving is not hard, but it's inevitably going to be imperfect, no matter the provider you use. There are so many bad actors out there, it's surprising that it works as well as it does.
Those have nothing to do with being spam, right? Spam is about content, those are about authenticity. Anybody can send authentic trash, or unauthenticated gold.
There's a pretty good chance this is because Shopify is sending a lot of email users mark as spam, or is using the same mail server as someone who does. Then you marking them as not spam gives them a better score but the sender's reputation is still so bad that it can't break the threshold to stay out of the spam folder.
Hilarious Gmail addresses send tonnes of spam so filtering by provider doesn't do much there days anyway. But Google insists to continue
Seems like we had the opposite problem 10ish years ago. But now the pendulum has swung a bit too far in the other direction.
Ultimately most of the spam I get these days is actually from individuals doing low volume cold outreach from personal email addresses...not companies sending bulk. The new gmail unsubscribe feature works great for marketing emails but is worthless against cold email spam -- which somehow rarely ever lands in spam.
This is a great guide that's been used and updated for many years:
https://www.purplehat.org/?page_id=1450
Once hosting email for yourself, you may want to add new project-specific domains, or host email for friends and family. The database user accounts actually make it easier to add and remove users after the system is up and running.
This Purplehat guide provides a step by step procedure that's allowed many people and orgs to bring self-hosted email online...
https://workaround.org/ispmail-bookworm/
I set up my own server more or less following the above guide, but eschewed the database in favor of plain text files. I wanted to keep things simple since I am the only user, but the above guide should scale to big enterprise setups.
A simple docker compose up can get a reasonably working setup [1]
[0]https://github.com/hyvor/relay [1]https://relay.hyvor.com/hosting/deploy-easy
If you do it yourself and do it correct it's a pleasure. I have automatic updates with automatic reboot, tailored systemd to make sure all is well and status reports per mail - total bliss, easy 2-3 years, with trixie now even 5 until you have to touch it again.
It's mature software.
Host yourself! The peace of mind and control is totally worth it.
That's what I thought of when I saw the title, too.
Where are my ...killer!jolet! people at?
I use Gmail as a free spam harvester to train my own spam filter - https://news.ycombinator.com/item?id=38843288.
But as others here have suggested greylisting is extremely helpful in this space as legitimate servers should always retry. Well only my power company is the exception and they will fall back to sending paper bills, but even Gmail falls foul for them. It's also one big reason I'm not worried about up to a week downtime. But I have two email servers, a receiving and a storage server, the receiving is cattle and I car re-deploy in minutes if needed. - https://news.ycombinator.com/item?id=38512732
On greylisting I would say using https://github.com/stevejenkins/postwhite (even if it's very old and not actively maintained) has proven very important for the annoying 2FA emails, I strongly contend that email isn't suitable for this use case but that's another conversation)
Receive only (e.g. account signup)? Use your own account. Not important anyway? Use your own account. Your recipient needs the email more than you? Use your own account. None of the above? Use a mainstream provider for that email only.
Not to mention I use a separate email per service so I can tell if there's a data breach (usually end up seeing an uptick in spam).
*Edit: Not to mention I can pay for a relay service other than Amazon SES without using Google's services, and for significantly less.
When replying reply from your self hosted server.
That way you can gradually shift over.
I had been self hosting like this for years.
But it’s not receiving that is the problem, that is generally fine, if ports are open at ISP / network level. It is the sending that is often tricky. Sending email on the other hand can be done from multiple servers (if SPF correctly configured) And nothing prevents you from sending email directly from your own relay. You could try that, and reception would not be affected.
I tried this over a period of years, aggressively changing my email server configuration as new challenges appeared, before realizing the basic problems were (a) a server's configuration is a moving target that requires constant revision, and (b) if your ISP has ever hosted a spammer, even briefly and inadvertently, then its entire address block may be universally blacklisted and you have to change ISPs, possibly several times.
So ... I gave up. If I had nothing better to do, if I just wanted to play email server whack-a-mole, that would be different, but I have a life apart from pleading with giant email recipients to trust my little server.
It's not as though Google, Microsoft, et al. have an incentive to trust small email servers -- quite the opposite. They can -- and do -- make the argument that they shouldn't trust anything but another big player like themselves.
- More of anti-UCE, with postscreen (greylisting, DNSBL and DNSWL checks), policyd-spf, body_checks, check_sender_access, check_client_access, postscreen_access_list.
- Setting "home_mailbox = Maildir/", to keep mail in user directories and in the Maildir format (which seems to be less prone to corruption than mbox is, and well-supported by MUAs).
- Leaving TLS defaults, except for the paths. I used to set mandatory TLS, but then ran into some servers not using it, and figured that I do not trust the involved servers more than channels between them anyway (especially the servers that do not support TLS). Being overly strict with allowed protocol versions (or even ciphers) also reduces compatibility, while for encryption it is better to rely on OpenPGP.
- I do set Dovecot (for both IMAP and SMTP submission); the recent configuration change did not seem like a big deal to me, and it was documented, so I found it easy to update. It is nice to be able to use email from a server (and that ability does not go away with Dovecot), but a local MUA also has its advantages.
- Registered at dnswl.org, to improve deliverability in some cases.
For 1984, I'd have expected UUCP and bang paths to peer mail hosts. Instead the article starts off by setting up DKIM, from over 2 decades later!
Keeping FreeBSD up to date is extremely simple. run pkg update && pkg upgrade. It rarely breaks. Can't remember it ever breaking.
The main reason I prefer FreeBSD over Linux distros are the far superior package managers(pkg for binaries and ports for source code).
I also host my own web server using nginx, and sometimes other stuff. All in separate jails.
Back when I was a kid, I used to have my own servers.
It is funded by pay-per-transgression. If you are a community member and someone receives unwanted email your reputation suffers. If you are gmail, et al you have to pay for each email sent & received.
Someone once wrote (let me know if you know the source) that users are not the customer, because they don't pay. It is advertisers who are the real email customers. This has resulted in a business model where users are prey animals. This is upside down and probably cannot be fixed without a hard fork.
I don't mean this is a good idea, or implementation. But I think it is a good direction.
"After self-hosting my email for twenty-three years I have thrown in the towel"
https://news.ycombinator.com/item?id=32715437
https://cfenollosa.com/blog/after-self-hosting-my-email-for-...
Just wanted to add that DMARC isn't really about the DKIM signature itself; it's about whether the domain in RFC5322.From aligns with either SPF or DKIM.
So if SPF is aligned, DMARC will pass even if DKIM fails.
Could you say more about what your friend did differently than you and what makes it so difficult? Are/were they self hosting but don't remember how it all works?
I needed some minimal mail delivery for user registration confirmation and password recovery, and I finally caved and just use some free service. It's okay since those emails are really, really, sparse in my case. But it sucks that email, this one old and open technology, is not realistically self-hostable.
Maybe there's a sleep() command in there so that it takes six days to send an e-mail from upstate New York to Sweden?
Because I can tell you that's how long it took in 1984.
I had a small blacklisting issue in the beginning that could be resolved within 24 hours, since then it works flawlessly.
If you host it at home, can you endure uptime?
You have a trust relationship with your VPS provider. Yes, they can access it, if they want to. The difference is that with a VPS you have contractual privacy and with e.g. Gmail they outright tell you that they scan your emails. So it is a big difference.
With everything, it's question of budget and what risks you accept.
Also, people in the FOSS community tend to be wary of "open source" projects primarily developed by a commercial company under dual licensing.
My hope is that Stalwart will get to their webmail project over the next couple of years. I am on SnappyMail, but basically need to use desktop and mobile clients to get the full mail/CardDAV/CalDAV experience, which admittedly is already pretty awesome.
Currently, I have Stalwart, PostgreSQL, SnappyMail, and Caddy, all inside a docker-compose file, and I have migrated, moved servers, and all that no problem.
Stalwart is perfect for small self-hosters: a single binary, a single-directory resilient datastore (by default), a UI for every setting, and defaults that guide you to a DNS config which maximizes your sender score. Plus support for all of the "power user" features such as ManageSieve and shared CalDav folders.
Honestly, I love hosting my email now. And the last remaining battery which could possibly be included is now WIP: webmail!
Unix philosophy need not apply when there is exactly one use case for integrating these tools. (Or at least, one case which covers 99% of users. The remainder can keep their managerie of arcane config formats and susceptibility to unsafe language CVEs.)
> Notes from setting up a truly minimal self-hosted email server with Postfix and OpenDKIM. Covers TLS, SPF, DKIM, DMARC, DNS, reverse DNS, multiple domains, and delivering to Gmail.
great article and discussion nevertheless, awesome to see people still doing it
1. receving mail
2. sending mail
Only #2 became difficult
Internet subscribers receive lots of email to which they never reply
Sometimes "throwaway" disposable email addresses are useful^1
Various third parties offer this as a "service", i.e., #1 is disconnected from #2
Self-hosting #1 can provide an alternative to using third parties
Generally, the only cost is a domain name registration
1. Also HN commenters have complained in the past that email sent via self-hosted SMTP to certain recipients, e.g., Gmail recpients, may end up stored on certain undesirable third party servers. This is because the recpient uses a third party for both #1 and #2, a so-called "email provider"
But this is not the only purpose for which these disposble addresses may be used
