It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.
I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."
It's simply not true we have no incentive to fix this. Here are a few:
Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem. Gmail does not, inherently, have any network effects (unlike FB, messengers, any other comms tool of Gmail's scale). Email itself, of course has a huge network effect, and that is because you can email anyone in the world, regardless of what email system they use. It's because email is open. If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on.
Secondly, we care deeply about having positive relationships with developers and all our users. I can tell you it definitely makes me sad to see articles like this. There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.
I agree Postmaster tools has been underinvested in and we could do much better there.
SPF, DKIM, reverse DNS, no blacklists, no open relay, longtime ownership of IPs, etc etc. Using various mail testers returns a 10/10 deliverability score.
And yet, messages sent to Gmail always go into the spam folder, or are never delivered at all. These are everyday regular messages, I have never used mailing lists or sent bulk automated messages.
The issue is, there is no recourse, no fix, no acknowledgement of the problem with false positives. There is no tool available to me to understand or correct the "problem". Hint: this comes across as Gmail not giving a shit.
Gmail has a responsibility to be more accountable, even if these problems are unintentional, because Gmail is such an enormous node in a federated network.
> If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on.
Correct. Gmail is contributing to the erosion of email reliability. Please course correct.
I just happen to have set up an email server and encountered the same problems with Google as described in the article. I own the IP since quite some time, it is not on any black list, reverse DNS is set up etc. but Google rejects email as spam.
And this even happens when the gmail account has added the sender in his address book and has send the first email to which I replied - thus there is a message id that should already be known on Gmail's side.
Use your AI to put email into the spam folder. Refusing it outright is a case for the European Commission which hopefully will slap you another few billions of fine onto the wrist until you remember to play nicely with the other kids.
There is no excuse to refuse SPF, DKIM, reverse DNS, proper MX, no blacklist, sender in recipient address book and reply-to msgId email.
Oh really? Then what is the mechanism by which an affected user can report an issue like this and receive individual, accountable response? Because that is what a company that cares about fixing issues does.
As far as I can tell, the only redress available for any issue like this with Google is 1) be a big enough name like Jamie Zawinksi (he has been posting about struggles with this exact issue for several months) that your blog post gets attention, or 2) hope your blog post makes the front page on a site like HN.
I'm inclined to think Gmail's approach to this problem is fundamentally flawed, because Gmail has been by far the worst at binning my server's emails. I don't have this problem with Hotmail, AOL, Yahoo, etc., etc. Now, feel free to argue that their approaches are too lenient, but if I have to choose between false negatives and false positives in my spam filter, I absolutely know which one I'd choose.
This is wrong. The current situation is "if you want to reliably contact gmail users you need a gmail address". This is no different than Facebook or messengers. And that may actually be the incentive for Gmail being such a bad player compared to everybody else.
It's only ever Gmail that sends mail to spam. Every test I run on that email marks it as clean (generally spam assassin)
Gmail is certainly NOT a part of the open email system you mention, but is a constant thorn in open communication.
Ironically, I use Gapps for one domain and because of my spf settings with "include:_ghs.google.com" I get every cat and his dog trying to send as users from my domain, which thankfully end up in that domain's gapps spam. (hint: let us use geo located includes, like _ghs-us.google.com or _ghs-au.google.com, so that there's a smaller include list!)
This is so deeply disingenuous and at odds with so many peoples experience it’s absurd. Or satire?
Or you've grown large enough that the base no longer matters.
we care deeply about having positive relationships with developers and all our users
it definitely makes me sad to see articles like this
There are going to be false positives, we will make mistakes
we could do much better there.
Most of this reads like it was written by Facebook's PR department after yet another scandal.
That depends on what you define as "health".
> Email itself, of course has a huge network effect, and that is because you can email anyone in the world, regardless of what email system they use
If that's the case, then why does Gmail system explicitly thwart this objective, as described by this very article?
> If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on
And that's exactly what Gmail is doing when it does what is described in the article.
> we care deeply about having positive relationships with developers and all our users
You personally might, but your company does not. What your company cares about is advertising revenue. If your company really cared about users, it would figure out a way to let them pay directly for your company's services (not just Gmail but search, maps, etc.) so you could see directly from users how valuable those services were, instead of having any benefit to users be a side effect of trying to capture their eyeballs for advertisers.
Google initially federated with XMPP, until you had critical mass and shut out the rest of the Jabber/XMPP ecosystem.
Of course now you seem intent on killing your _own_ messaging product so it's hard to divine any self-consistent intent here.
And there's the problem. The only effective way for OP to contact someone who likely has the ability to fix it was to write an article, post it on HN, get upvotes, and hope you happen upon it.
That's not a viable solution for everyone who runs a mail server Gmail falsely identifies as a source of spam. You don't have a good way to hear about issues like this.
Can you bring back the old web UI?
I experience obnoxious interaction and interface bugs in the new one on an hourly basis, it has horrendous interaction latency for many operations that used to be snappy, it looks worse, and has basically nothing new I want.
Actually, can you bring back the Gmail of like 10 years ago? Basically every change since then has been negative for me.
That's ... unconvincing to the point where it sounds like PR spin. How does Gmail not benefit from crappy email service outside of gmail? How do more gmail users not accentuate this?
>Secondly, we care deeply about having positive relationships with developers and all our users.
Frankly, talk is cheap.
I don't mean to be rude, but your response is either (a) completely missing the point or (b) disingenuous. Assuming the former, can you substantiate your claims?
I have a contra story. One of no delivery issues to Gmail or Hotmail, none, zero, nada. I've run a private email server (friends/family/small business/private mailing lists) for two decades. It's kept pace with every possible factor for reliable delivery - SPF/DKIM/DMARC/ARC, valid client SSL, IPv6, correct PTRs, DNSSEC etc and have no sketchy affiliates. In that time the IPv4 address changed exactly once and has never been RBL'd. Our mail gets delivered AOK.
And yet, even though I think my compliance level is good, I still feel like the blind man groping an elephant. I'm hoping I'm perceiving things correctly; I have no idea if I'm missing something. It helps that I'm an old-school ISP engineering inmate and contributor to well-known MTAs and MDAs, but few folks are lucky enough to have exposure to so SMTP radiation.
My take on the Postmaster Tools is that they've been created entirely to serve Google's purposes, and thereby serve no-one well because (as you point out) it's ecosystem engagement that makes a difference. If you sincerely have an incentive to improve, there is an awful lot of work to do there. It's okay to push the burden of compliance back to the sender, but the Postmaster tools offer only the most rudimentary levers to pull and provide almost no useful information, particularly for smaller/indy senders.
The message that comes through is that Google only really gives a shit about other large scale entities and struggles to see other points of view. This stands in quite stark contrast to Google's effort level over HTTP certificates and webmaster tools.
"Gmail does not, inherently, have any network effects" - you do and you can't help it.
I've been doing email for quite a while (20 odd years) and I don't find Gmail refusing my customer's email for silly reasons too often.
I have had some odd rejections from Gmail (int al) and no-one to talk to. You opine that Postmaster tools are under-invested in but actually miss the real point:
You (G) seem to under-invest in people and put too much faith in magic (AI/ML/nonsense). I really am not a Luddite (I'm giving the tyres on my smart new Node-RED home IoT thingie a good talking to via Javascript right now) but please don't forget ... "memento homo")
I thought it's great that I have so rarely stuff in my Gmail spam folder that I don't even have to check it regularly until I nearly missed a very important email. Apparently I was lucky that ot didn't get outright rejected.
I rather have to deal with some spam than to miss important, or even regular, emails.
I get why gmail isn't transparent about all their spam-filtering mechanisms. But when personal email to gmail users is being regularly classified as spam, that produces a serious usability problem for non-gmail users.
It absolutely does not. It is so large now that what the parent poster said is the real case. It has such dominance that people are pressured to switch to Gmail or Google apps to get stuff reliably delivered to the massive percentage of email users.
This is exactly like an IE6 PM coming on extolling the virtues of open standards and how they are critical to the success of internet explorer.
Seriously think about it (assuming you are even posting in good faith). If Gmail's success was "entirely predicated on the health", do you really think Postmaster tools and other interoperability efforts would have so little investment?
That simply is not relevant anymore if 99% of email is on Gmail.
I want to echo the exact same issues as the original author. Running my own email servers since the 90s I recently just “gave up” - no single indication about spam issues. All green on several tests. But Google rejects mail. Zero help or tools from google to get off their filter or whatever it is that makes these kind of decisions.
Friends and family all have Gmail accounts for those important emails that Must go through. Enough said.
I’ve moved my email to another provider and stopped self hosting.
It's sheer chance that you happened upon this HN post. Google needs to have an open, healthy customer service infrastructure in place to hear about, track, resolve, and follow up on reported issues. This is antithetical to "The Google Way". That is how you get posts like this.
https://github.com/dominictarr/your-web-app-is-bloated
Gmail used to be a small, light, fast and simple email service. Frankly, you should be ashamed of what Gmail has become.
To sum it up: sorry, but what you're saying here doesn't sound like it's true, and if it is let me tell you that you're failing at it, miserably. The only reason I didn't move away from gmail is that it's just a lot of work. But the time will come, eventually.
That's the box I use for my personal email. Out of principle. I believe in a distributed/federated net. It just means that I can't expect that emails I send actually get delivered. Sigh.
My experience for business has been that you simply have to bite the bullet and be on outlook.com or gmail.com. Or you can't expect that regular email always arrives. What everyone in the corporate world expects of course.
I've heard this story from everyone I know that operates their own mailserver. No hyperbole. I would be tremendously happy if Google would spend some serious (and visible) effort on this. Thanks Paul.
That's not true at all. I've noticed when sending from Gmail to Gmail it always gets delivered and quickly. I've noticed with other email service people, even ones using big providers like Yahoo, it's really hit or miss if they can receive from me, send to me, and it takes a long time to show up in my email. So there's a really strong network effect to use Gmail because Gmail doesn't work well outside Gmail. So much so that I keep a bunch of address (personal/Gmail, work, school, my personal web site and email server) and try to pick the one that works best based on who I'm sending to.
Really, it's not about incentives; it's about perception. Just like you can't act at Google's scale on your own, you can't perceive the effects of Google's actions on your own. You care about fixing issues when you hear about them, but how do you hear about them? What can the Google-animal see, and what is it blind to?
Why not count the amount of emails coming from those domains and give them a "softer" filtering algo if it's below a certain threshold? Or open a whitelisting program where you can go full Google on people who violate the terms? I think there are (automatable) ways to solve this problems, but only if Google understands that emails are for human interaction, not to receive ads.
(Edited for typos)
There needs to be a way to inspect what caused mail delivery so we are able to fix it. There needs to be a way to provide feedback to Google about mail that was wrongly rejected. If you can push this forward internally, please do it.
This is a far too regular occurrence to just ignore: it is a rule that delivering email to gmail from outside is unreliable, not an exception.
https://www.zdnet.com/article/former-mozilla-exec-google-has...
It is kind of funny to me that other email vendors could implement SMTP and co properly and Gmail _more recently_ started to have issues. Just because of this I started to look for alternatives because I will not stand and watch how an ad-tech company destroys the open internet, even if this will be very inconvenient personally.
They have literally shown what they have tried to do and it is not enough.
If this poor guy is a "false positive" then what is his path to resolution? It looks like he doesn't have one and saying "Yeah, that sucks" or "Because you made it on hacker news we will investigate" isn't helping all the other "small fry" out there.
But the problem with all Google services is that there's apparently nobody listening. Unless you're a paying customer, and even then, I gather that it's iffy. It's all automated, in extremely unhelpful ways.
I do appreciate, however, that it's largely a scale issue. Google is just so big, and operates on such small margins, that it's arguably not economically feasible to provide individualized support.
Unless you make it to the HN front page, or whatever.
As a practical matter, if you want to send email to Gmail accounts, you'd better be using Gmail. And apparently, even that doesn't always work.
How extensively does the Gmail team test interoperability with other mail systems? From my personal experience, one sin committed by many teams who in say they play nice with the ecosystem is to neglect to test integration against other players in the ecosystem. It's annoying to test and slows down development pace, so frequently no developer wants to do it. This leads to a situation where it may be unintentional, but there is definitely a "favored client."
This week I encountered the following situation the first time: 1. I received an email from a Google apps (or whatever it is called) user. My response to that email landed in the spam folder. 2. I exchanged one or two mails more with another person in the same organisation without problems. 3. Suddenly my response got rejected. I tried sending via various means such as via Thunderbird, via mobile on 4G, via mobile on Wi-Fi, changing the content slightly... No chance. My email got rejected. In the I ended up calling the person from number 1 on his mobile to get the second person's mobile number... Needless to say, I have been less than pleased.
Person 1 was very surprised about their spam filter misbehaving, but at least I can now offer an excuse by pointing to this blog post...
What then should we make of the fact that Google practically ensures that they won't hear about such issues by making it next to impossible to report them?
However, as one of the PMs in charge of one of the most important messaging applications on the planet, you cannot claim the following without stating your reasons:
> Gmail does not, inherently, have any network effects
Yes, compared to facebook, and within the context of messaging apps, gmail's network effect is weaker. But the fact that the very existence of the gmail monolith can kill off small mail servers (not implicitly, but by failure to deliver messages), is a real problem.
I understand it's a difficult problem. It's not a gmail scale problem, but the effects, times the number of small mail servers is important, because those small mail servers will be what is left if any of the big players in this ecosystem collapse.
Don't you see? This post on HN means you have created the CLOSED unhealthy ecosystem. You built what you just spoke against when you shut out email from small providers. Shame on you for speaking idealically and not confronting the reality of the situation.
I am not taking a stab at you personally. I truly believe that you care when you read the story here. But if you really cared enough there would be no story to be reported here.
It's a purely economic and automated solution that should work well enough for small operators (collateral for a small email volume should be small since spammers really need high volume)
Content filters should never be applied to abuse addresses, yet Google happily ignores this and never even replies to abuse complaints sent to @gmail.com or @google.com.
If you reject most non-Gmail messages it does!
In an ideal world, yes. Unless Gmail makes it harder for anyone using a different provider to communicate with Gmail users. In which case you've effectively "encouraged"/forced everyone else to use Gmail to take advantage of the Gmail network. This seems to be exactly what's happened to the author. This is very similar to the complaint voiced recently by the Mozilla leader as well.
I don't doubt that you as an individual have very good intentions. But individual intentions do not translate well to organizational priorities. The best way to prevent Gmail-lock-in in the long term is to have more diversity and competition in this marketplace.
we will make mistakes
What I don't understand: why aren't these false positives sent to the recipient's Spam folder? Why else does that folder exist?
It is also an excellent goal to use AI to scale as many processes as practical to provide ever-broader services.
Yet, it is obvious that, at the corporate level and despite being one of the most massively well-funded companies worldwide, Google is famous for abysmal customer service and most particularly, providing NO way whatsoever for a person who has been damaged by a "false positive" to every correct a real person at Google to get a resolution.
Yes, providing an avenue to real people who could resolve the issue would be expensive. I seriously doubt that it would be so expensive as to make any serious dent in Aplhabet's profitability or stock price (unless, of course, the problems are far more massive than anyone knows).
Moreover, providing the ability for people who have been "false positive" damaged by some spam, account violation, etc. to reach a person for resolution to a problem would also provide large amounts of the fine-grained detailed data that would allow Google to fix its issues at the algorithm level, thereby reducing the need for the support service as well as the frustration.
Why is this not done? Is it that the problems are actually massive but hidden? Is it that management does not care the way you do? Is it that mgt actually wants to drive off all small services in order to dominate? Obviously all speculation, including bad speculation. But leaving zero ways to contact, as well as zero information about the issues, leave the field ripe for all the speculation. This is the sort of reputational problem that can fester for years, seen only as a minor issue -- until it grows so massive and passes a tipping point, where the reputation and market position is unrecoverable. I hope Google does not go that route.
Guess I’ll find out whether correcting it a second time applies to the actual newsletter messages.
I bet you also take the security of your system seriously. What other Internet platitudes can you spout?
Holy shit, GMail's going away? This is the kind of thing we're told just before a Google product is killed.
That's the point.
It doesn't have to be conscious dereliction, it can just be 'big, dumb bureaucracy'. Interestingly, operational ineffectiveness can happen within well run organizations, particularly if there's no poignant reasons to change.
These kinds of things tend to happen when one part of the organization is mining Oil or Gold and throwing vast surpluses around the rest of the camp.
You have $100 Billion dollars to fix the problem - a hoard of the best and most highly paid talent on planet earth.
So fix it.
https://penguindreams.org/blog/how-google-and-microsoft-made...
I have SPF, DKIM, reverse DNS, DMARC .. all the things, and yet I still get e-mail dropped.
Will you guys fix Postmaster tools for small servers? What can I do to send e-mail to my friends?
I don't mind spending a penny to send an email, I don't send enough to make any difference. But to spammers, it makes abusing the email system unprofitable. And if I receive half the revenue from people sending me email, it'll likely work out to costing me about nothing.
As far as it relates to email, this is why platforms like Sendgrid got so popular, they manage the relationship with Google and others for you.
I do agree that Google's outsize influence in so many markets is concerning.
Also, Google email deliverability is good, inbound and outbound. I’ve been running an email service and have had no problems with them, unlike Hotmail/Outlook.
Source: Led a hiring committee this year. Interview invitations and job offers went straight to the junk mail folder. Also serve as Director of Graduate Studies. Admissions and funding offers, as well as general inquiries about availability, all went straight to the junk mail folder. We're not going to change our email service.
While’s gmail’s filtering is obviously neither your responsibility nor under your control, you also can’t really fault people for using one of the most popular email services on the planet.
I would hope that your department follows up some other way (you surely have a phone number!). Searches are crazy time consuming and expensive, and grad student recruitment isn’t exactly free either.
(For me gmail is inferior regardless as my privacy is violated)
Maybe this is just my personal organizational problem, but it was like a waking nightmare when I really started to look with fresh eyes at how unstructured my email management strategy had become with the tools gmail gives you. It took me weeks of nights and weekends to sort this all out (finding messages I actually want to save, identifying senders I actually want manually sorted to other folders, unsubscribing from lists I've been ignoring for years because they were conveniently out of site in "promotions") to a point where I felt ready to migrate to a more traditional mailbox setup. I can easily imagine many people seeing how daunting this task is and just deciding to stick with gmail.
Clearly they keep the hit in the search results for plausible deniability, but they're trying to influence people by suppressing speech. Given that Google is becoming synonymous with the internet, like AOL, this is a very concerning trend.
This has been a well known occurrence. I’m think about all those pictures of “Hillary Clinton is” [0] typed into each search engine and Google was polar opposite of all others.
I think one needs to be pretty obtuse to not see Google is “curating” results to fit an ideology.
Everyone should be mad at that even if they believe in the same things.
EDIT: Well... apparently some people don't believe me, so I just re-tested. "Hillary Clinton i" in four engines.
I couldn't use "Hillary Clinton is" because "for some reason" Google guessing completely stops working when you type "is" after her name - can someone find me another phrase besides "Hillary Clinton" that "breaks" the search fill when "is" gets added? No? Well.. That sure seems like Google is "curating" then.
Whatever lead to Microsoft being sued for pushing Internet Explorer in Europe and whatnot? Shouldn’t that kick in already for at least some of the recent Google bullshit?
There's definitely for GMail to become the new "Windows", with everyone else fighting to stay connected.
The administrators of the account decided to outsource the smtp relay to google. It is ridiculously unreliable, and regularly sends “no such address” responses to my synology NAS (which emails me periodically).
Similarly, when I used gmail’s IMAP gateway for work, it was regularly down.
I’m pretty sure Google will succeed at killing email. I hope gmail also goes down in flames if they succeed.
I hate to say it, I really do, because I used to respect Google and liked their products, but if you imagine combining the perception of "AOL is the internet" with the way Microsoft acted up to the mid-2000's and I think you have Google.
It terrifies me just how deeply Google has entrenched itself in the K-12 education market. At least when Apple was everywhere, we still had Microsoft Office vs Appleworks/etc.
Google is now more "the internet" than AOL could have dreamed of.
I have an email with some account information that I access a couple times a year at most but will consistently need at least once a year. Just infrequently enough that I can't remember the exact contents.
I know exactly what the subject line of the email is and it's a suggested search query in Gmail but about 2 years ago Gmail just stopped being able to find it. I can navigate to the email via label or star just fine but I can no longer search for it.
Is it because it's an 8 year old email? idk what the reasoning is but Gmail as deemed it irrelevant and stopped returning it as a search result. This isn't a platform issue either, it happens in App, or browser on multiple devices.
Google sells mail+spam filtering+more as a product, with a seven-digit number of paying customers, each of which pay per month and user. What more incentive could they have?
How did you even get your report to be read by an engineer at Google? This seems impossible unless you have some kind of business connection.
[1] https://www.computerworld.com/article/3389882/former-mozilla...
You’re right.
And it’s a small thing, but I’ve really been trying to correct myself when I use google as a verb. I’ve been trying to say “search it” instead.
You mean not to try to actively sabotage the internet that we once knew?
I worked on a team with a large (real double opt in, not spam) email list. We had to pay for a bunch of AOL accounts to figure out the trigger points, so that we could get our email updates out overnight without tripping AOL's spammer logic.
And there was a time when they had dominant email share like Google has now.
It has become very difficult / unwise to host your own SMTP server, though, but that's been true for ages now. Gmail's marketshare is going to make that veeeery obvious, but it really has been a bad idea for the better part of a decade, at least.
I use fastmail. They have humans that respond quickly when I have a problem (twice in 10 years)
That's not been my experience at all.
Why is this no longer true?
IMO, other competitors came along offering a better service/experience and AOL just couldn't compete. At the time, you could not know what this better product looked like. I believe the same will happen with the internet giants of today.
Even worse, they have incentive to create these problems.
Gsuite is extremely profitable for them, I am sure. Charging $5/user/month for hosted email must make them a pretty penny.
I would guess that no gsuite sent emails are ever blocked with this suspicious message 550 error.
How can anyone be sure of this? This is only one of Google’s practices that seems to follow a pervasive pattern of eroding open internet standards while presenting Google’s own proprietary implementation as somehow superior. Eventually, the open standard loses all meaning because the most popular implementation does not actually adhere to it. Meanwhile, Google reaps enormous benefits in the form of additional signals for its advertising business. How can this not be grounded in malicious intent?
> and that there are some very smart people working on spam prevention at Google.
There are some very smart people working on advertising at Google. The rush to forget the primary nature of Google - it’s an adtech firm - is why they’ve been allowed to skate for so long. Gmail’s spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting: after all, when viewed from Google’s own perspective certain e-mail messages contain no information which can be used for ad targeting, so they must be spam. The user’s interests are clearly secondary to this.
So, back to the “smart people” working on this: at what point do we begin judging engineers for working at Google? There’s a lot of highly vitriolic criticism that emanates from Google’s workforce on a variety of subjects, but how many of them would actually pull the pin and leave their employer? I don’t have any statistics to offer, but it seems to me that we still have a ways to go before Google has become completely drained of engineering mindshare.
There is a lot of evil done in the world that comes from this simple and straightforward principle. You don't have to have an evil plan to end up doing evil.
Personally I see this as just more Google incompetence. Gmail has just become bad. If you want to get all your mail you have to use something else.
> How can anyone be sure of this?
I know this would sound crazy to an alien learning English but the phrase, "I'm sure there is no malicious intent" in this context actually means "I don't presume there is malicious intent".
It does not mean, "I have a positive reason to believe there is no malicious intent."
Does anyone have suggestions on how to go about this? My first thought was to use a password breach like Collection #1 to get a distribution of domains used, then query each domain to see which provider they are using or if they are self hosted. But I would certainly appreciate other suggestions!
It can't be just that. Spam filtering makes email in general much more usable, and Google of course wants Gmail to be an attractive product.
hotmail-com.olc.protection.outlook.com[104.47.1.33] said: 550 5.7.1
Unfortunately, messages from [<redacted>] weren't sent. Please contact
your Internet service provider since part of their network is on our block
list (S3150). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)
Reporting-MTA: dns; <redacted>
X-Postfix-Queue-ID: 14A41FEB66
X-Postfix-Sender: rfc822; <redacted>
Arrival-Date: Thu, 14 Mar 2019 14:07:42 +0200 (CEST)Edit: not long ago I used a gmail account to contact microsoft support to get them to remove me from their blacklist again (couldn't use my normal email for obvious reasons). Ironically enough their reply got marked as spam by gmail (something in my included message parts might very well have triggered that but I had to laugh out loud at the situation).
I set up DKIM, SPF, and reverse-DNS records and resented every moment of it. Even after all that, there's some chance that an email from my server will be marked as insecure/spam or otherwise just not be delivered because Google has come up with some new brilliant mail security/auth/permission scheme that the world has to adopt tomorrow or be cut off from all Gmail users.
At one point I heard a rumor that no human had worked on Google Finance in over a year, simply because nobody was interested. Not their biggest service of course, but still a major site that presumably generates a lot of revenue. If they think that global finance is "too boring" for their amazing engineers then support for everything is liable to be canceled at any moment.
Don't I know it. This happened to us but not with Google. We use Exchange Online (Office365) and have DKIM, SPF headers configured. At some point our emails stopped being received by bell and cox emails addresses ... sporadically but sometimes for days at a time. It turned out that the spam filter provider (used by bell and maybe cox) correlated one innocuous phrase in our confidentiality notice (which is appended to all outgoing email signatures) with spam and therefore marked the entire email as spam as well.
I'd rather turn on encryption/signing/security for my mail than contribute to a system like the telephone network where bank accounts get emptied by transoceanic criminal networks on the regular.
Yep, I know the author's frustration very well. I made a previous comment[0] trying to warn others of personal email servers' outgoing email being spam-holed -- and yet some of the replies still argued I was overstating the difficulties.
Everybody's risk tolerance is different. Personally, I just don't have the bandwidth to administer my own private email server and constantly worry if recipients are receiving my emails.
Remember for a couple of years, every time you'd send a message to someone at Earthlink, you'd get an automated rely demanding that you verify yourself before the message would go through?
Now I can't remember the last time I saw an @earthlink.* e-mail address.
Google has apparently learned from that and put the "error" in the 550 messages, where they can't be seen by the end user, and lead to non-helpful resolutions for sysadmins.
The result is that the blame for missing messages goes to the sender, not to the recipient's email service.
If Gmail at least notified the sender that there was a problem, then a pattern of responsibility could be established. But this is just another dark pattern.
Google has definitely been changing things, everyone presumes it’s some rogue ML system. My company had solid Gmail delivery for almost 10 years, and at a fairly large volume. Now we can’t get it delivered any more. Very close to showing users a message not to use Gmail.
1) The RFCs mandate a server to accept multiple recipients--100 minimum per RFC 5321 (inherited from RFC 2821 and RFC 821).
2) A server can only accept or reject a message for all recipients.
Theoretically a global rule shared across all recipients could be implemented in a way that rejects at the transaction level. But because you also have to implementing lazy, per-user filtering (or more importantly, per-user whitelisting), in practice the architectural and engineering focus is on filtering after accepting the message and closing the transaction.
Without the ability to provide immediate and accurate failure responses upstream, filtering issues are left to metastasize and fester. (Because of spamming and spoofing mitigations bounces are next to worthless in terms of proper integration into the end-to-end software stack.)
One of two things needs to happen with SMTP.
1) RFCs drop the requirement for multiple recipients per transaction.
2) An extension mechanism is added that permits per-recipient transaction responses.
I think the only practical option is #1. Both cases require some amount of infrastructure patching, but for #1 it's extremely minimal and in most cases none at all. #2 is infeasible, at least without #1 preceding it (i.e. once you can no longer rely on batch sending as part of the basic SMTP command set, you'll have to support the extension.)
Once that's accomplished vendors who don't implement inline filtering can be rightly criticized and shamed. As inline rejections become more prevalent, mail being effectively black-holed will become a thing of the past and we'll finally begin to see proper back-pressure (literal and figurative) return to the e-mail network. There'll be more pressure on large vendors like GMail to improve their systems and policies once the fog of complexity and plausible deniability lifts.
A couple of things that regularly seems to trigger false positives in spam algorithms:
- no or misconfigured SPF and/or DKIM
- no or misconfigured reverse-DNS
- automatically included footer texts (confidentiality, copyright, safe a tree don't print, etc)
- regular automatic replies from the domain (such as out of office notifications)
- the use of embedded images (logos, human signatures, etc)
We sometimes joke that these triggers were built in by the algorithm developers as a means of punishing those who litter their email with pointless texts and images.
So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way. And all is fine until you start receiving random hard bounces with no real debugable answer for Google.
It got me deeply sad and questioning my decisions: since you can’t really ignore Gmail, email isn’t in practice “open” anymore. So I might as well sign up for Facebook, WhatsApp and the likes. It’s been years and I haven’t yet, but it’s getting harder and harder.
It's like there's an assumption that gmail is perfect, and the problem is with the sender. Even if that was true, a normal mail hosting company would at least tell its customer why the mail is not being delivered, so that the customer can tell the sender what to fix.
The gmail recipient is never exposed to this side of google. So they don't know what a nightmare comapny it is to communicate with.
Why should everyone and their dog be solving gmail users's problems with receiving messages? It's such a demented system. It should be the other way round. Recipients, via their provider should be solving their issues with spam filtering and blocking.
If the gmail user would be blocked by my mail server, I would not tell them to go guess what's wrong, fix gmail, and to have fun. It should not be acceptable the other way round either.
I started to host emails to many friends, small businesses and even a SaaS I developed. The subscription needs an email validation and I'm aware that the activation email ends up in the Spam folder for the new customers using Google emails. This activation email has everything from dkim, spf, dmarc, to unsubscribe link, full physical address of the business, etc and still I can't hit a good enough score.
I was thinking to start using Google service to send the activation link and hosting my personal domains, but seeing that I am not alone, I will continue to improve my little email projects.
Thanks all for cheering me up on this. I'm sure we can come up with a solution and I would be happy to help. When do we start?
Hilariously, Google will flag these emails sent to myself using my own credentials and their infrastructure as spam. I have no faith in them ever getting this right.
https://toolbox.googleapps.com/apps/checkmx/check?domain=tab...
These checkers for example don't find any errors with the DNS record:
https://dmarcian.com/dkim-inspector/?domain=tablix.org&selec...
https://mxtoolbox.com/SuperTool.aspx?action=dkim%3atablix.or...
One thing that this second one does find is the lack of a version number in the record. I see that the RFC [1] got updated since the last time I checked and having that is now recommended instead of optional. I'll add that. Thanks!
EDIT: after adding version record, Google's tool now shows green checkmark for DKIM as well.
In any case, it's impossible to test a DKIM setup just by looking at the DNS records. The true test is to verify the actual signature in the message. Last time I checked, the mail on the receiving end (Gmail), did have "dkim=pass" in the Authentication-Results headers.
I don't send much mail to gmail, though. Sans that, my only issue has been a mail server that uses Reverse DNS, which I don't have set up, and entirely ignores my email without it.
I suppose I can understand this if some people get a great deal of spam, but requiring so much of this on an unencrypted message seems more like useless reassurances than anything. I'm not criticizing email for being unencrypted, but this seems more like another hoop to jump through than anything.
Also of note, almost all of the spam I receive is from gmail addresses and I wouldn't be surprised if the invalid addresses that send messages demanding bitcoin are also from gmail, but with fake From fields.
I'm in a similar boat; been running email servers since before GMail existed. My personal one I've been running out of a home server closet since 2001. I've also done everything I can to guarantee I'm not running an open relay and not sending email unsolicited. Have been mostly lucky so far, but occasionally I will have people on mailing lists I manage (people I have met IRL and put them on the list to organize group meetings IRL) not get email. Used to be other stupid mail providers (AOL comes to mind), but these days it appears to be Google, sometimes.
I've had this domain nearly twenty years and run email on it for that same amount of time. I'm not going to "just switch", especially to a privacy invading ad-spewing "alternative" that doesn't give me as much control. Fix your damn servers, Google.
I can't help but think that we are seeing the google transition to late 90s microsoft now.
I’m not sure what else people want from a secure email service you don’t have to pay for. Also, any work around the 2FA by a human simply means less security for everyone.
After this episode I made damn sure I had recovery codes stored in a safe place.
You could literally have two long standing, legitimately used accounts send an email to each other containing a link to a URL like http://0xANYTHINGHERE.com/ and it would be insta-spammed. I suspect it was a hard coded rule to avoid people using "long IP" URLs to circumvent other filters.. except there are lots of legitimate 0x domains that aren't long IPs.
It was fixed sometime in the past year but I got a lot of use out of it in talks I've given about email deliverability over the years.
Without these customisations that forum would be overrun with all sorts of spam.
However, these customisations only stop spam postings but can't stop actual registrations.
Based on the users that I see who are registering I see a great majority of these spammers love using Gmail accounts.
So while it is good that Google Gmail is trying to fix these spam issues, from where I stand Gmail users seem to be a big part of the spamming problem.
Spammers love Gmail only because they can easily create spamming e-mail accounts.
Of course, email forwarding turns out to suck, but we're just going to suck it up and move to G Suite for organizational email addresses and let folks forward from there. E-lists, OTOH, I haven't found a good integration to automate membership in the organization vs. G-Suite; perhaps it's time to just move to a forum.
https://www.google.com/nonprofits/offerings/apps-for-nonprof...
The amount of fighting you have to do to stay on everyone's whitelists is absurd.
Google seems to be suffocating the internet bit by bit on all fronts and it needs to be stopped.
Every new recipient I email, if I don't hear from them within 2 days, I have to contact them out of band to ask them to check their spam folder. The problem is usually Gmail's heavy filtering.
At work we use sendgrid because of this. Have to trust a centralised third party to send out api keys. It's frustrating.
Hey, I remember you were one of the replies[1] in my previous thread that said I was exaggerating the "send emails" issue. Maybe your difficulty with Gmail's mystery filtering algorithm will warn others that depending on personal email servers for reliable outgoing email is a non-trivial endeavor.
Perhaps personal email servers are not impossible to set up but they're also not as easy to debug as some make it out to be.
This is similar to the +whatevertag trick that gmail pioneered for tagging, however that can be removed by malicious parties (spammers) via a simple regex. So Google have almost all of the infrastructure but should just add a bit more to get the rest of the way there.
What I mean in specific :
1. I want to sign up to and receive your newsletter (you[re Ted) but I don't trust you yet. so I should navigate to gmail.com, click something like "generate another inbox", leave it set it to "For now deliver this mail to my inbox", add the description "for Ted's Possibly Spammy Newsletter", and then click "generate". It should give me inbox3943578423@gmail.com - similar to a phone number but a bit longer and personalized to one recipient - and then I should give that to the recipient to use, in this case the possibly spammy newsletter. It should always be delivered to my inbox, as I've set. Once one of the spammers sells my email address (for example I start getting advance payment scams) I'll be able to disable further spam from there by sending it to the trash but also know that Ted's newsletter is the one that got compromised or sold it. You can do this today by going through the steps of registering a new gmail address and turning on forwarding, but it takes like 10 minutes to do so. it should be like 10 seconds.
This should be possible because people always have easy access to the gmail web interface. There's no reason it can't be a bit more like a social network where you confirm it from the web interface as well.
that's my idea anyway.
e.g. https://pastebin.com/u48DAaLP
That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.
After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.
Mailman, which most open source projects use for mailing lists, have developed work arounds to address some of the issues. Unfortunately my experience is that many projects run older versions that don't have these work arounds or if running newer versions they have not been enabled. Most likely because no one has revisited the configuration since initial deployment on an older version. After all they didn't start the project to spend their time being mailing list admins.
The people having problems here are non-Gmail senders sending email to Gmail recipients.
I'm not interested in ProtonMail's encryption (and it's potentially a liability, attracting aggressive state action). I'm mainly interested in their apparent respect for the privacy of users' private communications. And also hoping that ProtonMail has a bit more reliable delivery than GMail.
In any case, rising competition lifts all performance boats, or something like that.
I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA has a fairly strict SPF configuration - any email with an invalid spf result is rejected. This can come about because a legitimate company has misconfigured their spf records (happened twice in all the years I have hosted, discussions via postmaster@ helped them configure their dns correctly), but 99.999% of the time it is a spammer. What is worse is that rejecting email for domains without any SPF records can still result in valid email being lost, in 2019.
In this specific case, I don't think Google are "being evil". They're trying to reduce spam in the email ecosystem and they're doing it by using standards they themselves adhere to (Gmail send me reports of dmarc statistics each day google domains receive email from my box).
On the other hand, I do of course support either self hosting, or using another provider so as to ensure we do not end up with a Gmail monopoly. If I did not self host, I would find another provider like (but may not) Fastmail, Posteo etc (I would have to seriously review the options, which I haven't done).
If gmail is improperly flagging a message I get as spam, I can create a filter to never send it to spam. If they're rejecting it like the OP talks about, if I want to stay with gmail I have no way to get that email.
One day a colleague and I discovered that he had not received some of my emails (intra-domain - me@example.com to him@example.com).
This is all within the confines of Google. Google had flagged some messages as spam, and by what determination I could not fathom. The content seemed perfectly typical.
I have had really pleasant experiences with G-Suite human support, at least in terms of the quality of interaction. But they could not answer why some intra-domain emails were being flagged as spam. I have suspicions that it would take a whole team of G engineers to maybe identify what bit of logic in their systems (incorrectly) marked some of the emails as spam.
It seems the beast (automation) is just almost not under their control anymore.
#1 - Does anyone send test emails and measure delivery rates? As in send yourself a bunch of emails and see what happens.
USPS and its major customers and vendors do this with physical mail. They measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006 https://www.uspsoig.gov/document/undeliverable-addressed-mai...
#2 - What is the responsibility, liability for email relays to treat everyone equally? For comparison, a US retailer has to accept US currency, but can (sometimes) turn away problematic clients. Is there anything like that for electronic exchanges, transactions?
Reminds me of jabber.ccc.de that stopped providing new accounts because they felt they were ruining a federated system.
It should be noted that ultimately these efforts to "learn" result in ML, AI whatever pointed AT you, not working for you.
Like the author, I've been running my own mailserver for over a decade and am very conscientious about ensuring that no attackers use it as a spam relay.
While the vast majority of the people I exchange email with don't use GMail at all, so it can take a while before I notice any issues with it, I did happen to notice that GMail was rejecting my outgoing email a couple of weeks ago.
This week, I finally got around to trying to address the problem (it's not high priority because having GMail reject my emails isn't really a huge deal).
...and I found that it is working again without my changing anything. Weirdness abounds.
Sending SMTP yourself (directly, without an SMTP relay service) sets you up for trouble.
Edit: And of course, I do have DKIM and SPF configured.
> I can't tell other people to go off Gmail
I disagree. There are reasons to switch off gmail. Not just Google eating mail but also for privacy reasons. Google knows all about your banking, eCommerce orders, your media subscriptions, health issues and many other dependencies.
A good alternative is protonmail. It is private, has a mobile app, is a free but you can also pay to support the service. I also consider protonmail much more secure than gmail.
Definitely Google has done something to mess up their spam filter algorithms in the last year.
[1] https://news.ycombinator.com/item?id=19536465 [2] https://news.ycombinator.com/item?id=19500357
Spam from small domains might be pretty high as a category, but of course we don't want statistical judgements about categories to outweigh the merits of the individual. Maybe Google's algorithms have been watching too much Fox News.
I use FastMail for instance and never have this issue, but I know so many people who gave up on running their own mail servers at the small enterprise level because of stuff like this I often wonder how FastMail does not have these issues but others do. Is it a headers thing I wonder?
People hosting their own servers enabled wide spread abuse due to misconfigurations. Because everyone could do it and because defaults were shit for decades, stuff like open relays were common. People defaulted to the wrong ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam would have killed mail by now if it hadn't been for major players like Google, GMX, Hotmail/Outlook/etc.
Back in the day, greylisting was commonly regarded as a best practice, leading to the impression that email is unreliable and prone to latencies.
I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy to have a spam-free inbox every day and if this is the price for that, I'm sure about 1-2 billion people are willing to pay it.
I'm quite astounded that there have been no updates to mail protocols in the last couple of years to at least mitigate the most common issues, but all I see are band-aids that are complex to setup and horrible to debug in case of issues.
I currently self-host for non-mission-critical email, use FastMail for business, and continue to use google apps for personal/mission-critical.
(tablix.org doesn't have a DMARC record)
The real problem is lack of provided reasons for blocking, so people waste a lot of time trying to figure it out by guessing and trying random shit.
especially when used as a filter in Postscreen.
In the end I just gave up and started using mailgun as a relay.
Perhaps there really would be a lot more spam without such filtering, but it points to the actual problem being elsewhere. Perhaps we need some kind of cheap and userriendly (uniform but decentralized) email court system, and fine / ban email accounts that misbehave?
What can we do about Google's email monopoly?
The frustration comes with scenarios such as the one outlined in this blog post, where small mail server operators get bullied even though they are doing everything right. I can completely understand not wanting to operate a mail server due to this situation, or not having interest in leveling-up server administration skills.
For those of us who do have expertise in running mail servers, it's a shame we have to deal with these obstacles.
The upsides, sharing many of the common features of all self-hosting, are things such as pure and total control, data privacy, ultimate flexibility of configuration (e.g., infinite and unfettered aliasing), customization of interface, high-performance, etc.
Until this happens. Which it does very often, for lots of reasons that are out of your control.
I like my emails with pesto sauce, tuna, and cheese ... wicked combination.