I'd argue mail servers (as SMTP endpoints) are more interchangeable than Heroku. They have a completely proprietary configuration and deployment process.
I can understand that competent sysadmins can recreate heroku's underlying hosting, but there's still a decent switching cost (especially if your whole team is accustomed to the heroku way). I've consulted with several companies that have 5-figure a month hosting on heroku that they can't migrate after all their investment.
If SendGrid changes my plan (or somehow degrades), it's a simple config change to send my email through another provider. A more apt comparison would be if your mail provider used a custom ReST endpoint instead of SMTP (say, as MailChimp does).
