This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery to an eventually-consistent shared store. The requirement for strict, durable message ordering is the one thing that forces hard synchronization among multiple machines doing delivery.
POP3 actually made a step this direction with the common "UIDL" extension; IMAP4 felt like going backwards on this.
While I'm on the subject, another less-common POP3 extension I liked was "XTND XMIT". This gave a simple way for the POP3 client to send mail; sadly it never really caught on in favor of just using SMTP submission. Originally though this only worked OK because SMTP was open-relay. Once spam put an end to that it became complicated to send a message when accessing your mail remotely. Then to solve that problem we had to graft an auth layer on top of the existing SMTP protocol and upgrade all of the clients and servers to support it. If only "XTND XMIT" had won in the beginning everybody would be submitting outgoing mail over that already-authenticated channel and so much work could have been avoided.
So anyway, those are my two requests for any "IMAP killer": allow the server to use unique (but non-sequential) IDs for messages and provide a way to submit outgoing messages over the same channel.
But one thing that you MUST have for sensible synchronisation is the global MODSEQ. Otherwise there's no single state you can use to know if there's new data on the server short of (as you do with POP3) fetching the entire UIDL list every time.
The way we do this with Cyrus replication is to reinject the message if we get a UID clash - so you can have delivery at both ends of a (theoretical - not yet finished) master<=>master replica pair. Upon discovering the mismatch, BOTH messages get new UIDs larger than any yet used, which brings all clients into agreement.
But anyway - our unique identifiers in JMAP can be anything the server would like. We use 'fNUMBERuNUMBER' at FastMail, which is a folder/uid pair - but a more modern server could use UUIDs that don't change for moves between folders. We could even implement it in Cyrus on top of the Digest.SHA1 field if we built a reverse mapping database and deduplicated on it.
But synchronised modseqs... I don't see a way around that. We've been discussing "modseq validity range" stuff - where a delivery logs an intent to use a modseq, and then clears the intent upon commit. Any request that includes a range past that intent needs to wait for the commit to complete or abort.
1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail.
2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing from a few thousand up to 40 million emails. This would identify all mail updated since last connection, within 4 roundtrips with a minimum of bandwidth. The tree should be incrementally updated using XOR at both client and server whenever an update is made. It's possible to use a tree with less depth for fewer roundtrips but with support for less emails or requiring more bandwidth.
3. Push sync. Server would push all creates/updates to client while client is connected.
Also, I strongly believe that per-message UUIDs should always start with the delivery time. That way as long as your servers are all reasonably synchronized you can get reasonable "sort by folder order" behavior without any sequential ID.
You're right that the modseq issue is hard, but the Merkle tree suggestion from @jorangreef can address that. Refer to my reply to him for more details.
Agreed on the messageid thing. I frankly think that IMAP is too messy to try too hard for interop with. Using a relatively similar data model and being able to work correctly with a maildir should be considered enough.
But I love this idea and hope it takes off in spite of that. It's really time to revamp the interface to mail servers to be more friendly to webmail. It'll make a huge difference to the level of choice of webmail providers/software.
A proper mail submission command would preserve parity with SMTP by keeping envelope and headers distinct. I just want to leverage the fact that I've already auth'ed the user and shouldn't have to redo that to a separate server.
Also, presumably one of the advantages to the REST+JSON design of JMAP is to make it easy for javascript clients to use it. Since they can't do direct SMTP anyway there will presumably be some REST service that they will use for submission.
It just seems that it should be part of the same standard. Most mail readers are also mail senders.
I don't understand this criticism, the criticism is usually that IMAP does not require maintaining UIDs (the server can announce UIDNOTSTICKY[0])
> While I'm on the subject, another less-common POP3 extension I liked was "XTND XMIT". This gave a simple way for the POP3 client to send mail
https://news.ycombinator.com/item?id=4826447
> IMAP has actually been extended to support that feature (not just kind of randomly, but by the IETF "Lemonade" working group), even allowing (in concert with other extensions) compositing a message from parts (such as attachments) on the server without having to download them first to the client.
> However, and I think this is actually more relevant in this context, Mark Crispin was an "opponent". (Note: the following e-mail snippet by Mark Crispin, was written years after the Lemonade Submit proposal expired, and thereby should be considered to already take into account that context.)
>> For many years, there have been various proposals to add mail sending capabilities to mail access protocols such as POP and IMAP.
>> These proposals are always strongly opposed. It is one of the "attractive nuisances" of email protocols. The value of the capability is obvious to many people, but the high cost of having it in POP or IMAP is much less obvious.
>> I am one of the opponents. For the past 25 or so years we have been in the overwhelming majority. It is quite unlikely that this concensus will change. If anything, it has become stronger in recent years.
>> […]
>> If you're not really that curious, suffice it to say that the people who design the protocols and systems understand the attraction but have excellent reasons not to do it.
http://mailman2.u.washington.edu/pipermail/imap-uw/2009-Janu...
[0] http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
Their reasons aren't excellent, they are some theoretical notion of purity which leads to things like the "MOVE" capability taking years to finally happen, and everyone implementing copy+store+expunge independently and poorly in the meanwhile.
He's dead now, so he can't tell me I'm a feckless Gen X with no clue how to do real protocols - but there's a reason why proprietary protocols like ActiveSync get traction, and it's not because you need to pay a licence fee - it's because they actually solve problems that need solving.
We will be adding sending to the JMAP spec, although support may be optional. The way it currently works at FastMail is a bit too specific to our architecture though, so we want to clean it up before adding it to the spec.
Yes, exactly. UIDs should be sticky, but they should also be allocatable by widely distributed nodes without the need for central coordination.
If your UIDs are 32-bit monotonically-increasing integers then this is impossible. If they are 128-bit random numbers you get it for free. If you prefix them with a timestamp you even get reasonable ordering.
The entire UIDNOTSTICKY problem is a result of IMAP UIDs being so restricted.
I can see why a proposal as complicated as Lemonade failed to get traction, but I find the argument pretty thin. The "attractive nuisance" was the old open-relay SMTP infrastructure. If you were writing a POP client in 1991 you wouldn't think twice about sending mail since you just had to hit a SMTP server (didn't even have to be the "right" one!) and the mail would get delivered. When this all came crashing down far more effort got put into authenticated SMTP (talk about not separating concerns!) than would have been required to just get message submission right in the first place.
IMAP4rev1 is one thing, but there are many extensions supported in modern imap servers to speed up syncing and mailing, like CATENATE/BURL/CONDSTORE/MULTIAPPEND/MOVE/etc
Granted, the IMAP protocol is pretty hairy and difficult to work with. On the other hand, there's a huge ginormous deployment of it on the client and server, and the IETF WG behind it. I doubt JMAP will replace it anytime soon unless the WG itself takes up the issue. And given the neckbeards there ;-), some of whom have been working in IMAP for decades, you face the same opposition to change as those trying to legalize drugs or gay marriage. ;-)
We're still trying to get IPv6 deployed!
Maybe. I'm curious what the effect would be if a single mobile platform provider (iOS, Android) supported a new protocol with a single main free mail provider. Specifically, Gmail and Android could possibly push major change through.
Look at SPDY, it was originally an experiment with Chrome and only a subset of Google services, and is now being deployed to other properties and iterated on rapidly in other browsers, too.
Honestly, if it's not significantly nicer to work with than IMAP, it won't see traction, and that's fine.
Mind you - MOVE took bloody long enough in IMAP, I was arguing for it for years before it finally got support behind it. There's still not much support for SUBMIT via IMAP though, which means multiple connections via different protocols to do basic emailing, with all the support fun that entails.
Anyway - worst case, we still have a better protocol for our own stuff, and that's not nothing.
Er…
> 5.5. Multiple Commands in Progress
> The client MAY send another command without waiting for the completion result response of a command, subject to ambiguity rules (see below) and flow control constraints on the underlying data stream. Similarly, a server MAY begin processing another command before processing the current command to completion, subject to ambiguity rules.
IMAP already supports batching.
Our plan is to port our web interface across to this spec and implement both a direct backed in Cyrus IMAPd and a standalone implementation which can proxy for existing IMAP servers.
But I'm not sure it's a good idea to emphasize the fact that you use JSON. Sure, it's cool right now, but I find it technically more interesting that it replaces IMAP with something that is built on top of HTTP, a somewhat RESTful protocol.
(don't even get me started on trying to write a proxy for IMAP that's aware of the connection state)
Those things called standards generally find their way through bodies like the IETF. Maybe I missed it while reading on my phone, but I don't see this submitted as an IETF draft...
JMAP isn't the be-all and end-all of mail protocols, but it goes some way to address some of the problems in IMAP. We're using it in production right now at FastMail so we already know its pretty good as a low-latency, stateless protocol.
As for the IETF (and standards bodies in general), they're useful in their place, but they're also a slow-moving beast and thev're published a great many "standards" that have either never been implemented or are just rubbish.
But there's nobody forcing you to implement it. Its there, its documented, and we're encouraging others to get involved to make it work. If others get on board then we'll see improvements everywhere. If not, then FastMail will continue to use it and get the advantages that come from it.
Anyhow, this is all for a “Standard track” RFC. An “Informational” or “Experimental” RFC has no such requirement, and can basically be submitted by anyone for anything.
For me instant notification of inbound message is a must-have.
Our actual push does include some extra detail, but the important bit is just "here's a new state that exists on the server - if you haven't already heard of it, you probably should poll for changes now".
It's out of band(ish) - because JMAP is connectionless. And because depending on which platform you're using, there might be a nice channel for push notifications already.
In any case, I think you could have something a lot closer to REST, even if you carry on with grouping calls together.
Something like:
[
[ "GET", "/messages", { "search": "foo" }],
[ "GET", "/mailboxes", { "etag": "bar" }]
]
REST is way too slow. The JMAP format massively reduces round trip times, especially when you have sequential operations where you must wait for one to finish before the next can happen. And once you're not doing REST, I don't personally think putting HTTP verbs in the method calls doesn't really make it any clearer, and is just likely to confuse matters.
I hear you (REST and batch operations don't mix too well). But wouldn't it be possible to simply provide higher level operations, grouping several of these commands?
> And once you're not doing REST, I don't personally think putting HTTP verbs in the method calls doesn't really make it any clearer, and is just likely to confuse matters.
As I said, it wouldn't necessarily be restricted to HTTP verbs (eg, using 'create' and 'update' instead of 'POST' and 'PUT'), but I do like the idea of separating the resource you act on from the action on this resource.
Do you have an example of the type of operation that would map poorly to REST?
Also interested in hearing if you know anything about GMail's json api? Surely they must have something similar to JMAP which their web client is using even if it the api is proprietary?
However! as someone who has had to debug some particularly gnarly email issues in the past I really do quite like the ability to telnet into an IMAP server to dig around. I also have an appreciation of some of the more horrific things that can go wrong...
So my point is that I'm not sure I agree with "generally much easier to work with than IMAP" - there's two types of "working with". One is building a new client/server from scratch - I'd agree this might be easier, but not something that is done that often. The other is being able to quickly investigate issues with hand-crafted queries, and unfortunately I'd say a simple plain text question-answer interface is better - and here IMAP wins.
Maybe I'm just getting old - it's probably true that almost everyone is happier with HTTP than "telnet-able" protocols (SMTP, IMAP, Redis, etc.) nowadays.