I believe E2E means that data is encrypted and decrypted on endpoints and nothing else. It doesn't imply how the keys are produced or who else knows the keys.

But I was thinking about a scheme, where the key is encrypted with a passphrase (that user's ought to remember) and kept on server. You fetch the blob, decrypt it (server can't), get the key and thus are able to decrypt the existing data (message archive).

This lowers security, but adds a significant convenience of being able to recover history if the only device is broken or lost. Which may be important for casual users.