Secrecy of correspondence still has exceptions. That's what is always lost in these discussions -- every right of every person is not absolute. Just because you have a right to personal property, doesn't mean you don't have to pay taxes or store nuclear material in your basement. That's the hard part.
But end to end encryption with forward secrecy at no cost to user makes your right to private communication absolute. It's a new thing and the balancers can't balance it against other rights of other people, so this happens.