The idea of finding clever criminals by decryption their communications is stupid by itself. And stupid criminals leave so many trails bragging about their plans that you don't need decryption and back doors. Most european terrorist attacks were done by criminals already known by intelligence, but not monitored strongly (see Berlin). Monitoring all communications would lead to an explosition of the number of suspects. It would not lead to more security.
For every terrorist there are tens of thousands of people who say something hateful or advocate violence. There are no resources to track them all down. Also note that many kinds of attacks require very little coordination.
I think the key is to use the intelligence you have effectively. I think of how the Boston Bomber spent 6 months in Chechnya and the Russians gave us a heads up about that and I think he deserved a little investigation since hardly anybody every goes there and Chechnya is famous for having the world's best terrorist training camps.
How do you track their whereabouts and communication channels if everything is encrypted?
You need to consider what is also unecrypted which may be used.
Given you have a username you can rather easily check whether that username is registered on any other socialmedia-platform; a site like knowem.com lets you search for a username across +500 socialmedia sites.
From there on you can start to interpolate information from usercontent as well as usercontent-metadata.
I think you are the OP, so I will add this also: The security agency properly have software toolsets which are used for hacking.. If you ain't familiar with Vault7 you should definitely give it some reading: https://en.wikipedia.org/wiki/Vault_7
Related to Vault7 there was also a lot of fuzz about 0-day threats which the NSA had stockpiled, these 0-day's may be used for other things not related to cyber-warfare, like e.g. monitoring potential terrorist activites..
Besides this you have whole pentesting-suites like Kali Linux and similars, which may be used to exploit weaknesses on target-machines.