The fraction of terrorists in "all people that use the internet" is approximately equal to the fraction of terrorists in "privacy seekers" -- both are roughly zero.
The NSA only was given such pervasive power to catch terrorists. The fact that privacy seekers are more likely to be "bad actors" is moot unless that means terrorists because regular "bad actors" are supposed to be innocent until guilty, handled by police/fbi, etc. NSA only is allowed to work essentially with no due process because it's for cathing "terrorists". Interestingly, now that terrorists know about the NSA, they no doubt will simply not use the internet (or phones) at all, thus making it so the NSA can't catch a terrorist by any of its methods.
I definitely expect all terrorists are extremely careful about internet activity now that they know the NSA is so invasive, thus making the NSA's actions even less defensible (not that I thought they ever were).