But what if that makes the problem worse rather than better, by removing from the agency those people most able to sense and push-back against secret abuses?
If the armchair-ethicist standard is: "if you have any qualms, you'll quit" – then the type of people doing recruiting, and being recruited, and staying in the agency, all become even more self-selected for total devotion to total surveillance than may already exist. Whatever oversight or shame might remain as an internal check would decay. Whatever hints/leaks we get would dry up even further.
That isn't necessarily any better of a result for us. It doesn't necessarily bring reform/correction any sooner.
I'd like to think the no-fly-zoning of Bolivia's president is an example of that sort of thinking exposing itself for public embarrassment/criticism. Here's hoping they keep digging their own hole deeper and deeper.
You seem to be assuming that every person joining the NSA makes it worse, and that's not clear to me. In particular, a small, cohesive, monocultural institution will be more likely to commit abuses and more able to keep them secret.
I doubt the NSA/FBI/DoD/etc are going away anytime soon, so a simplistic strategy of "try to deny them good people, force them to get worse so they eventually collapse" is unlikely to result in either marginal improvements in their behavior, or their wholesale replacement by better institutions. This 'talent boycott' could just mean more acrimony, abuse, and even violence without ultimate remedy.
Some diseases don't get worse before they get better. They get worse before they get fatal.