That is a valid argument, but can be applied to any hush hush effort. But such "limited visibility" organizations often push to limit public control, after which their mission or methods may morph to support internal goals that may not be shared by the general public they were created to serve.
It would be naive to require that all government information is shared with the public, but we should maintain robust oversight on all clandestine activities and give that oversight teeth to correct problems when such activities to too far. My 2c.