Certainly it can feel rude to us, but the law allows it and moreover the feeling of rudeness comes more from believing we "own" our likelinesses in public spaces, which we do not. So, yes, I could feel intruded upon by a documentary or street photographer, no question, but at the same time, they have the right to take my picture, no question.
[1]https://en.wikipedia.org/wiki/Nussenzweig_v._DiCorcia
[2]https://wmuphoto.wordpress.com/2014/02/07/nussenzweig-v-dico...
If you disagree with the position, post a link to law which states otherwise. There is lots of case law asserting the right of photographers to take pictures unimpeded in public spaces.