You're right - but in that instance the justification in my mind (and what you'd presumably argue if challenged legally) is that they're difficult to work with - which I'm okay with.
But not wanting to concrete that person's driveway because they're in the army, or because they're a politician or whatever else along those lines I think shouldn't be allowed.