And I don't even mean racism - say your potential candidate is older with kids (assuming you're younger) would you pass them because they cannot go out drinking with you after work (a.k.a social circle)?
That said, I agree that you want to work with people you get along with, but it has to be at work. Social circle is the issue for me.
This is, and has always been, a real problem: That your ability to get good positions is limited not by your skill set but by your ability to network.