This doesn't sound like it's that. He does, after all, call it 'inexplicable'; literally unable to be interpreted or expanded upon. It was a 'gut' feeling, a combination of all the things you pick up quickly and instinctively, the summation of which is discomfort.
That _also_ doesn't mean the guy is bad, or that it's necessarily fairly representative of him. Just that he gave the OP a feeling of discomfort.
Gut feelings are often surprisingly accurate, sometimes we all pick things up a lot earlier than our consciousness cottons on. But that is no standard to hold in public discourse. There are a bunch of people who give off creepy vibes who are great and a bunch of people who give off positive vibes who are creepy. Binning people like that should ideally be done with evidence or at the very least an argument to give the comment some substance.
We've figured out that good looking people are not more upstanding than ugly people. Truly the next step is to avoid comments like that. Truth is no defence for a comment like that; a comment that says nothing is automatically truthful but also meaningless.
There, see the difference? You said it yourself - as it turns out, being good looking vs. ugly is not meaningful evidence as to whether you're going to treat others fairly. Being actively disdainful of others to the extent that you're giving off huge "creeper" vibes (perhaps unwittingly, perhaps intentionally!) can be evidence of sorts - at least in a very loose, "more likely than not" sense.
Of course, this is not to say that it should be considered anywhere close to OK to spread wild rumors about "the creeper vibes that this creepy guy gave me once", or anything like that - just think about how open this would be to abuse! But OP wasn't doing that, at all. He/she was seeking to confirm the assessment that others had already, independently come up with, and that can be a very good thing.
I am not saying that the OP or anyone here is being racist. We need to provide more context to these statements if we are going to say these in public. Otherwise, some groups of people will be at the receiving end of these kind of statements more often than the rest of us.
They don't have to justify themselves to you. He saw what he saw, and felt what he felt. if you have any problem with that, that's on you.
