I think when people say that, it's typically along the lines of "that person is very [attractive|smart|rich|combo] and could have any of millions of partners at will. I'm unremarkable in all [perceivable|probably-important-to-them] ways, so it's an extreme longshot to pursue them."

Doesn't mean they're not wrong. Everyone has a chance, sure--Beauty and the Beast, right? But they might want to optimize for "people in their league," especially if the fear of rejection is in them like the guy in OP.

...back on topic!