I once had this really fascinating discussion with my father. He said, his IQ while not ridiculously high was in the ~1 in 30,000 range which suggests in theory something like 10,000 people in America are 'smarter' than that and more importantly around 200 his age. But, many of them don't even end up at such schools and 200 people can't fill the fill the 'top' schools anyway. Thus, these schools can't be anywhere near that exclusive.

At the same time, everyone bases their internal scale at their level so while he met a few people that where smarter and many where far more educated there was always this feeling like he was surrounded by people that where just a little slow.

PS: He also generally referred to IQ in terms of speed. Most people have interesting things to say as long as the conversation is slow enough for them to follow.