Attending college increases pre-college IQ scores a mean of just under 20 points (I don't recall the precise number from the study now, I remember it was between 15 and 20). That's not "smarter people get into college" - that's before-and-after on the same subjects.

Growing up wealthy entails a lot more advantages than four years of courses, at that.