You are overstating the amount of mixing between populations (and somewhat discounting the mixing within populations).
1000 years ago, people in villages and towns were regularly marrying their (2,3,4) cousins and rarely marrying people from 'different lands'.
Another way to look at it: for 1 person to become the ancestor of just several thousand people takes 'a few generations'. Getting to whole populations takes lots of generations.