It depends how much of an advantage a certain diet brought from a procreative standpoint.
If milk drinkers were the only group who could obtain the necessary amount of calories for bearing offspring and sustained existence, then the adaptation would happy much more quickly.
