There are other factors, like the quality of your peers, facilities and colloquia, security of funding, teaching load, culture, etc.
And a high reputation school will help in grant applications and so on, where you don't get to include your advisers name.
