It's fine as long as they're open about it. It's when they say "We're a very selective institution that only accepts the academic best of the best from the entire world" and then also allow pay to play, clarifying "Also the people whose parents donated us buildings juuuuust so happen to be the academic best of the best from the the entire world" that people start to question just how selective admission really is, and just how world-class their student body and standards really are.

If they kept stats on who was an endowment/legacy admit and gave them a different colored diploma so people could filter them out when assessing things like grades and graduation rate and they didn't effect the curve I think there would be less criticism of the process.