The existence of ultra-wealthy people isn't inherently bad.

The question is about the alternative. If everyone's quality of life improves (including relative to the ultra-wealthy), I'd rather that happen than reduce everyone's quality of life out of a sense of "fairness."

Claiming most libertarians act against their own interests because they think they can become ultra-wealthy is straw-manning the view.