There are lots of us who feel that way.
Don't you ever think, If I was president I wouldn't have made X decision? Or if I had the power I would abolish the IRS, scale back the government, etc?
As a libertarian you would believe those things to be good, so you would want them, right? Supposed somebody offered you the position of power to institute them, wouldn't you accept it?
I don't want to be in charge of your life, but I don't want to be responsible for paying for it, either.
No. Money does not give you control of my life.
Bill Gates can have all the money he wants, but it doesn't mean that he chooses where I can go to school, whether or not I want to risk having health insurance, whether or not I can fill a ditch on my property, whether or not I can marry my same-sex domestic partner, or even whether or not I have to use Windows8.
Government, however, controls all those things and much more. In the context of this thread, the pejorative use of the word "elites" typically refers to those who approve of using government-like power to control the lives of others. Usually, because they are members of the ruling class that has influence with the government and/or because they have plenty enough money to circumvent many of the ways that the government can control their lives.