To me, the key downside of philanthropy is the "without democratic oversight" part. It's all fine when the billionaires are funding homeless shelters or schools: things everyone can agree are a public good. We are extremely lucky that guys like Bill Gates suddenly got a soft spot for fighting malaria after amassing their fortunes. But nothing exists to stop these billionaires from donating to causes against the public good.

That's the problem. The public should define what "public good" means, not rich people. As inefficient as taxes/government is, at least the voters are theoretically in the driver's seat, and get to at least indirectly define public good through elected representatives. With philanthropy, we're simply letting a single wealthy demographic decide "this is public good, trust me bro", and the public can't vote out rich people if they disagree.