The prototypical wealthy person is a small business owner or a well paid professional/executive. It's likely that they're very busy, and the prospect of them spending Wednesday afternoons volunteering at their schools is... unlikely.

>if all of the greedy, self-interested, disengaged people decide to leave, how is that not an improvement?

It's a huge problem if they leave and take their tax dollars with them.