I think it’s mostly the middle class suffering. Would a billionaire save 300€ a year for a pool? And does the IRS need so many more agents to chase rich people?
Also without an army of tax consultants and lawyers it’s easy to not be aware of everything you need or don’t need to pay, and most likely you’ll end up overpaying. At the same time richer people will have opportunities to legally reduce their tax bill that ordinary people don’t have.
Unlike for instance in Germany, I’m not paying for my boss to have a lower tax rate than I do.