The math... depends.
Not all publishers offer advances, and those that do offer advances with different strategies.
Here is one strategy: a commissioning editor uses a sales modelling system to estimate the maximum possible advance against likely sales.
At larger publishers, you can be sure the editor has a very good idea what the likely sales will be. Because that's what publishers do, and they have years, sometimes decades of data and experience to draw on.
The editor will set the advance so the book never earns out.
This gives you more money than you might expect for an advance. And you can usually haggle upwards from the initial offer.
But of course you've just signed what is really a work for hire contract. You will not see any more money than the initial offer.
Would you earn more self-pub? Possibly. That depends entirely on your ability to reach an audience and sell your book to them. Parking a title on Amazon isn't nearly enough.
If you have good marketing skills and if you hit a desirable niche you can do very well - much better than trad pub.
But if you miss - you've wasted maybe a few months, with not much to show for it.
So it's a risk/reward calculation. If you know you can write the book, an advance is money in the bank. It's a known quantity. You don't have to do marketing too.
Sometimes that's a workable deal, especially if you have a day job and marketing is not a strong competence. You won't early nearly as much as you would in dev, but - it's a book. With your name on it. And some cash.
If you have the time to spare and either know or hope you can build an audience, and are sure the title really does fit a niche, by all means self-pub.
But you should only do this if you've done the groundwork. Like any other business you have to think about product/market fit, and marketing, and finding your audience, and all of the usual hurdles.
And unfortunately, marketing is a skillset in its own right. Of course you can learn it, but starting cold is challenging.
Personally having been through this cycle a few times I wouldn't trad pub now. But I also wouldn't assume that all I have to do is write a book and it will automatically sell in reassuring quantities.
I know it won't. There's more work to do, and I wouldn't want anyone to underestimate how hard it is.