For the common, everyday use case it isn't meaningfully simpler, which is why the US hasn't switched. The conversions are certainly harder to memorize, but by the time you're an adult you have memorized all the common ones (12 inches to a foot, and so on) so that downside only applies to people who have to learn this stuff (largely children, who don't get a vote). The math is also harder than just moving decimal points, but when you carry a computer in your pocket that isn't actually making life harder for anyone.

So, the two big downsides of the imperial system (conversions are harder to learn and the math is harder) aren't actually a problem for the vast majority of adults in the US. But switching to metric would cause a ton of friction as you have to relearn how to estimate measurements for everything all over again. And those two factors combined are why the US doesn't switch. Most people will not gain any upside, while they have to pay significant downsides. It's perfectly rational to not switch when that is the case! You could argue that it's selfish (because future generations of kids have to learn the conversions, so they would benefit from metric and they don't incur the downside either), but it's not stupid. As much as people like to go "haha people in the US are so stupid for not switching to metric", that simply is not the case.