The worst spots aren't much worse than the median spots (over calculating silicon, not cache). Anything big enough to be a hot spot, like a big ALU, is big enough to represent a large portion of your power budget. The main goal is to get all the heat away from the chip, and putting peltiers on a large portion of the chip gives you more heat to take away. For anything significantly smaller than that, the heat bleeds out without the need of a peltier. There might be a middle ground where peltiers could make a real life difference, but I'm skeptical.

> or it could mean circuits with far more layers than we can manage now

That sounds like you're cooling the entire chip, which is the worst time to use a peltier.