The number of miners does not really matter. Whatever miner mined a given block must receive enough to cover the electricity cost, regardless of how many other miners are involved. There is also no particular incentive for a miner to reduce his own electricity consumption in response to the difficulty being reduced. The only reason miners will shut their equipment down is if they are unable to cover their electricity costs because e.g. nobody is willing to pay the higher transaction fees.

Obviously everything I described is a simplified model where energy costs change equally for all miners. The real world is not that simple, but there is some correlation in energy costs across different regions as there is a global market for typical fuels (coal, natural gas, uranium, etc.).

It is also important to remember that transaction fees are not in proportion to transaction sizes. People doing large Bitcoin transactions could absorb much higher fees, so in all likelihood nobody would be priced out of the market (and Bitcoin would be dominated by large transactions, which is more or less the case right now).