I got quite disoriented once going to the southern hemisphere because my northern hemisphere-trained instincts though that the sun should be towards the south at noon, when it's actually to the north.
In Sweden, I had to really adjust my calculations based on the season. If it's summer then the sun sets almost in the north. In the winter, sunset is much closer to the south.
That said, I agree with you. It's easier to say "in the US, if the time is 12:00 then the sun is roughly south" than to have to localize it for the different UTC times for noon in Hawaii through to Florida.