At the same time that is largely a problem of the colder areas. In more sane (temperature wise) places, that energy cost is negligible.
Also, that assumes you are the only one (or the other person (s) in the house have very similar schedules) in the house.
An assumption which if you have a house wife and a few kids, is largely invalid