Its one thing to do hard physical exercise, like gym work, cycling, swimming, running etc for like and hour or two everyday, and then get good rest- Work in the time there is in between. This is actually far healthier than a full time job doing hard physical labor. There is certain thing call physical wear-and-tear which is real, together with injuries, and you won't be getting paid nearly as well or with the same benefits and perks. Eventually your mind will find a job repulsive that doesn't pay as much.
I also heard some where at some point nearly every one working these hard labor jobs indulges in after work drinking as a means of pain relief analgesic.
The thing that you describe about the average working job Joe, is really physical abuse in the exact opposite direction to the extreme physical labor. Both are bad. Balance is how you enjoy life.