Unemployment is only 4.9% because the participation rate dropped 3% points in the last 10 years: http://data.bls.gov/timeseries/LNS11300000

That's close to 10 million people who have stopped looking for jobs who we would otherwise expect to have them.