In that case, they can claim a "Foreign Earned Income Exclusion" [1] which reduces or eliminates their American tax liability for up to $97,600 of income that they are "paying full taxes" on.
> A place you're not living has no business collecting further taxes from you.
As other comments have noted, just because you are not living in the United States does not mean that you are not materially benefiting from citizenship. Despite living abroad, you can still vote, travel to and from the United States, and take advantage of US diplomatic representation in case of arrest.
[1]: http://www.irs.gov/Individuals/International-Taxpayers/Forei...
Basically, the US government's attitude toward Americans living outside the US, much like the US attitude toward non-Americans wherever they live, is "F* You."
Perhaps you think that this attitude is somehow justified. As a US citizen myself, I think that it is detrimental to long-term US interests.
I think it is a complex issue, and should not be dismissed as "not making any sense" or the government wanting to screw citizens living abroad over. As I have already stated, it is clear that the purpose behind the legislation is to prevent unreported income from being stored in foreign bank accounts.
Like with many complex issues, I think the current solution is imperfect and could be improved. Stating that it is stupid is not constructive.
The reality is that the truly rich (like Mitt Romney) have plenty of legal ways to shield their income, offshore or otherwise. So these regulations screw (the vast majority of) Americans aboard with relatively modest incomes, while they are ineffectual against the rich few with tax accountants on their permanent staff.
tl;dr - these rules create more harm than benefit.
