As a French I never found an American TV show that I had difficulties with (love the Appalachian accent btw) but English shows were another matter (usually when there are criminals).
But if you tell me it's difficult even for Americans I'm a bit reassured about my abilities ;-)