Funny. I'm like that too, if the interviewer isn't focused on the actual position and what I'd be doing in it and the problem solving and knowledge surrounding the actual work I'd be doing on a day to day basis, I bail because it's a bad sign to me that I'm talking to people who are way too comfortable wasting time.

But luckily I'm usually in the hiring manager's chair these days and my interviews aren't easy but they're not a stupid game.