When that happens to me, it's a sign that I'm listening at a rate that's either too slow or too fast.

I highly recommend speeding/slowing your audiobooks to the perfect point where your mind doesn't wander.