Further, your supposed use case of audio books is when you're driving, when most of your attention should be spent... Driving.
As for fiction, efficiency totally misses the point. Efficiency isn't reading Harry potter as fast as possible. Efficiency is not reading it at all.
My most charitable interpretation is that we just have different aims when it comes to consuming literature, but that still leads back to the ops point that this is basically the gamification of book reading.