American adults read so little that this attitude only comes from a minority of the minority. For the most part, people see it as a waste of time because they may only read 1-4 books a year, so they might feel those books should have some meaningful purpose in their lives (or they only read a book because it is supposed to be great or have some use they can apply to their lives).

Read more, worry less about what others think. I read an average of a book a week (and I'm a slow reader), and most people find that to be argument enough to read whatever I want.