I think they're viewed as the antitheses of "driving enthusiasts" (for better or worse).

Someone who appreciates physical buttons in a BMW likely falls closer to that side of the camp than a stereotypical Prius driver.

I personally would consider myself a driving enthusiast but would have no issue with driving a Prius as a commuter. Quiet, reliable, efficient.. and then I can feel less guilty if I want to have an inefficient "fun car" for the odd sunny weekend.