That's true, you certainly don't. Honestly, it's true of any car - if something looked like a PT Cruiser (to pick a stereotype) but did what I wanted far better than anything else, I probably would have gotten despite the exterior. Though that is absurdly hard to imagine, so it's hard to be sure what would happen.
In any case, I think exterior certainly comes into play at a subconscious level after looking at a bunch of different models for a month... and if two vehicles are otherwise very similar, I might opt for a slightly nicer exterior over, say, 3-level vs 2-level heated seats, or rain-sensing wipers. (On that note, I've never found a rain-sensing wiper system that I liked...)
For context, though, I'm in Vancouver, and as a city we're rather snobby about car appearances.