I've only had that happen a couple of times for me, but that's because I'm mostly a Lyft user. What was more common on Lyft was drivers calling me to ask "Where are you going?", and when I'd respond with "I already put my destination into the app", they'd hang up and cancel on me.
Edit: About the cancellation penalties. I don't know about Uber, but on Lyft when I as a rider have to cancel on somebody, I sometimes see options pop up saying "driver asked me to cancel" or "driver is not moving". They know when drivers are using trickery to get out of dinging their cancellation rate. And when I have to cancel for "driver is not moving", I also get a message saying they'll waive the cancellation fee because there was an issue with my ride (note: I get this message before I see the list of cancellation reasons to pick from, not after).