If I'm listening to ambient music I don't suddenly want to be ambushed by something uptempo. If I'm listening to e.g. Debussy, you might be excused for suggesting something vaguely new age in a similar mood and tempo, but certainly not rock.
Another problem is that you can't just take raw overlap in tastes, because some people like "everything", and the fact their tastes overlap with mine does not mean I'll like everything else they like.
I've yet to hear a recommendation system that chooses music I want to listen to reliably enough that I can generally stand to listen to them for more than a few songs at a time without it turning into an endless annoying sequence of skipping.
Respecting genre (segues need to be gradual, if at all), respecting mood and tempo needs to come first. Then you can consider what others who likes the same songs within those constraints also likes within those constraints. Honestly if I have to choose between personalised recommendation and precise control of genre and mood/tempo, I'd take genre and mood/tempo over personalisation any day.
Another pet peeve of mine is lack of visibility into how to teach a system what I want. E.g. if I dislike or skip a song, will it get that it doesn't fit my current mood or what I want to listen to now, or will it wrongly infer I don't like the song at all?
Sometimes it feels as if the people designing these systems don't use them.