> can just leave and let the others continue
That can be hard -- many / some people would be worried it'd be impolite to leave and in a way say "the things you're talking about aren't interesting to me".
And they'll stay and feel frustrated, in silence.