I used to think the same. But watch everyone else: aside from the really confident/pushy/needy types there are times for everyone when they are "out of the loop" a bit too. You just don't notice so much as you are either in a conversation (or actively listening to one) at the time. Also you may mistake someone else listening in attentively as someone being an actively interested party in the dynamic where they are in fact listening in because it is the best way at the time not to be out of all conversations.

The old adage "you are not as lonely as you think you are" (replace lonely with ignored/unhappy/bored/whatever) tends to be wrong (IMO), but "other people are just as out of it as you are" is generally true. The world would be more content in general if more people realised this: a lot of depression and anger tends to be magnified by the feeling that everyone else is having a better time of it when that often is not the case.