I think this is more a meme than necessarily a reality - you'll tend to see a lot more comments online from people who think the old joke is funny or who want to validate their own choice than from people who don't care or have tried both without such strong feelings.

In reality, there's really no barrier to 'manage' to use both emacs and vim. If anything, since they're both powerful and configurable programs whose best configuration is personal and some way from the default, it probably is harder to construct your own perfect config in both than for simpler alternatives like web browsers - I guess this may lead to a practical aversion.

Personally, I think vim's modal editing is superb but generally prefer the behaviour of emacs' modes and client model, so I use emacs with evil-mode but have no problem using vim if necessary.