Part of the issue is due to the success of Python in very different niches. The likes of Rails or Node can concentrate on specific ecosystems, which account for the bulk of their users and have a limited set of scenarios they have to support; whereas Python users come from sysadmin to data-crunching to web to desktop development to games to to to...

So each packaging tool comes with certain ideas, usually a result of the author's experience; maybe they work very well in this or that scenario, but then they break badly on others and sizeable chunks of the community revolt. So a new tool comes around and the cycle starts again, but now people also want compatibility with the old tool.

I suspect part of the solution will require splits between niches. It already happened with Anaconda, which has basically become the standard in a particular group of users (academia / datascience). Since that came around, lamentations around building C libraries have substantially reduced (to be fair, the arrival of precompiled wheels on PyPI also helped). Some similarly-specialized tool might eventually emerge as standard for other niches.

Python developers are cats and they are pretty hard to herd at the best of times, which is unsurprising -- who would stick around a language that is almost 30 years old and was never promoted by any major vendor? Only hard-headed fools like myself.