I think the real solution here is to just only use python dependency management for python things and to use something like nix for everything else.
NixOS is a stark contrast to Python here. It makes things that can't be done deterministically difficult to do at all. Maybe this sounds extreme from the outside, but I'd rather be warned off from that dependency as soon as I attempt to use it, rather than years later when I get a user or contributor than can't make it work for some environmental reason I didn't forsee and now everything rests on finding some hacky way to make it work.
If Nix can be used to solve Python's packaging problems, participating packages will have to practice the same kind of avoidance (or put in the work to fix such hazards up front). I'm not sure if the wider python community is willing to do that, but as someone who writes a lot of python myself and wants it to not be painful down the line, I am.
Now when I cd into a project, direnv + nix notices the dependencies that that project needs and makes them available, whatever their language. When I cd into a different project, I get an entirely different set of dependencies. There's pretty much nothing installed with system scope. Just a shell and an editor.
Both of these are language agnostic, but the level of encapsulation is quite different and one is much better than that other. (There are still plenty of problems, but they can be fixed with a commit instead of a change of habit.)
The idea that every language needs a different package manager and that each of those needs to package everything that might my useful when called from that language whether or not it is written in that language... It just doesn't scale.
And yet, python is a fantastic language because it’s the remote control to do the heavy, complex, low-level, high-performance stuff with relative ease.
I wish there was a standard interface that tools like pip could use to express their non-python needs such that some other tool can meet those needs and then hand the baton back to pip once they are met. Poetry2nix is an example of such a collaboration. (I'm not trying to be a nix maximalist here, it's just that it's familiar).
The python community is large enough to attempt to go it alone, but many other language communities are not. I think we'd see faster language evolution if we asked less of them from a packaging perspective:
> focus on making the language great. Provide a way to package deps in that language, and use _____ to ask for everything else.
Apt and brew and snap and whatever else (docker? K8s?) could be taught to handle such requests in a non alter-your-whole-system kind of way.
