What's wrong isn't so much the package managers as the necessity for them and for an army of third party volunteers to maintain packages and all the problems that predictably arise from that. Linus famously complained about how things work at DebConf 14[0], but I guess he's an idiot for doing so? That's a pretty hot take but whatever.
If a package is not in the repo? Sorry, you have to compile from source. Want a newer version? Compile from source and hope that the build environment dependencies are in the repo. Want an older version for some reason? Break out docker or KVM so you don't break your system.
None of this is fundamental to the model, that much is true, but in practice it is how all Linux distributions using a package manager/repo model without things like Snap, AppImage, and Flatpak work.
Here's the best part though: Even with Flatpaks and AppImage you can still use a repo! In fact Fedora Silverblue, which uses an immutable base system and installs everything through Flatpak and Toolbox, uses a Fedora controlled Flatpak repo by default.
[0] http://saimei.ftp.acc.umu.se/pub/debian-meetings/2014/debcon...