Having granular access != no control. On the contrary, I want more control over the apps. There is no benefit to me if my banking app or any other app can also have access to GPS, while I am navigating with a map app. AOSP is much much better in this aspect. Normal desktops are nowhere near. GNU userspace also simply has very litte preparation for this. Creating granular SELinux rules, and then separating each hardware into its own system agent and then isolating them further with various containerization is no small task. GNU userland is still struggling with styling Flatpaks and keeping them performing well and small. Android's frameworks libraries provide a consistent UI with little overhead.

I don't know why you need ssh with Xorg on your phone though? There are ways like scrcpy to control your phone remotely. Apps like KDE Connect on Android already give quite a bit remote control opportunities too. There are open source remoting apps that fit the Android user interface better like RustDesk. If you still want an Xorg running, there is nothing stopping you from encapsulating it in an Android app too.