It'd be nice if you could just customize Android on ARM and have lots of different spin-offs in the same way you have various x86/64 and PPC Linux distros. I wrote about this:
http://penguindreams.org/blog/android-fragmentation/
The problem is with all these ARM boards. As other comments have pointed out, vendors have tons of binary blobs and shim layers that link them to the kernel. Nvidia/AMD do this too with their video drivers, as well as Intel/Broadcom/Atheros with their Wi-Fi/BT chips. The difference on the PC platform is that it's standardized enough that you can run any kernel/distro on almost any x86/64 board and it will boot and give you a console.
Android vendor kernels are patched to hell, duct tape, just enough crap to get to production style software. They don't submit patches upstream because their code is often junk. ARM SoC are also all incredibly different.
You can download x86/64 versions of Debian, Gentoo, Void, Slackware, whatever and it will at a minimum boot on any PC hardware made in the past few years (probably even a Pentium if you use a 32-bit distro). Not all your hardware will work, but it will at least boot. ARM makes no similar guarantees. Cellphones don't support device trees, and even if they did the whole device tree system is a mess of its own.
Google has ultimate control over all vendors with the OHA. They can mandate standard kernels and drivers across devices. They won't though. There's simply too much money in requiring people to replace devices every two years.
