Then you haven't heard of Xubuntu? It's an alternate flavor of Ubuntu that has Xfce as the default install. Here's an iso that will get you basically the exact setup I have, no futzing required:
http://xubuntu.org/getxubuntu/Already installed stock Ubuntu? Install the xubuntu-desktop metapackage (in Synaptic or 'sudo apt-get install xubuntu-desktop'), log out and select "Xubuntu Session" from the login screen. That's it. I did this on my laptop after realizing I didn't want to switch to Unity. (For bonus points you can remove the Unity desktop apps, but that's hardly necessary unless space is at a premium.)
I don't think I'd put up with Linux myself if it was as much of a pain as you seem to think. For me, Xubuntu has been no trouble at all.
