I had a very similar experience - dual-boot is nice but not ideal because restarting takes time and you lose all your context. Now im running a (headless) ubuntu server on the host with a windows VM with GPU passthrough on top. With a modern 6-8 core CPU + extra RAM (+ a cheap 2nd GPU for linux if you want a desktop environment on both linux and windows active at the same time) you can have both windows & linux running simultaneously with no need to dual-boot. VFIO & GPU passthrough makes the VM run pretty much at native speed although it does take some tweaking to set up.
That's an interesting idea: certainly worth experimenting with. As you say, with GPU passthrough it ought to be possible to get a near-native experience on the virtual desktop.
