VFIO does work, but it's hacky, so it's very inconvenient unless one has plenty of time and patience.
For example, AMD GPUs don't support rebar (took a while to figure out), and they used to have (I don't know if it's been definitely solved) a problem with reset, which caused random lockups when exiting the VM.
Additionally, power saving/device ownership must be handled. Which means: either one terminates the X session each time before starting/after stopping the VM (so X will handle the power management), or they leave the card owned by the vfio driver (but this makes the card run hot while sitting doing nothing, because the vfio driver doesn't handle the power management), or they swap drivers on the fly hoping that it will work fine (on nvidia, it's not a stable strategy; no idea on AMD).
And these are just a few issues. I've experienced others in the past, and it's very frustrating and time consuming.
With all these problems, a combination of dual boot (for demanding 3d tasks) and VMWare (for light ones) is much simpler, and stable. I hope to replace VMWare with VirtualBox for the latter use case.