I've found that putting swap files on things that don't behave like filesystems can cause interesting behaviors. In this case, all writes to the file would go through the VFS. I imagine there could be some curious issues if that write path has any allocations or significant amount of mutation in it.

I would trust more something that got rid of the VFS layer and simply allowed VRAM to be used directly as a second level below RAM using the transcendental memory model.