Are you sure this isn't more the Desktop Environment/Display Manager than X11? Or otherwise something to to with your use case?
I've primarily been using AwesomeWM for the last few years and occasionally XFCE (both on ArchLinux) and I cant recall ever experiencing what you describe.