Linux is not as secure as most tech people would assume at first glance. The monolithic kernel with all device drivers in ring0 is, let's just say, not the best approach if one were writing a new OS from scratch.
It is mostly "secure" due to it being used in practically every server and billions of devices, so there is an active maintainer community around it. Xorg has none of that.