But most of the real nightmare scenarios I've heard related to backwards compatibility have more to do with third-parties doing things they were never supposed to do.
Things like hitting private, undocumented APIs. Or checking the Windows version with a "9x" wildcard, giving us the jump from W8 to W10 over a decade later.
Microsoft has made their own mistakes, but supporting the mistakes of third parties has been absolutely vital to them keeping their core customer base.
I don't personally believe the "Windows 9" story - if a program is old enough to feel the need to check for Windows 95/98 then it should already be fine to run under Windows' own app-compat layer which spoofs the Windows version string anyway. I believe it's marketing-based out of fear consumers would see "MacOS 10 vs Windows 9" (like how it was PlayStation 3 vs Xbox 2 - hence "Xbox 360").
In any case, your reasoning doesn't really make sense. I can run a program that was written for Windows XP on Windows 10 without the need from a app-compat layer. Given that a developer can hide/show all sorts of random functionality with an if-branch-on-version - the user will see a broken or strange app and it won't be clear (and MS probably didn't have the means to detect) that the app should run in compatibility mode.
I still believe that MS wanted to ship an OS that "just worked" and did so under 10, than trying to compete in version numbers with an OS that has had 10 in its name for last 18 years.
X means 10 in latin.
Look at old MacOS X 10.3 books, X always stood for 10.
This is just a rumor made up on reddit. It's completely false. Windows returns its version number as a series of integers. And even if it was a string, they could've just called it "Windows Nine".
That's entirely Microsoft's fault too: they made internal APIs accessible to applications, and did not provide comprehensive documentation on the full features of their public API.
The latter means that application developers were forced to just guess what a function could do. And since no API performs proper input validation, undocumented usage became the norm.
Public APIs are documented, guaranteed to work like documented (otherwise it's a bug and will be fixed) and intended for others to call. Internal APIs are none of these things. They are not documented because they're not supposed to be called. Anything not documented in an API it (to me at last) something that's not guaranteed to remain like that. Whether it's a complete function or just a side-effect of something that is public.
They did, in Windows 8. It is called Metro/Modern/UWP. Everyone hates it.