Both are are really bad desktop environments and part of the reason the FOSS desktop never took off. Also they are mostly developed by full time paid employees with zero community involvement. As such they are the small niche.
> The glaring flaws in the API have been known for that long.
X11 has flaws but I wouldn't call them "glaring". They are a nuisance at best. You don't pay for functionality you don't need. Wayland has glaring flaws because it does not provide and standardize functionality that people need. It also has severe technical flaws like implicit sync which makes all your application stutter when one application has high GPU load.
> Xorg server actually lacks backward compatibility with lots of non-standard X11 extensions that for whatever reason were either removed or were never merged upstream.
Great, so there is a regular organic and efficient clean up process happening that keeps the unused or unpopular stuff out of X11. There shouldn't be much "old cruft" around then. If this is the case, why do we need Wayland?