X11's extension mechanisms can - and has - been used to enable backwards incompatible protocol changes. E.g. BigRequest changes the length and format of every single protocol request.
Very few client libraries are only capable of speaking "the existing" protocol if you take that to mean the original unextended X11 protocol.
Adding an X11 extension that when enabled cleans up a lot of cruft would not have been a problem.
> where ideally the window manager would make choices about things before the process continues
Nothing stops you from introducing an extension that when enabled requires the client to wait for a new notification type before continuing, or re-defines behaviour. That said, using my own custom window manager, I don't know what you mean here. My WM does decide the initial window placement and size, and it's the clients damn problem if it can't handle a resize before I allow the window to be mapped.
The X protocol is crusty in places, but it is very flexible. People haven't fixed these things because they chose to invent compatibility hindrances that weren't real when their response was to invent an entirely new protocol with no compatibility at all.
