You seem to always start with the assumption that SGML was (and perhaps is) an end goal. I deny this.
HTML was designed as an SGML vocabulary, but, where it mattered, never implemented as an SGML vocabulary. If Tim Berners-Lee ever even expected it to be treated as SGML very much, I suspect he hadn’t thought things through well enough (though that could also just be hindsight bias on my part).
There has never been any particular virtue in HTML being an SGML vocabulary. No one that mattered (which mainly means browsers) cared about SGML, then or now, and no web developers or end users care about SGML, so being SGML is just needless complication and potential for confusion (due to that implying different behaviour from reality). SGML is a hideous, complex beast that no one wants to work with, and which almost everyone that has heard of it is glad is dead.
Yes, SGML had some nice ideas. Yes, we keep on reinventing parts of it. Yes, a variant of Greenspun’s tenth rule applies. But SGML was just too flexible/generic, large and ugly. It doesn’t actually solve things. And the current HTML parser is the best thing since sliced bread and my favourite popular file type spec by a large margin despite its size, because it’s clear, unambiguous, and implementable.