> These particular four elements don't need closing, except this one might if it's not followed by one of this other long list of possible elements.
That’s how it’s worded in the spec, because that’s targeted to browser implementors, not HTML authors. To someone authoring HTML, it’s much simpler: the tag is closed by any encountered opening or closing tag that the tag is not allowed to contain. That’s nothing new to memorize, because you already have to know what elements are allowed in that context. E.g., even in XHTML served with an XML MIME type and parsed with an XML parser, <p><p></p></p> is invalid XHTML, because <p> can’t contain <p>. I already have to know that rule when writing a webpage! Knowing that <p><p> is equivalent to <p></p><p> doesn’t require higher mental overhead.
> I'll bet the reason it's those four is edge cases in a parser that someone wrote in a couple of hours 30 years ago, not any sensible logical classification
Nope, the self‐closing behavior has been present in HTML since the very beginning (inherited from SGML). As for why these elements specifically, it’s because they are among the easiest and simplest to remember due to their definition. <li> can’t contain <li>—the concept doesn’t make sense—so it makes sense for <li><li> to implicitly close the first one. <td> can’t contain <td>—the concept doesn’t make sense—so it makes sense for <td><td> to implicitly close the first one. And so on.