I don't think that follows.

First, it's absolutely possible to design an E2E system where users can join the meeting after it started: https://signal.org/blog/private-groups/

Second, you can have your phone gateways be stateless and unprivileged: when a user calls up the phone gateway, it generates a new keypair. The user enters their PIN and the phone gateway derives a key from the PIN using your favorite password hashing algorithm, HMACs their public key with the PIN, and sends it to the existing participants. The other participants have the same PIN, so they can decide to let this public key join the call without allowing random callers to join. (I'm not sure if Zoom does this, but it's straightforward enough and it makes the phone gateways much less juicy of an attack target, especially because you can now reboot the gateways from read-only media and you don't need to provision them a secret, so I hope they do.)

Now we're left with the argument about whether it really counts as "end-to-end" if the plain-old-telephone-system part of the connection isn't encrypted, but also it can't be, so I'm not sure anyone reasonably expected it to be encrypted. Anyone who really wants "end-to-end" encryption can just make sure nobody joins their call by phone. (In the end, end-to-end encryption is a tool to make sure the right people join your meeting - i.e., anyone who cares about end-to-end encryption already cares about who the ends are.)