It provides a very weak link to the entire claim of end-to-end.
Security minded knows that you also need attribution and chain of command. Your example of hopping through hoops is your own doing, which you are free to do on your own. For free. This product is provided as a SECURE means of communication and it IS NOT.
You are not an attractive target. That is ok, usually preferred. That is not the situation for everyone. However I bet someone will using Zoom is likely to be a person of influence in a major industry of organization that you have an interest in. With a target in mind, you now have a goal: Find a way to convince zoom to send encrypted comms to any device within reach. Note it doesn't mean you NEED a device to be dumb. You just need the smart device to convize Zooms servers that is is "dumb" (or a land line, fax machine, etc). Once convinced it will happily send the data onward.
This is the type of problem that will eventually be exploited in a major way if their mixed messaging is not curtailed. Suggesting otherwise is only kicking that can down a longer road, off a bigger ciff.