Right, and this is kind of the key problem that Zoom is having to solve right now. The only material you (used to) need to know to join a meeting was a 9-ish digit meeting ID, and since that meeting ID entitles you to negotiate keys, it more or less nullifies encryption guarantees (except the post-hoc guarantee of the meeting host being able to see if something was up).
I would point out that this is in no way unique to Zoom, though. In fact, after the changes Zoom made in response to all of these issues, Zoom probably has the highest by-default level of meeting security of just about any product out there.