Actually now that I look into it again, it looks like since the middle of March of this year it's even possible to invite others without sharing your phonebook.

https://www.blogher.com/social-media/clubhouse-invite-withou...

https://www.gizchina.com/2021/03/16/clubhouse-new-update-use...