Being a leader of any sort makes it worse, but I don't think it's a necessary condition for justifying his removal.
Each person contributes to the flavor of the community in which they participate. It is within the right of those who lead the community to curate its flavor by adding or removing people when necessary.
In this his case, his removal was well-justified for the cause of keeping the community welcoming and safe-feeling for members of all genders.