Using male pronouns instead of genderless/more inclusive pronouns in your speech is indicative of a bad managing style (or just bad style all together). As a manager you should be inclusive of all genders both directly and indirectly, which includes how you use whatever language you speak.

The people you manage could easily identify with a gender that is not physically obvious and using incorrect pronouns leads to a less inclusive and potentially more hostile workplace. This is Management 101.