You obviously need to be tactful about it, but if you're genuine and willing to listen to people about the problems they face and you're in a position to solve them, in even just a tiny way, they won't be annoyed. Totally the opposite in fact.

People assuming things like "they'll be annoyed to talk about the problems they face" is really another form of discrimination. It's so much better not to make assumptions about people and let them decide if they'll help you or not.