I think you have a responsibility to understand how your words and/or actions may impact someone else's emotions. It is then your decision to act in a suitable way to you.
But if something needs to be said, you shouldn't shy away from saying it, if you're worried about how they might react.