Just because something doesn't offend you specifically (even if you are part of the group something offensive is being said about) doesn't mean it isn't offending anyone.
It's very common in countries like the US, for you to have a lot of rights, but their coverage always ends where others' rights begin. You cannot infringe on someone else's rights, and free speech does not cover hate speech. Hateful or derogatory speech has a significant negative impact on the mental and emotional well-being of a lot of people, and suggesting that it shouldn't just because it doesn't affect you is not a valid reason to not try to educate Jimmy on how not to be insensitive.