What is intrinsic? Is being poor intrinsic? Can I spew hate against the poor and the homeless then? The problem is these definitions are unclear, and policing hate against black people obliges you to police hate against the homeless, and pretty soon everything else. I have seen far more examples of slippery slopes happening than not, so forgive me if I'm skeptical.
The whole point is that lines are not easy to draw. Religion is a belief system and is not an intrinsic property either. Are you going to allow speech like "Kill all muslims" which is not very different from "Kill all capitalists".
Free speech is so critical to a functioning free society that I'd rather err on the side of allowing bad speech rather than banning not-bad speech by mistake.