I think by now we're really arguing semantics, but I would argue that I do not lump anyone in with the people I call retards.

I would never call a mentally handicapped person a "retard". To me, that would be an incredibly insulting and inappropriate thing to do. The only people I'd call retards are people who are not mentally handicapped.

Similarly, if I call someone a dick head, I do not consider that to be insulting, or even related, to physical dick heads... it's just an insulting term. If I call someone (a guy, usually) a bitch (e.g "Yo, bitch, pass me the beer"), that does not imply any connection with female dogs.

Btw, I don't see how you can argue that being mentally handicapped (which by definition means having mental issues) can be neutral. It's clearly a net negative. It doesn't mean the person is more or less good, much like missing a leg or an arm (aka physically handicapped) doesn't make the person more or less good, but it's definitely a "bad" thing.