In the same way as Racism and support for Apartheid are completely orthogonal.
Come on...
For instance, I do know many people who oppose gay marriage but are not homophobic.
There is no "I have nothing against them but they should not have equal rights". If you opt not to support equal rights for any given group of people you believe their way of life is not as valid as yours.
A person with sexual attraction to the same sex can restrain his behavior and not engage in sexual activity, despite his attraction. A person with a skin color cannot restrain his skin color, and cannot ever not "act on" his skin color, no matter what choices he makes. Homosexuality is not externally visible, but race is. It is not automatically shown at all times despite the behavior, wishes, or intention of the person, but race is. Someone who makes judgments based on willful sexual behaviors is fundamentally different from someone who makes judgments based on involuntarily hereditary attributes.
This is the difference that must be realized when we talk about racism as it compares to "homophobia". Homosexual behavior, and homosexual marriage, is a thing people choose to do, whatever biological or psychological forces may or may not be at work in the promotion of that behavior. This is completely different from visibly possessing certain levels of skin pigmentation.
"Homophobes" are not the same as racists. Whatever your opinion is, you must acknowledge that "homophobes" oppose a sexual behavior, whereas racists oppose the existence of a class of persons, no matter what choices they've made. Whether you think homosexuality is OK or not, there is a clear difference here.
Please acknowledge the failure of your analogy.