Calling someone who is gay "gay" is generally not a problem. (Depending on context it obviously can be, but that's then more dependent on the context, not the word.)
The implication of calling someone who isn't "gay" is that being gay is somehow bad or something to be offended about. That is as clear a homophobic statement as there can be.
It is homophobic, by the way, even if the intent of the speaker isn't there.