No, you don't. You just have to accept that the facts about physical phenomenon that can be learned through language are different than the physical experience of physical phenomena themselves, and that learning the latter is distinct from learning the former. The latter, however, is obviously physical.
And, even if it wasn't wrong in that way, that would be an argument for the nonphysicality of some subset of the subjects of knowledge, not an argument for the nonphysicality of consciousness.