"Toxic" is vague and imprecise in a way that raises the likelihood that multiple participants in a conversation will have different interpretations without noticing they don't all agree on what it means.
This lack of clarity is unavoidable in all communication. We can't transplant thoughts directly from one brain to another. But I find that words like "toxic" convey an assessment, rather than description, in contexts where that level of abbreviation is not helpful.
If the more-precise descriptor of behavior was used in the first place, the toxicity of that behavior will likely be self-apparent. But if "toxic" alone is used, we either have to circle the conversation back to pick up the better descriptor, or just assume that the speaker is right.
Toxic is a bad word because it's become a go-to choice for poor articulation. Using it conveys an impression that an assessment of "toxicity" has been made off-screen, but is broad and amorphous enough that I can't trust it implies anything more specific than a snap judgment.