I think that's right but not complete.
I think the biggest pushback came from the notion that a large segment of the population appeared to be against the scientific consensus on Covid, contemptuous about risk mitigation strategies, skeptical/fear-mongering about the vaccines, and pushing conspiracy theories, and then these were the who turned around and said there's a great drug that works miracles against Covid and the government doesn't want you to have it.
Now, this may not be an accurate representation of all those who were talking about ivermectin, but they were the most prominent voices -- the "do your research" types.
The bias that this produced in others against even considering ivermectin was strictly irrational, of course, since it was against the sources and not about studies, but, that said, many of these same people had been pushing hydroxychloroquine, massive vitamins C injections, and urine therapy, all without evidence. The underlying desire for ivermectin to be real seemed (at least from the outside) to stem more from discovering some cheap secret weapon that "the scientific community" didn't want them to know about.