None of those factions will be gone, but their battles will weaken their cause more than they have since 2016.
Some of this can be seen by how even his own popularity falls any time he actually has power, since there are no effective ideas there, only misplaced blame, and that doesn't sustain support for four years. Without him there at all in an out-of-power period, the "blame the Jews"/"blame the brown people"/"blame the women"/"blame the baby-killers"/"blame the anti-Semites"/"blame the sexual deviants" factions will likely fail to find another person they all rally around.