In the end it's a "what if?" type question, but I think a decent case can be made that anti-Semitism was not key to the Nazi success, although it was part of it. The NSDAP was far from the first or only anti-Semitic party, even at the time. But it was the only major fascist party in Germany at the time. In other countries fascist parties managed without such strong explicit anti-Semitism, Italy and Spain being the most notable.

In my reading of events of the 20s and 30s, it was much more of an ideological battle and disappointment with the ruling class than anything else. This is also why the communist party did well at the time, and one reason the Nazis spent so much effort fighting them even though there are more similarities than both liked to admit.

Or in brief: most people voted mostly for the fascism, not anti-Semitism. The basic concept of "strong leader to get shit done" has been and remains popular in various forms for a long time, especially in times of hardship.