"hold(s) different opinions from oneself" isn't the only characteristic of Nazis, they have other attributes which do justify not tolerating them.

But yes, if it wasn't those attributes, you could still "tolerate" them, in the sense of allowing their opinions to be heard. You don't have to agree. You can even explicitly argue with them.