Oh-my-zsh is responsible for being slow as hell. If you really want plugins, Prezto probably still has what you want and is significantly faster on startup. (In my experience, with very minimal customization.)