The reason that Yosemite/Mavericks stall for a long time (although they inform users wrongly that there would be few minutes left, causing panic) is due to the copying and moving things in /usr/local/. If you have a huge /usr/local, it will take hours. Chances are you will need to remove /usr/local/Cellars and reinstall home-brew anyway. So, removing it first will save you some time.