Refactoring is a bit different in my mind. I often discover better ways to write a piece of functionality while I am writing it. Sometimes it's worth essentially starting over, and sometimes it's not. That decision IMO should be heavily influenced by business deadlines. There are times to push back the deadline because of a major refactor, but they damn well better be a rare exception, and you (and your manager) damn well better be able to explain the business reasoning behind it.
Lumping poorly judged refractors, last minute pet features, and optimizations (actual gold plating) in with critical technical practices like testing and review is probably part of the problem in your organization..