The phrase conjures up images of tight code that executes the minimum number of instructions with the least possible memory use and efficient reads and writes. In a piece of code that's executed maybe once in blue moon. Nobody bothered to profile the program and find out where the real hotspots are, so someone optimized the wrong thing.

Writing code designed to be flexible according to how the programmer believes it will need to change is also a kind of premature optimization. The code is written in the optimal way for changes along direction X to be easy. Unfortunately, when the programmer guesses wrong about the nature of future change, that work to optimize for change along X is wasted, or worse. Worse, if it makes changes along direction Y much harder.

We're pretty bad at predicting the future.