Understanding what you are optimizing FOR and where the most attention should be spent is the crux of Knuth's argument. Trying to be clever up-front is often counter-productive.
There is nothing wrong with making some architectural decisions up front, but that is much different than avoiding pImpls at all costs because indirection is slower. Indirection doesn't always matter, and it should only be tackled when and where it does.
Given that it makes every object instantiation perform a memory allocation, followed by required indirection to accesss it, and will also prevent the creation of default copy constructor and assignment operator etc. due to use of unique_ptr, it adds complexity as well as two sources of performance loss.
As a result, I would use this pattern only where strictly necessary. For example, I've used it when wrapping C libraries with C++ classes. It means the C headers aren't included in the public C++ headers, only the implementation, making the implementation cleaner. Or I might use it in situations where private members drag in dozens of headers which expose a lot of unnecessary implementation details to the user, which might drag in transitive library dependencies simply by #including the headers. The fact that the compilation speed might be marginally faster is incidental to this.
HTH
However, if compilation times have gotten painful enough that we need to examine performance improvements to our headers, the pImpl pattern is one of many tools in the toolbox. So are forward headers and other compiler firewall techniques.
