I always thought it’s good practice in C/C++ to have only one translation unit in release builds e.g. SQLite amalgamations, instead of relying on LTO. It also speeds up compilation because it isn’t recompiling the same header files over and over again.
And while it helps with compile time (especially with mediocre compilers like MSVC where a large time is spent in the front-end), it doesn't help with stripping unneeded code/data compared to LTO.
It does significantly speed up clean builds in many cases, to the point that those clean builds can even be faster than many "incremental" builds in some cases.
I do not know of any compiler that can compile different parts of a translation unit in parallel. But plenty of systems (including the venerable `make -j`) can compile separate translation units in parallel because they are independent.
You don't want to clear caches for release builds. That just makes release builds unnecessarily slow and impedes the flow of Continuous Deployment. You just need to have separate caches for different build flags, including a dedicated cache for release builds.