That does just do CMake time. Sorry if my comment was confusing. I forgot that I used a wrapper or something myself to collect the compile times. Anyway, my point is, you aren't the first person to have this issue. I've definitely profiled compile times before and viewed a flame graph with some gadget made by Google. The easiest way you to do it may depend on your compiler and build system. You could attack the problem from Ninja or Make, or by substituting your compiler binary for a wrapper script.
I also want to point out that if you want Ninja, it is a very minimal binary to build with no dependencies that wouldn't already be installed.
This is one of the simpler ways to find out what's going on: https://alangrow.com/blog/profiling-every-command-in-a-makef... Instead of replacing the shell you can also replace CC or CXX variables (if I recall properly) either in CMake or the Makefile to replace your compiler with a wrapper script that logs stuff.
More generalized Makefile profiling: https://github.com/konturio/make-profiler
Profiling with Clang: https://aras-p.info/blog/2019/01/16/time-trace-timeline-flam... (Maybe what I used last time...)
I am pretty sure you can get shiny outputs from a free profiling tool, but I think you can search for that yourself. It should be easy to find something. Good luck!