Maybe. If your build is too slow, fix it. But pre-emptively microoptimising your build time is as bad as pre-emptively microoptimising anything else. Set a build time budget and don't worry about it unless and until you see a risk of exceeding that budget.

This is an incredible exaggeration. The vast majority of these projects don't even approach it. I use clap in a number of projects and they compile in just seconds.