How many lines of code does your project use? How complicated is your module structure? Is everything in one huge target or do you split your app into 100s of modules? What do you use for dependency management, what is your deployment target? Are things running in one process like an iOS app or is it some sort of OS X project?
Pretty much every large swift project I've seen has had the problems described. Lyft, Uber, Linked In, etc.