Personally I think a happy medium is to compile to C99. Then, after your own compiler's high-level syntax transformation pass, you can pass it through the Tiny C Compiler which is somewhere on the order of ~10x faster than Clang -O0. When you need performance optimizations at the cost of build speed, or to support a compilation target that TCC does not, you can freely switch to compiling with Clang, getting much of the value of LLVM without ever specifically targeting it. This is what I do for my own language, and it makes my life significantly easier and is perfectly sufficient for my use, since as with most languages my language will never be used by millions of people (or perhaps only ever one person, as I have not deigned to publish it).

I think writing a compiler targeting machine code from scratch only really makes sense if you have Google's resources, as Go did. That includes both the money and the talent pool of employees that can be assigned to work on the task full-time; not everyone has Ken Thompson lying around on payroll. To do better than LLVM is a herculean feat, and most languages will never be mainstream enough to justify the undertaking; indeed I think an undertaking of that scale would prevent a language from ever getting far enough along to attract users/contributors if it doesn't already have powerful backing from day 0.