Because everything is a tradeoff. Getting the binary to be smaller means taking more compile time, because compilers have to do work to reduce the size. It is much faster to produce larger binaries.
Additionally, most compilers produce something that's useful for development and debugging by default, and that means including extra stuff for that purpose.