I'm sure you're familiar with the arguments. It's the explicit goal of several AI companies to make something more capable than us at engineering. Once you have that, there's the possibility of very rapid self-reinforcing improvement. If you lose control of something like that, it's game over.
GPT4 may not generate world class code, but it does it at a scale and speed unmatched by humanity. Alpha Zero took a week to go from nothing to better than any human in history at Go.
