My experience with OpenAI is that it is very good at exactly this, because it is so good at understanding context and follow-up questions. I was able to make it produce code that appeared correct, but was basically pseudocode with correct syntax, so it compiles/runs, but does essentially nothing. However, when prompted to actually make working code and explain how and why it works, it does so. And its also socially appropriate, not rude and what else you could/would expect when being called-out or corrected on its bullshit. I can only imagine future versions of the current AI model will be even better at this.

Yeah, absolutely. My position on AGIs for a long long time has been that they're great tools for a) generating insights into a large amount of data very quickly, and b) generating new instances of <thing> from examples of <thing> to help with exploring the possibility space of <thing>; but that any output or conclusion they generate _must_ be checked by a Human In The Loop, or at the very least their actions must be reversible without damage in the case of error.

Good point. It isn't like it is hard to find wrong answers on SO that were written by (hopefully) well meaning humans.