I really depends on how much you care about undefined behaviour. If you're just playing around with a prototype or making a personal tool, corrupt state doesn't have huge ramifications, but for any software people would pay for (be that with money or attention), there's an expectation of a minimum level of robustness, security, performance, and a variety of other business requirements.

To achieve the same with an AI that doesn't have a real understanding of the business logic, programmers would still be needed to write the test suite. But unlike most test suites that are typically underspecified, the test suite would need to be likely more complicated than the program itself. You could use ChatGPT to expedite writing of the test programs, but attention would still be required to actually verify the tests themselves.

> Let’s say it can simulate theory of computation better than 99% of the population and can very capably synthesize and infer from any text based sources. I think that would shake the world, and it wouldn’t even need to be near AGI.

The total percentage isn't exactly what matters. Emergent properties as a metric is a smokescreen.

If that last 1% incorrectly demonstrates that A<C<B implies A<B<C, that means the system is not reliable enough to perform logical computations. You'd need to have a person oversee 100% to catch the last 1% of serious but basic errors. In such a case you might as well hire the person directly for the job.

May be it's because a lot of code out there doesn't actually require logic but merely requires copying what is already on the internet? I mean, I don't need to tell you that.

A computer can sort an array of numbers faster than likely 99% of the population, it doesn't mean it's useful.

Professional developers make up less than 1% of the global population (around 25 million people apparently). Assuming they are the ones that have a good theory of computation, I think “better than 99%” is not a high enough bar to shake the world, right?

I mean, it depends on what we expect the AI to do. Maybe it would be revolutionary to just have, like, an average programmer with a ton of free time (so, the AI only has to beat like 99.7% of humanity to do that). On the other hand, if we want it to change the world by being much better than the average person, I guess we’d need a couple more 9’s.

The AP test is about potential, though. You don't hire a high schooler to work as a developer based on their AP CS score.