The output of the verilog optimizer is different every time. The output of a fab is different in every batch. Each chip in a batch is different from others in that batch. Quality control drops the fraction of truly poor chips, and hardware design features might downgrade some of the partially failed chips to be classified as lesser versions of the same initial design. The final chips work as intended, mostly, but perhaps the error tolerance to overclocking or the mean time between failures is slightly different between chips. We can all work with them just fine almost all the time. The same principles apply to complex LLM-orchestrated code projects. I dont mind if my compiler gives different code each time because it uses a stochastic optimizer, but I want my code to do what I want and to not fail more than a certain tolerance I have for this code, which depends on the application. By giving more insight into the layers of testing to more people, and by encouraging the new documentation practices that Andrej mentioned, LLM coding will change the practice of software engineering rather dramatically. Code 2.0 was flexible and could yield results that were better than human coded efforts for complex problems, but the architecture, code, data, were selected by humans. In code 3.0 humans have access to (non-deterministic) building blocks that are written in natural language, to bug fixes and feature addition that happen in a conversation style. Similar engineering principles as with code 1.0 still apply (even more so than with code2.0, unless the product is a neural net), but the emphasis on verification increased dramatically as a fraction of the total effort, even though the total effort has gone down a lot. I can’t wait to see increased help in code verification efforts from this batch of people in the AI startup school as a result of Andrej’s presentation.