I think there's a vast ocean of different software engineers and the type of code they write on a daily basis, which is why you get such differing views on AI's effectiveness. For me, AI has only ever been useful for very basic tasks and scripts: If I need a quick helper script for something in Python, a language that isn't my daily drier, AI usually gets me there in a couple of prompts. Or maybe I am writing some powershell/bash and forgot the syntax for something and AI is quicker (or has more context) than a web search.

However, my main job is trying to come up with "elegant" architectures for complex business logic that interacts with an existing large code base. AI is just completely out of its depth in such cases due to lack of context but also lack of source material to draw from. Even unit tests only work with the most basic of cases, most of the time the setup is so complex it just produces garbage.

I've also had very little luck getting it to write performant code. I almost have to feed it the techniques or algorithms before it attempts to write such code, and even then it's usually wrong or not as efficient as it could be.

I have the same experience. I have my own benchmark, I take a relatively complex project like FreeSWITCH on github, which is part of the training set for all coding LLMs anyway so they should know it and I ask the AI to code small snippets, tests, suggestions and fixes to see how well it understands the codebase and the architecture.

I just tried the latest Cursor + Sonnet and it failed in every task. The problem is that there is no way to understand the code without either complete understanding of the domain and the intents or running it in some context.

Telecom and media domains in particular are well documented in specs and studied in forum discussions. I am sure they are part of the training data, because I can get most answers if asked directly. So far the LLMs fail to reason about anything useful for me.

Having spent a couple of hours with some llama models and others I've given up on them for code. Code generation from XML or grinding it out ad hoc with stuff like sed on top of a data or config file is faster and more convenient for me.

The Jetbrains thing is rather rudely incompetent, it consistently insists on suggestions that use variables and fragments that are supposed to be replaced by what I'm writing and also much more complex than what I actually need. I suffered through at least a hundred mistaken tabbed out shitty suggestions before I disconnected it.

I don’t use ChatGPT or chat AI for coding, at least not often. It does help me get unstuck on occasion, but most of the time the context switch is too much of a productivity sink.

However, I use copilot autocomplete consistently. That makes me much more productive. It’s just really good autocomplete.

As an experienced developer I’d put my productivity improvement at 2-3x. It’s huge but not 10x. I’m limited by my decision speed, I need to decide what I want the code to do, AI can’t help with that - it can only do the “how”.

Far from introducing more bugs, using Copilot frees some mental cycles for me to be more aware of the code I’m writing.

It's going to be the same issue as poorly-structured Excel workflows: non-programmers doing programming. Excel itself gets the blame for the buggy poorly-structured spreadsheets made by people who don't have the "programmer mindset" of thinking of the edge cases, exceptions, and extensibility.

So it will be with inexperienced people coding with LLMs.

Absolutely the same for me. Whenever I encounter a problem I'm pretty sure nobody else has encountered before (or has not written about it at least), ChatGPT writes complete nonsense.

On stuff that's a bit obscure, but not really (like your Vulkan example), ChatGPT tends to write 60-95% correct code, that's flawed in the exact ways a noob wouldn't be able to fix.

In this case, nicking code from Github seems to fix the issue, even if I need to adapt it a bit.

Then comes the licensing issue. Often, when searching for an obscure topic, the code ChatGPT generates is very close to what's found on Github, but said code often comes with a non-permissive license, unlike what the AI generates.