surely this is meant partly tongue-in-cheek?
GPT is legitimately interesting as an alternative search interface to StackOverflow. I've found that 15 to 45 minutes of searching with Google/StackOverflow can be reduced to just 5 minutes with GPT.
But beyond that, it's been very disappointing. Whenever I've tried getting it to write something even slightly non-trivial (i.e. tougher than just copy/pasting from online documentation or an answer on StackOverflow) it's produced code that is horribly broken, but where the flaws are subtle enough that they might not pop out right away to a novice programmer. It has consistently struggled with programming problems that I would rate at 4/10 or 5/10 difficulty.
Most of the code I write is fairly trivial, but it's glue code that is highly specific to my particular code base, so GPT isn't helpful because it doesn't know about my codebase, and if you try to copy/paste your large chunks of your codebase into the prompt it runs into issues with forgetting.
And GPT isn't helpful for the non-trivial parts of my code either (as mentioned above). So what's left?
So when I see people say that it 10x'ed their productivity, I wonder if they exclusively write very trivial code that is effectively copy/paste from Stack Overflow or if they've allowed GPT to fill their codebases with flawed code without realizing it.
Maybe future iterations of GPT will get it. GPT4 is definitely not there yet.
I usually prompt it with short questions, but I recently saw a video where the person provided a lengthy (50 - 150 words) prompts detailing the requirements for what they were requesting. I was shocked at the results. (Still required iterations of corrections/modifications though)
I haven’t tried it myself yet, but it’s an avenue that might yield better results than what you’ve experienced — perhaps even vastly better results.
You can even prompt it to write a unit test for the function it wrote.
As for your concerns about code errors, I’ve found that you have to approach the coding support as an iterative process, where you request code, and then ask it to improve or correct the result it just gave. You can even prompt it to check its previous result for errors.
I do think you have a point about it being most useful for trivial or boilerplate problems but it’s still very useful as there is always much of that.
I'm currently writing a demo that I'll present next week using Jetpack Compose, and it's a UI toolkit I'm not really familiar with, so it's been really helpful for that. In fact, I have a tool that is almost like a build system that compiles English language spec files down to the code, and then allows me to edit them and to continue to work with the AI by just changing the spec and the code simultaneously. That's been really tremendously effective, especially with GPT-4.
On the other hand, for working on my main product, it's pretty useless because all of that work is debugging and making a lot of small changes all over the code base, which is too advanced for it currently. And I think that will get solved, but it isn't solved yet.
BTW the above paragraphs were dictated using the Whisper API. I didn't change a single thing about it. Whisper is just as impressive and useful as the LLMs, in my view.
I’m curious because it seems like perhaps providing longer, detailed prompts (and then iterating) might be the best approach for getting coding help.
Thoughts?
There must be a reason why he manages to have great people on his show, but it’s one I can’t comprehend.
