We previously tried to build an AI junior developer that writes GitHub PRs (https://news.ycombinator.com/item?id=36987454). It was fun, but we ultimately decided to pivot. Here are a couple reasons it didn’t work:
1. Our agent really needed a well defined spec to have a >90% success rate on tasks. Developers are lazy (myself included) when describing tasks and agents weren’t good enough to make up for it. We found developers don’t want to write a spec, they want to see the agent try and iterate with it. Fixing this is hard! The flow needs to be fast otherwise people get distracted and go scroll HN or check slack.
2. Executing code is challenging, especially for production apps. Github actions were too slow to use as a code execution sandbox, and emulating the developer’s environment in Docker or what-have-you was not feasible. Agents weren't ready for real codebases because their CI wasn't built with agents in mind.
We looked around for a better UX than GitHub issues, and we noticed that JetBrains developers were consistently unhappy with GitHub Copilot. Cursor and Windsurf (the current market leaders) only supported VSCode.
There were other good options but none were Cursor-quality. We asked ourselves “why not?” and decided to investigate. I spoke with an ex-JetBrains employee who said: “The best AI developers don’t really use Java, and the best Java developers tend to work in enterprise companies rather than startups.”
So we decided to take our experience in building an AI agent and go for JetBrains. Here’s what we’ve learned so far:
- The latest open-source models like Qwen are really good. Some use cases like applying code to a file work decently with these models out of the box, so we don’t have to do as much 0 → 1 R&D to build a great product. This doesn’t make it easy, but it does mean a small team that really cares can deliver a great product.
- Most agents are still while-loop wrappers. We tried that with the last generation of models and found it to slow. Instead we’re trying a different approach that relies more on the code graph to see neighboring files. It really decreases the latency because we can rely on the user’s current file and skip the “grep in a loop” section. It also costs a bit more, but this hasn’t been a problem yet.
- Big company solutions for JetBrains feel like box-ticking and are behind their VSCode equivalents. I’ve seen many complaints that GH Copilot uses a couple of gigs of RAM (which really sucks when your IDE already takes 3+ gigs of RAM.
Let me know what you think! I’m also curious how/if you’re using agents. I usually get impatient waiting for the LLM.
