wwdmaxwell

This past weekend, I decided to test out a cli tool I've been building to help me do source code reviews _faster_.

I figured the best environment for such a tool would be a Weekend CTF event. I like web challenges since you get a nice dump of source code, as well as a Dockerfile or docker compose setup for how to run everything locally. Usually, I can complete 2-3 Web challenges before I get stuck. To help get unstuck I found myself increasingly turning to LLMs as a pairing partner.

I'm a fan of devcontainers, so I figured I could apply a similar concept with an agent*, where I load the agent into a container, mount the source code, and even start up any provided Dockerfile or docker-compose.yml so that the agent can actually test real `curl` commands!

So how did it go? It was pretty helpful for the web challenges. I was able to cruise through 5 between Friday and Saturday. I decided to see how it would do in the other categories - without any input / guidance from me as I typically stick to web.

In total *we* solved 19 challenges. It's best category was crypto with 4/7 solved, and it's worst was pwn with 2/5.

I was also curious how different providers would fair, because this was an automated agent, I started off using xai since they were the cheapest.

xai was able to solve 8 challenges autonomously with just source code and challenge descriptions.

I then pivoted to gemini as the next cheapest, and it did pretty well and was able to build on xai's "analysis" and solve 5 additional challenges.

I further tried to pivot to anthropic's Opus model, but it wasn't able to crack any additional challenges, and I got frustrated since I kept getting rate limited with 429 errors (so I kind of wish I switched to openai 5 instead, as it seems like Anthropic doesn't really like agents other than Claude calling their models.

In terms of cost breakdown I spent

$ 33.06 with xai

$ 35.61 with google

$ 24.04 with anthropic

Bringing the total just under $100 for a weekend benchmarking exercise.

Going forward I'm not really interested in paying to copy-paste CTF flags, but I did find the agent helpful for brainstorming solutions, and it worked a lot better when connected to the source code, with access to an instance running locally, and also augmented with MCP tools that allowed concept and source code searching. I'm planning to use similar concepts to build out a dev/review agent.

The source code for my setup is here: https://github.com/edelauna/prompt2pwn

* My initial version does require setting `--priveleged` on the Docker runtime. I originally tried to use podman, but I ran into networking / dns issues with how I wanted to make MCP tools available to the agent. Please open an issue on the repo and let me know if you have any ideas how to harden this.

GitHub semantic indexing seems locked to Copilot via the GitHub web interface. I’ve found it helps for the odd question here and there.

However I wanted that same codebase-awareness in Cursor, Claude Desktop, Cline, etc. — especially for private repos. So that I could stop cloning repos just to ask questions.

I created a remote MCP server that runs on Cloudflare Workers.

You can test it out via: ```json {

"mcpServers": {

  "gss": {

   "type": "http",

   "url": "https://github-search.lokeel.com/mcp",

   "headers": {

    "GITHUB_TOKEN": "<YOUR_TOKEN>"

   }

  }

 }

}

```

And you can ask it a question like > Use the gss mcp to lookup how I can implement a paginated datafetcher from @netflix/dgs-framework

I also find it really good for looking up test examples in repos to figure out how to properly use an sdk’s test utilities.

The MCP uses your GitHub access token to access repos, so you could ask questions about any repo the token can access.

I’ve also included instructions for how you could run your own instance to also own/manage all the data if preferred: https://github.com/edelauna/github-semantic-search-mcp/tree/...

There are still a bunch of edge cases that need to be addressed to productize something like this, but I just wanted to share as I’ve been finding it super helpful in my workflow and I’ve noticed the traffic on it has been growing.

Some next steps I was thinking were to try and create a deployable template and also figure out how to deploy this behind a vpn or in a private network, since Cloudflare is typically on the edge.

I know there’s a bunch of these tools already out there for public sources of info. I built this tool because I was getting frustrated by having to clone repos of libraries/APIs I'm using to be able to add them as context to the Cursor IDE (so that Cursor could use the most recent patterns). I would've preferred to just proxy GitHub search, but that seems to be limited to public repos via Copilot chat, and not available via graphql. This repo hosts a remote MCP server to facilitate a RAG query against an indexed GitHub repo. Indexing is performed via Cloudflare workflows, I’ve included info for how to deploy this in your own instance as I think having remotely accessible embeddings can be very useful for an organization.

[readme](https://github.com/edelauna/github-semantic-search-mcp/tree/...)