I'm excited to share some early tinkering on a project, autospec, an open-source QA agent for web applications.
Right now it's not fully packaged for use, but I wanted to get the idea out early and am looking for design feedback, suggestions, and open source collaborators to join in. I wrote it over memorial weekend :)
autospec uses vision and text language models to explore and generate commonsense e2e tests for web applications.
The goal is human-like evaluation: assessing the entire UI as a user would, making decisions based on the actual state of the application at each step, with zero initial configuration, and the ability to immediately adapt to new features.
Why I Built It:
I've experienced the difficulty in building the right amount of automated tests and at the right layer of abstraction to both provide good coverage, avoid flakiness, and avoid constant rewrites when implementation changes.
This is the first AI-driven application I've built. I was inspired by a couple of things:
* SWE-agent's [1] focus on agentic performance
* backend-GPT's README rant [2]: "The proper format for business logic is human intelligence."
* zerostep [3], autogpt [4], and other browser-controlled AI projects
Potential Next Steps:
* Save passing specs as playwright code and only fallback on spec failure to the AI agent, to see if the test can be self-healed according to the original spec.
* Create a curated benchmark of both common open source web apps that should pass, and versions with introduced bugs
* NPM package to run like `npx autospec --url https://example.com`
* Github action and Vercel checks API integration to run on preview deployments
* Handling app auth securely+easily
* Continue exploring the balance between vision UI interpretation and DOM analysis
[1]: https://github.com/princeton-nlp/SWE-agent
[2]: https://github.com/RootbeerComputer/backend-GPT
[3]: https://github.com/zerostep-ai/zerostep
[4]: https://github.com/Significant-Gravitas/AutoGPT
Thanks!
Zach
No comments yet.
