No comments yet.
This is Arjun and Saikat, and like other product engineers, we've been excited to build with LLMs. Getting powerful models available as off-the-shelf HTTP endpoints is a huge leap forward to integrate and ship ML to end-users.
While building on top of LLMs, we've also experienced the pain of non-deterministic behavior – especially for applications that require smaller models. Iterating through model configuration while ensuring no regressions across hundreds of scenarios is a tricky balance.
To make this easier, we built Empirical. Here’s a demo video: https://www.youtube.com/watch?v=p8gSGphcOSU
We've focused on:
- Fast iteration cycles and interactivity when you need to change the prompt or add a new sample. We wanted to build something that feels like “hot reload” for LLM development
- A capable UI that combines objective and subjective evaluation, since eye-balling outputs makes it easier to build intuition around model behavior
- Ability to customize which model to test, or how to score it — with JavaScript (or Python, if you really must)
- Embedded analytics for evaluation results, powered by DuckDB under the hood (more coming up on this!)
You can try Empirical today – with a one line CLI command – locally or on CI/CD. And oh, Empirical is 100% open source – so file an issue and we’d be happy to make it work for your use-case
$ npx empiricalrun
