draismaa

In traditional software, we write unit tests to catch regressions before they reach users. In AI systems—especially agentic ones that model breaks down. You can test inputs and outputs, use evals, but agents operate over time, across tools, mcps, apis, and unpredictable user input. The failure modes are non-obvious and often emerge only in edge cases. I'm seeing an emerging practice: agent simulations—structured, repeatable scenarios that test how an AI agent behaves in complex or long-tail situations.

Think: What if the upstream tool fails mid-execution? What if the user flips intent mid-dialogue? What if the agent’s assumptions were subtly wrong?

from self-driving cars to AI agents? The above aren’t one-off tests. They’re like AV simulations: controlled environments to explore failure boundaries. Autonomous vehicle teams learned long ago that real-world data isn't enough. The rarest events are the most important—and you need to generate and replay them systematically. That same long-tail distribution applies to LLM agents. We’ve started treating scenario testing as a core part of the dev loop—versioning simulations, running them in CI, and evolving them as our agent behavior changes. It’s not about perfect coverage,it’s about shifting from “test after” to “test through simulation” as part of iterative agent development. Curious if others here are doing something similar. How are you testing your agents beyond a few prompts and metrics? Would love to hear how the HN crowd is thinking about agent reliability and safety—not just in research, but in real-world deployments.

We've been working with teams building AI agents (agentic systems, with actual execution) But here's the thing: everyone says “agents are the future,” yet no one really knows how to test them. Some teams are manually walking through conversations, others are just shipping and "vibe checking" what comes back. Both break down at scale. The real problem? We’re testing agents like software, but agents don’t behave like software. They make decisions, adapt, escalate, reason across contexts. They're more like processes than functions. Rogerio, our CTO, wrote up a deeper dive on how we see the future of agent testing, and why agent simulations (not hardcoded flows) are becoming the new unit tests for AI systems. We built LangWatch scenario to let teams simulate real-world agent behavior and catch regressions early on. Would love feedback from folks who’ve been burned by this or hacked together their own simulation setups.