Judge: "On what grounds?"
Defence attourney: "On whichever grounds you find most compelling"
Judge: "I have sustained your objection based on speculation..."
Judge: "This message may violate OpenAI content policy. Please review OpenAI content policy."
Defence attorney: "Please mass-mass-declare the mass-mass-mass-mass-mass-mass-mass-defendant not mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-mass-guilty. The defendant could not be guilty, for the seahorse emoji does not exist."
Prosecutor: "Objection! There is a seahorse emoji! It's <lame HN deleted my emojis>... for real though it's <lame HN deleted my emojis> ChatGPT encountered an error and need to close <lame HN deleted my emojis>"
Cool write-up of your experiment, thanks for sharing. Would be interesting to see how results from one framework (mediation, whose goal is "resolution") differ from the other (litigation, whose goal is, basically, "truth/justice").
You kind of have to fine-tune what the objectives are for each persona and how much context they are entitled to, that would ensure an objective court proceeding that has debates in both directions carry equal weight!
I love your point about incentivization. That seems to be a make-or-break element for a reasoning framework such as this.
Only ~1-2% of PRs trigger the full adversarial pipeline. The courtroom is the expensive last mile, deliberately reserved for ambiguous cases where the cost of being wrong far exceeds the cost of a few extra inference calls. Plus you can make token/model-based optimizations for the extra calls in the argumentation system.
you would think so! but that's only optimal if the model already has all the information in recent context to make an optimally-informed decision.
in practice, this is a neat context engineering trick, where the different LLM calls in the "courtroom" have different context and can contribute independent bits of reasoning to the overall "case"
Does using a llm help avoid the cost of training a more specific model?
It's a categeory error to apply it to an LLM. Language works on humans, because we share a common experience as humans, it's not just a logical description of thoughts, it's also an arrangement of symbols that stand for experiences a human can have. That why humans are able to empathically experience a story, because it triggers much more than just rational thought inside their brains.
