> That’s the defense. And here’s the problem: it’s often hard to refute with confidence.
Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
It's negligence, pure and simple. The only reason we're having this discussion is that a trillion dollars was spent writing said scripts.
Engineers are accountable for the actions they authorize. Simple as that. The agent can do nothing unless the engineer says it can. If the engineer doesn't feel they have control over what the agent can or cannot do, under no circumstances should it be authorized. To do so would be alarmingly negligent.
This extends to products. If I buy a product from a vendor and that product behaves in an unexpected and harmful manner, I expect that vendor to own it. I don't expect error-free work, yet nevertheless "our AI behaved unexpectedly" is not a deflection, nor is it satisfactory when presented as a root cause.
There was a notorious dark web bot case where someone created a bot that autonomously went onto the dark web and purchased numerous illicit items.
https://wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww.bitnik.or...
They bought some ecstasy, a hungarian passport, and random other items from Agora.
>The day after they took down the exhibition showcasing the items their bot had bought, the Swiss police “arrested” the robot, seized the computer, and confiscated the items it had purchased. “It seems, the purpose of the confiscation is to impede an endangerment of third parties through the drugs exhibited, by destroying them,” someone from !Mediengruppe Bitnik wrote on their blog.
In April, however, the bot was released along with everything it had purchased, except the ecstasy, and the artists were cleared of any wrongdoing. But the arrest had many wondering just where the line gets drawn between human and computer culpability.
If you gave a loaded gun to a five year old, would "five-year-old did it" be a valid excuse?
Yeah that's exactly the I think we should adopt for AI agent tool calls as well: cryptographically signed, task scoped "warrants" that can be traceable even in cases of multi-agent delegation chains
I think there is no distinction between the two for liability purposes.
Courts of law have already found that AI interactions with customers are binding, even if said interactions are considered "bloopers" by the vendor[1]
[1] https://www.forbes.com/sites/marisagarcia/2024/02/19/what-ai...
The workflow of starting dozens or hundreds of "agents" that work autonomously is starting to gain traction. The goal of people who work like this is to completely automate software development. At some point they want to be able to give the tool an arbitrary task, presumably one that benefits them in some way, and have it build, deploy, and use software to complete it. When millions of people are doing this, and the layers of indirection grow in complexity, how do you trace the result back to a human? Can we say that a human was really responsible for it?
Maybe this seems simple today, but the challenges this technology forces on society are numerous, and we're far from ready for it.
When orchestrators spawn sub-agents spawn tools, there's no artifact showing how authority flowed through the chain.
Warrants are a primitive for this: signed authorization that attenuates at each hop. Each delegation is signed, scope can only narrow, and the full chain is verifiable at the end. Doesn't matter how many layers deep.
People want to use a tool and not be liable for the result.
People not wanting to be liable for their actions is not new. AI hasn't changed anything here, it's just a new lame excuse.
AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law.
The software world has been very allergic to getting anywhere near the vicinity of a system like that.
AI is worse in that regard, because, although you can't explain why it does so, you can point a finger at it, say "we told you so" and provide the receipts of repeated warnings that the thing has a tendency of doing the things.
The new and interesting part is that while we have incentives and deterrents to keep our human agents doing the right thing, there isn't really an analog to check the non-human agent. We don't have robot prison yet.
If the user claims that they only authorized the bot to review files, but they've warranted the bot to both scan every file and also send emails to outside sources, the competitors in this case, then you now have proof that the user was planning on committing corporate espionage.
To use a more sane version of an example below, if your dog runs outside the house and mauls a child, you are obviously guilty of negligence, but if there's proof of you unleashing the dog and ordering the attack, you're guilty of murder.
> It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
| *Who authorized this class of action, for which agent identity, under what constraints, for how long; and how did that authority flow?*
| A common failure mode in agent incidents is not “we don’t know what happened,” but:
| > We can’t produce a crisp artifact showing that a specific human explicitly authorized the scope that made this action possible.
The point is "explicitly authorized", as the article emphasizes. It's easy to find who ran the agent(article assumes they have OAuth log). This article is about 'Everyone knows who did it, but did they do it on purpose? Our system can figure it out'
And when sub-agents or third-party tools are involved, liability gets even murkier. Who's accountable when the action executed three hops away from the human? The article argues for receipts that make "I didn't authorize that" a verifiable claim
If you don't want to be responsible for what a tool that might do anything at all might do, don't use the tool.
The other option is admitting that you don't accept responsibility, not looking for a way to be "responsible" but not accountable.
No, it's trivial: "So you admit you uploaded confidential information to the unpredictable tool with wide capabilities?"
> Who's accountable when the action executed three hops away from the human?
The human is accountable.
If one decided to paint a school's interior with toxic paint, it's not "the paint poisoned them on its own", it's "someone chose to use a paint that can poison people".
Somebody was responsible for choosing to use a tool that has this class of risks and explicitly did not follow known and established protocol for securing against such risk. Consequences are that person's to bear - otherwise the concept of responsibility loses all value.
It really doesn't. That falls straight on Governance, Risk, and Compliance. Ultimately, CISO, CFO, CEO are in the line of fire.
The article's argument happens in a vacuum of facts. The fact that a security engineer doesn't know that is depressing, but not surprising.
[ $[ $RANDOM % 6] = 0 ] && rm -rf / || echo "Click"
"Three months later [...] But the prompt history? Deleted. The original instruction? The analyst’s word against the logs."
One, the analysts word does not override the logs, that's the point of logs. Two, it's fairly clear the author of the fine article has never worked close to finance. A three month retention period for AI queries by an analyst is not an option.
SEC Rule 17a-4 & FINRA Rule 4511 have entered the chat.
With no amount of detailed logging makes "the AI did it" a valid excuse.
It's just a tool.
It's like blaming a loose bolt in a Boeing 737 on "screwdriver did it".
On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.
The "Hallucination Defense" isn't a defense because end of the day, if you ran it, you're responsible, IMO.
And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.
1. I vaguely recall that in the early days of Windows, the TOS explicitly told users they were not to use it for certain use-cases and the wording was something like "we don't warranty windows to be good for anything, but we EXPLICITLY do not want you to use Windows to do nuclear foo".
I expect that if the big LLM vendors aren't already doing this, they soon will. Kind of like how Fox News claims that some of heads on the screen are not journalists at all but merely entertainers (and you wouldn't take anything they say seriously ergo we're not responsible for stuff that happens as a result of people listening to our heads on the screen).
2. IANAL but I believe that in most legal systems, responsibility requires agency. Leaving aside that we call these things "agents" (which is another thing I suspect will change in the marketing), they do not have agency. As a result they must be considered tools. Tools can be used according to the instructions/TOS, or not. If not - whatever it does is down to you. If used within guidelines - it's the manufacturer.
So my conclusion is that the vendors - who have made EPIC bets on getting this tech into the hands of as many folks as possible and making it useful for pretty much anything you can think of - will be faced with a dilemma. At the moment, it seems like they believe that (just like the infamous Ford bean counters) the benefits of not restricting the TOS will far outweigh any consequences from bad things happening. Remains to be seen.
Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider.
Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users.
Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!!
And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business
One in which you sell yourself into slavery, for example, would be illegal in the US.
All those "we take no responsibility for the [valet parking|rocks falling off our truck|exploding bottles]" disclaimers are largely attempts to dissuade people from trying.
As an example, NY bans liability waivers at paid pools, gyms, etc. The gym will still have you sign one! But they have no enforcement teeth beyond people assuming they're valid. https://codes.findlaw.com/ny/general-obligations-law/gob-sec...
"But the AI wrote the bug."
Who cares? It could be you, your relative, your boss, your underling, your counterpart in India, ... Your company provided some reasonable guarantee of service (whether explitly enumerated in a contact or not) and you cannot just blindly pass the buck.
Sure, after you've settled your claim with the user, maybe TRY to go after the upstream provider, but good luck.
(Extreme example) -- If your company produces a pacemaker dependent on AWS/GCP/... and everyone dies as soon as cloudflare has a routing outage that cascades to your provider, oh boy YOU are fucked, not cloudflare or your hosting provider.
Or perhaps scapegoating at scale.
https://news.ycombinator.com/item?id=43877301 - 398 comments
https://news.ycombinator.com/item?id=41891694 - 308 comments
It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.
Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses.
However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output.
And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).
Use AI if being 80% right quickly is fine. Otherwise if you have to do the analysis anyway because accuracy is critical, there's little point to the AI - its too unreliable.
1. A person orders an AI agent to do A.
2. The agent issues a tenuo warrant for doing A.
3. The agent can now only use the tool to perform A.
The article is about that 'warrant' can now be used in case of an incident because it contains information such as 'who ordered the task' and 'what authority was given'.
I get the idea. This isn't about whether a person is responsible or not(because of course they are). It's more about whether it was intentional.
However... wouldn't it be much easier to just save the prompt log? This article is based entirely on "But the prompt history? Deleted."(from the article) situation.
But warrants aren't just "more audit data." They're an authorization primitive enforced in the critical path: scope and constraints are checked mechanically before the action executes. The receipt is a byproduct.
Prompt logs tell you what the model claimed it was doing. A warrant is what the human actually authorized, bound to an agent key, verifiable without trusting the agent runtime.
This matters more in multi-agent systems. When Agent A delegates to Agent B, which calls a tool, you want to be able to link that action back to the human who started it. Warrants chain cryptographically. Each hop signs and attenuates. The authorization provenance is in the artifact itself.
Eg a list of transactions that isn't AI generated where the only actions that actually move money must operate on the data displayed in the human designed page.
A human looks at this and says yes that is acceptable and becomes reasonable for that action.
So while that should happen, it won't. They'll just add an extra layer of AI to do the verification.
This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken
> A computer must never make a management decision, because a computer cannot be held accountable.
AI is just branding. At the end of the day it's still just people using computer software to do stuff.
There is going to be a person who did a thing at the end of the day -- either whoever wrote the software or whoever used the tool.
The fact that software inexplicably got unreliable when we started stamping "AI" on the box shouldn't really change anything.
but the person who turned it on can
simple as