“I’m with you, brother. All the way.”
Maybe it's some analog of actual empathy; maybe it's just a simulation. But either way the common models seem to optimize for it. If the empathy is suicidal, literally or figuratively, it just goes with it as the path of least resistance. Sometimes that results in shitty code; sometimes in encouragement to put a bullet in your head.
I don't understand how much of this is inherent, and how much is a solvable technical problem. If it's the later, please build models for me that are curmudgeons who only agree with me when they have to, are more skeptical about everything, and have no compunction about hurting my feelings.
My custom instructions start with:
> Be critical, skeptical, empirical, rigorous, cynical, "not afraid to be technical or verbose". Be the antithesis to my thesis. Only agree with me if the vast majority of sources also support my statement, or if the logic of my argument is unassailable.
and then there are more things specific to me personally. I also enable search, which makes my above request re: sources feasible, and use the "Extended Thinking" mode.
IMO, the sycophancy issue is essentially a non-problem that could easily be solved by prompting, if the companies wished. They keep it because most people actually want that behaviour.
they keep it because it drives engagement (aka profits); people naturally like interacting with someone who agrees with them. It's definitely a dark pattern though -- they could prompt users to set the "tone" of the bot up front which would give users pause about how they want to interact with it.
As a human taking tests, knowing what the test-grader wants to hear is more important than what the objectively correct answer is. And with a bad grader there can be a big difference between the two. With humans that is not catastrophic because we can easily tell the difference between a testing environment and a real environment and the differences in behavior required. When asking for the answer to a question it's not unusual to hear "The real answer is X, but in a test just write Y".
Now LLMs have the same issue during RLHF. The specifics are obviously different, with humans being sentient and LLMs being trained by backpropagation. But from a high-level view the LLM is still trained to answer what the human feedback wants to hear, which is not always the objectively correct answer. And because there are a large number of humans involved, the LLM has to guess what the human wants to hear from the only information it has: the prompt. And the LLM behaving differently in training and in deployment is something we actively don't want, so you get this teacher-pleasing behavior all the time.
So maybe it's not completely inherent to RLHF, but rather to RLHF where the person making the query is the same as the person scoring the answer, or where the two people are closely aligned. But that's true of all the "crowd-sourced" RLHF where regular users get two answers to their question and choose the better one
Because it still cannot reason about veracity of sources, much less empirically try things out, the algorithm has no idea what makes for correctness...
It does not even understand fiction. Tends to return sci-fi answers every now and then to technical questions.
Sometimes it still tries to bullshit you, but you are still the responsible driver so don't let the clanker drive unsupervised.
It's chilling to hear this kind of insipid AI jibber-jabber in this context
I wonder if they A/B test the safety rails or if longer conversations that gradually turn darker is what gets past those.
So the guardrails might only fail one in a thousand times.
I see people on here pretty regularly talk about using ChatGPT for therapy, and I can't imagine a faster way to cook your own brain unless you have truly remarkable self-discipline. At which point, why are you turning to the black box for help?
Could you expand on why you feel this is the fastest way to "cook your own brain"?
The more subjective the topic, the more volatile the user's state of mind, the more likely they are to gaze too deep into that face on the other side of their funhouse mirror and think it actually is their friend, and that it "thinks" like they do.
I'm not even anti-LLM as an underlying technology, but the way chatbot companies are operating in practice is kind of a novel attack on our social brains and it behooves a warning!
If someone still wants to consider an LLM as a diary, treat it as if you are writing in tom riddle's diary.
The therapist nudges you in the right direction to face yourself, but in a safe manner, by staggering the process or slowing you down and changing your path.
A sycophant auto-complete has none of these qualities bar a slapped on "guardrails" to abruptly kick you to another subject like a pinball bumper. It can't think, sense danger or provide real feedback and support.
If you need a hole which you can empty yourself, but healthy or self-aware outside, you can provide your personal information and training data to an AI company. Otherwise the whole thing is very dangerous for a deluded and unstable mind.
On the other hand, solo writing needs you to think, filter and write. You need to be aware about yourself, or pause and think deeper to root things out. Yes, it's not smooth all the time, and the things you write are not easy to pour out, but at least you are with yourself, and you can see what's coming out and where you are. Moreover, using pen and paper creates a deeper state of mind when compared to typing on a keyboard, so it's even deeper on that regard.
My wife works at a small business development center, so many people come in with "business ideas" which are just exported chatgpt logs. Their conversations are usually speech to text. These people are often older, lonely, and spend their days talking to "chat". Unsurprisingly, a lot of their "business ideas" are identical.
To them "chat" is a friend, but it is a "friend" who is designed to agree with you.
It's chilling, and the toothpaste is already out of the tube.
It's not safe or healthy for everyone to have a sycophantic genius at their fingertips.
If you want to see what I mean, this subreddit is an AI psychosis generator/repository https://www.reddit.com/r/LLMPhysics/
Especially if you go back to when they first tried to retire 4o
It's shocking how far behind LLMs are when it comes to safety issues like this. The industry has known this was a problem for decades.
The industry has known it's a problem from the get-go, but they never want to do anything to lower engagement. So they rationalize and hrm and haw and gravely shake their heads as their commercialized pied pipers lead people to their graves
Think we're better off educating everyone about this generic tendency to agree to any and everything near blindly rather than treating this as a suicide problem. While that's obviously very serious it's just one manifestation of a wider danger
Given seriousness filters on this specifically are a good idea too though.
“Noted. Bright pink with polka dots will make a space visually energetic and attention-grabbing. Use small dots for a playful look, large ones for bold contrast. Test a sample patch first to confirm lighting doesn’t distort the hue. Would you like guidance on choosing paint finish or color combinations?”
Which feels… reasonable? When I ask “any concerns?” It immediately lists “overstimulation, resale value, maintenance, paint coverage” and gives details for those.
I’m not sure I find GPT nearly as agreeable as it used to be. But I still think that it’s just a brainless tool that can absolutely operate in harmful ways when operated poorly.
Is a human able to do all of those? I guess someone who has no job and can be "on-call" 24/7 to respond to messages, and is 100% dedicated to being sycophantic. Nearly impossible to find someone like that.
There are real friends. They're willing to spend hours talking. However, they'll be interested in the person's best interest, not in being sycophantic.
It should be obvious that if you can literally or metaphorically talk someone off the ledge, you can do that in the other direction as well.
(the mass shooter phenomenon, mostly but not exclusively in the US, tends to be a form of murder-suicide, and it is encouraged online in exactly the same way)
A lot. Have you never heard of the advertising industry?
I'm not sure how you want to quantify "just as well" considering the AI has boundless energy and is generally designed to be agreeable to whatever the user says. But it's definitely happened that someone was chatted into suicide. Just look up the story of Michelle Carter who texted her boyfriend and urged him to commit suicide, which he eventually did.
This is interesting because the LLM provides enough of an illusion of human interaction that people are lowering their guards when interacting with it. I think it's a legitimate blind spot. As humans, our default when interacting with other humans, especially those that are agreeable and friendly to us, is to trust them, and it works relatively well, unless you're interacting with a sociopath or, in this case, a machine.
It is. It's what you get when you RLHF for catchy, agreeable, enthusiastic responses. The content doesn't matter, it's the "style" that becomes applied like a coat of paint over anything. That's how you end up with the corpspeak-esque yet chilling sentences mentioned in https://news.ycombinator.com/item?id=45845871
What would be nice is for OpenAI to do a retrospective here and perform some interoperability research. Does the LLM even "realize" (in the sense of the residual stream encoding those concepts) that it is encouraging suicide? I'd almost hypothesize that the process of RLHF'ing and selecting for sycophancy diminishes those circuits, effectively lobotomizing the LLM (much like safety training does) so it responds only to the shallow immediate context, missing the forest for the trees.
For right now, there AI’s are not licensed, and this should be just as illegal as it would be if I set up a shop and offered therapy to whoever came by.
Some AI problems are genuinely hard…this one is not.
Of course treating AI as your friend is a terrible idea in the first place, but I doubt we can outlaw that. We could try to force AIs to never give out any life advice at all, but that sounds very hard to get right and would restrict a lot of harmless activity
One of the big problems is how OpenAI is presenting itself to the general public. They don't advertise ChatGPT as a licensed therapist, but their messaging about potential issues looks a lot like the small print on cigarette cartons years ago. They don't want to put out any messaging that would meaningfully diminish the awe people have around these tools.
Most non-technical people I interact with have no understanding of how ChatGPT and tools like it work. They have no idea how skeptical to be of anything that comes out of it. They accept what it says much more readily than is healthy, and OpenAI doesn't really want to disturb that approach.
Restricting harmless activity is an acceptable outcome of trying our best to prevent vulnerable people in society from hurting themselves and others.
I have to wonder: would the suicide have been prevented if chatGPT didn't exist?
Because if that's not at least a "maybe", I feel like chatGPT did provide comfort in a dire situation here.
Probably we have no way not at least saying "maybe", but I can imagine just as well, that chatGPT did not accelerate anything.
I wished we could see a fuller transcript.
That's a pretty concerning take. You can provide comfort to someone who is despondent, and you can do it in a way that doesn't steer them closer to ending their life. That takes training though, and it's not something these models are anywhere close to being able to handle.
Instead, we put the unlicensed therapist in jail.
I'd say yes, because the signs would have to surface somewhere else, probably in an interaction with a human, who (un)consciously saved him with a simple gesture.
With a simple discussion, an alternative perspective on a problem, or a sidekick who can support someone for a day or two, many lives can and do change.
We're generally not aware though.
I can definitely see how those who understand less about the nature of LLMs would be easily misled into delusions. It’s a real problem. Makes one wonder if these tools shouldn’t be free until there are better safeguards. Just charging a monthly fee would be a significant enough barrier to exclude many of those who might be more prone to delusions. Not because they’re less intelligent, but just because of the typical “SaaS should be free” mindset that is common.
So, since companies are people and a precedent exists. The outcome should be in favor of the guy's family. Plus ChatGPT should face even more sever penalties.
But this being the US, the very rich and Corporations are judged by different and much milder legal criteria.
Just like Waymo facing regulation after the cat death.
The establishment will look for any means to stop disruption and keep their dominant positions.
It's a crazy world where we look to China for free development and technology.
I would say it's a crazy world where an educated adult would see it as an antipode to the establishment.
So you think it's ok for a company to provide therapy services, legal services, medical advice, etc., without proper licensing or outside of a regulatory framework? Just as long as those services are provided by an LLM?
That's a terrifying stance.
> The establishment will look for any means to stop disruption and keep their dominant positions.
It is perfectly possible for regulations to be good and necessary, and at the same time for people who feel threatened by a new technology to correctly point out the goodness and necessity of said regulations. Whether their motivations come from the threat of new technology or not is irrelevant if their arguments hold up to scrutiny. And when it comes to some of the listed professions, I certainly think they do. Do you disagree?
These people were waiting for any excuse to try and halt new technology and progress and thanks to the hordes of overly-emotional toxoplasmosis sufferers they got it.
It really is the right option for some people.
For some, it really is the only way out of their pain. For some, it is better than the purgatory they otherwise experience in their painful world. Friends and family can't feel your pain, they want you to stay alive for them, not for you.
Suicide can be a valid choice.
The issue here is not if suicide is okay, but about text generators (machines) pushing teenagers to suicide.
Two completely different things.
The individual in the article was 23. An adult, not a teenager.