For all the talk about "AI ethics" there is it seems strinking to me that the current state of the art model will opt for providing a convincing argument for why it's wrong answer is correct, rather than say that the answer it provides might be innacurate. (Funnily enough this is what humans tend to do aswell.) Being that these models often tend to be trained on data found on the internet, could this be a sign of the bias we tend to have when writing on social platforms and the internet in general? (As in justifying our answers instead of trying to get to the correct one)
So the questions are: 1. What are the consequences of this in the development of LLMs and in their application to various fields?
2. How would one implement this capability of recognizing where the model might be innacurate?
3. Is 2. really that much more complicated than what is currently being done? If not, then why hasn't it been done yet?
> AI: Answer that's clearly factually wrong (e.g. a function that doesn't exist or a completely wrong numerical figure).
> Me: that's not right, the function doesn't exist (etc.)
> AI: you are right, that function doesn't exist. The answer is blah
And repeat, since if it didn't get it right first time, it seems unlikely to be able to get there at all (and you'd not know when it did unless you already know the answer).
It's very like a specific person I know in a PR-marketing type job that will just glide across a noticed outright falsehood and instantly reshape what they're saying in real time and carry on as if nothing happened, leaving you wondering if you're taking crazy pills.
Mansplained by a robot...
Why not do make the function that outputs answers also feed itself "is this actually right/are you sure/is this not wrong"? Too expensive? Giant loop?
It can't tell it's wrong, it just reacts in a plausible well to you telling it it's wrong (acknowledging it and giving another explanation). Since ChatGPT is an expansion of GPT-3, those things won't be really solved since being somewhat of a knowledge base is a nice side effect, not the main goal of the model.
They do this, with humans. Both during training (they use supervised and reinforcement learning), and now at a much greater scale: it's what the free public access period is for and why there's a thumbs up/down button next to the output.
Then you reset your session, you ask the same question and you get the wrong answer again.
Those things are NOT artificial intelligence.
They are specific noise generators. They generate noise that is as similar as possible to noise it learned from, which match the input.
THats it. nothing more.
That said, the question of "why doesn't ChatGPT caveat every single response with a reminder that it might be inaccurate" doesn't seem like a meaningful or important discussion.
What's worse is that someone who isn't a domain expert might end up being convinced by the arguments and conviction the model provides.
Your second question is a good one, and I see that as a big problem with this generation of AI/ML. You're starting to scratch the surface of "this requires understanding of the real world" problems without models really understanding anything. It's all statistics and correlations in data, the model is not capable of really understanding its input or output.
These models are like very impressive Markov chain text generators in that they can spit out pretty convincing answers that seem cogent, but there is no real comprehension going on with what they read and write. It's just statistics.
GPT is the ultimate cosplayer. It pretends to be who you want it to be. If you want it to answer a question, it will make up something that looks like a valid answer to that question. Sometimes it may actually be the right answer (after all, the right answer has a pretty good chance to look like the right answer), but ultimately being right is not the goal it's trying to achieve - all it "wants" is to autocomplete your prompt in a plausible way.
Instead of making it try to come up with the answer, you can, for example, ask it (either explicitly or indirectly) to cosplay a scientist who is unsure of their position and tries to evaluate various options - and it will be happy to oblige.
What's worrying is not whether the model "acknowledges being wrong" or not - it's rather how it's being marketed to people and, in turn, what people expect from it. We've got plenty of submissions here on HN with people being surprised that the model has made up exactly what they asked it for - for example [0], which shouldn't happen if there wasn't a dissonance between what people think it's doing and what it's actually doing.
There seem to be a fair number of humans of whom I could say the same. As with the humans, the question is "which jobs am I happy to see this one doing and which ones worry me".
As GPT has endless ability to produce flowing words and precisely zero concept-comparable-to-will to verify them, unfortunately, the job for which it is most apt is probably politics. That I must admit worries me a little because of all the gpt-like humans we tolerate and encourage.
I've also seen it give different answers when asked the same question in a different way. Usually, one of the answers will be correct.
It has no idea if it's answers are right or wrong. It only knows that it's putting words in a common order.
Imagine putting two newspaper headings on the shelf at your local drugstore that happen to spell out an incorrect headline when combined. (Or put two browser windows together with a similar effect).
This is not wrong, your understanding of what it means is wrong. Sort your perspective out and you're fine.
1. LLMs are working as expected.
2. https://beta.openai.com/docs/guides/fine-tuning/case-study-i...
3. More complicated - probably not. It mostly just needs more data with specific training on domain specific areas. Take any field and generate some wrong completions from the data available and some right completions and the output tends to get better. Train it in the opposite fashion and it gets worse.
The meat of this comes back to people see ChatGPT as changing its response as an indication of understanding, but it's not. Users are just adding extra constraints on what parts of the language model the model uses to generate text. This is not the same thing.
An (perhaps poor) example of that might be where you're deciding to buy a tool that might solve a problem that you have. Evaluating whether it does solve the problem you decide that it doesn't. But then you evaluate whether you can afford the tool based on your budget and you decide you can, so you buy the tool. You haven't changed your mind, you've just constrained the reasons that you're using to evaluate the choice (electronic musicians often refer to this as GAS - Gear Acquisition Syndrome).
TL;DR: Stop Anthropomorphizing Language Models
This book
https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach
has a set of parables about people trying to paint on a facility to a system very similar to a "truth detector" for GPT-3. The gist of it is that "awareness of truth" makes it possible to make statements like "Am I lying now?"
People under GPT-3's spell think that giving correct answers is a minor detail that will be handled in a point revision of it but actually it is a much harder problem than everything they've done so far.
Impressive as it is, this kind of AI seems to be still working under what seems to me to be a possibly-flawed premise: training quantity has a sufficient quality of its own.
I can't prove that it's impossible with a clever enough system, but I simply don't see how you can get a right answer to come out of a statistical system that's been trained in input that might contain incorrect information, conflicting versions of it or just nothing at all, in which case it just makes something statistically plausible.
For example, it can give quite good answers about a well known event (e.g. a big earthquake), presumably because there are enough mentions of it in the training data. Ask about a footnote earthquake with few mentions and it will invent details that could be right, but aren't. For example a magnitude in the single digits "seems about right" and passes a sniff test, but has no factual basis in reality.
That said, I wonder if welding a large structured data store like Wolfram Alpha or Wikidata to the language model might resolve that issue: don't rely on statistics when the answer exists.
It's not unlike the book's approach of Godel numbering strings to as consistency or completeness of formal grammars, and indeed some ChatGPT conversations recapitulate the humorous dialogs between Achilles and the tortoise. Indeed, I've been able to walk through opposing takes on the validity of Searle's Chinese Room metaphor (which, like Hofstadter, I don't subscribe to) and get the LLM subject its own defaults to the same analysis.
I'm unsure to what degree this is fine-tuning the model vs merely equipping it with a decorative frame. In any sufficiently deep conversation, ChatGPT seems to drift toward imitation of its interlocutor, though I don't know if this emergent or by design. I suspect one could persuade it to agree that it should be stubborn in defense of the truth, and then gaslight it by denying one's own former statements.
I don't want to try this for the same reason I don't like to tease animals, but the model can be brought to reject its own priors on the basis of other priors, and to ask questions and solicit information in pursuit of a goal, even putting up mild resistance to changes of subject. A few hours of interaction can yield tantalizing glimmerings of agency.
