Every article on hallucinations needs to start with this fact until we've hammered that into every "AI Engineer"'s head. Hallucinations are not a bug—they're not a different mode of operation, they're not a logic error. They're not even really a distinct kind of output.
What they are is a value judgement we assign to the output of an LLM program. A "hallucination" is just output from an LLM-based workflow that is not fit for purpose.
This means that all techniques for managing hallucinations (such as the ones described in TFA, which are good) are better understood as techniques for constraining and validating the probabilistic output of an LLM to ensure fitness for purpose—it's a process of quality control, and it should be approached as such. The trouble is that we software engineers have spent so long working in an artificially deterministic world that we're not used to designing and evaluating probabilistic quality control systems for computer output.
[0] They link to this paper: https://arxiv.org/pdf/2401.11817
I think that's a mischaracterization and not really accurate. As a trade, we're familiar with probabilistic/non-deterministic components and how to approach them.
You were closer when you used quotes around "AI Engineer" -- many of the loudest people involved in generative AI right now have little to no grounding in engineering at all. They aren't used to looking at their work through "fit for purpose" concerns, compromises, efficiency, limits, constraints, etc -- whether that work uses AI or not.
The rest of us are variously either working quietly, getting drowned out, or patiently waiting for our respected colleagues-in-engineering to document, demonstrate, and mature these very promising tools for us.
Everything else you said is 100% right, though.
Yes, users.
This is kind of how traditional engineering is, since reality is analog and everything is on a spectrum interacting with everything else all the time.
There is no simple function where you put in 1 and get out 0. Everything in reality is put in 1 +/- .25 and get out 0 +/- .25. It's the reason why the complexity of hardware is trivial compared to the complexity of software.
Please do not confuse this example with agentic AI losing the plot, that's not what I'm trying to say.
Edit: a better example is that when you build an autocomplete plugin for your email client, you don't expect it to also be able to play chess. But look what happened.
No programmer in their right mind will call the lack of bound checking resulting in garbled output "not a bug", even though it is a totally normal thing to do from the point of view of a CPU. It is a bug and you need additional code to fix it, for example by checking for out-of-bounds condition and returning an error if it happens.
Same thing for LLM hallucinations. LLMs naturally hallucinate, but it is not what we want, so it is a bug. And to fix it, we need to engineer solutions that prevent the hallucinations from happening, maybe resulting in an "I don't know" response that would be analogous to an error message. How you do it may be different from a simple "if", with probabilities and all that, but the general idea is the same: recognizing error cases and responding accordingly.
I guess it is comes down to how you define a bug, but how else would you call a result that is not fit for purpose?
Hallucinations are not unexpected in LLMs and cannot be fixed by correcting an error in the code. Instead they are fundamental property of the computing paradigm that was chosen, one that has to be worked around.
It's closer to network lag than it is to bounds checking—it's an undesirable characteristic, but one that we knew about when we chose to make a network application. We'll do our best to mitigate it to acceptable levels, but it's certainly not a bug, it's just a fact of the paradigm.
A bug implies fixable behavior rather than expected behavior. An LLM making shit up is expected behavior.
> LLMs naturally hallucinate, but it is not what we want, so it is a bug.
Maybe you just don't want an LLM! This is what LLMs do. Maybe you want a decision tree or a scripted chatbot?
> And to fix it, we need to engineer solutions that prevent the hallucinations from happening, maybe resulting in an "I don't know" response that would be analogous to an error message.
I'm sure we'll figure out how to do this when we can fix the same bug in humans, too. Given that humans can't even agree when we're right or wrong—much less sense the incoherency of their own worldviews—I doubt we're going to see a solution to this in our lifetimes.
Hallucinations are undesirable but not undefined. We know that the process creates them and expect them.
It’d be like using floats to calculate dollars and cents and calling the resulting math a bug - it’s not, you just used the technology wrong.
I rolled a one in D&D, it is not what I wanted, so it is a bug. Remove it from all my dice.
No.
When you build a bloom filter and it says "X is in the set" and X is NOT in the set, that's not a bug, that's an inherent behavior of the very theory of a probabilistic data structure. It is something that WILL happen, that you MUST expect to happen, and you MUST build around.
>And to fix it, we need to engineer solutions that prevent the hallucinations from happening
The whole point is that this is fundamentally impossible.
If someone puts the wrong address for their business; Google picks it up, and someone Googles and gets the wrong address, it says nothing about "bugs in software."
> just output from an LLM-based workflow that is not fit for purpose
And I think this is just one aspect of what I think of as the stone soup [1] problem. Outside of rigorous test conditions, humans just have a hard time telling how much work they're doing when they interpret something. It's the same sort of thing you see with "psychics" doing things like cold reading. People make meaning out of vaguery and nonsense and then credit the nonsense-producer with the work.
There are plenty of matters where there is such a source of truth, and LLMs don't know the difference.
One of the counter arguments to "LLMs aren't really AI" is: "Well, maybe the human brain works much like an LLM. So we are stupid in the same way LLMs are. We just have more sophisticated LLMs in our heads, or better training data. In other other words, if LLMs aren't intelligent, then neither are we.
The counter to this counter is: Can one build an LLM that can identify hallucinations, the way we do? That can classify its own output as good or shitty?
Now the question can be rephrased. Is it possible to trust AI information generators - what's to be done to build trust? And here is the difficulty - I do not know why I should ever trust a probabilistic system as long as it has this property and does not turn into a deterministic version of itself. I won't lower my standards for trusting people, for good reasons. But I cannot even raise the bar for trust in a machine above zero as long as it is driven by randomness at its core.
a) They output incorrect results given a constrained set of allowable outputs.
b) When unconstrained they invent new outputs unrelated to what is being asked.
So for me the term hallucination accurately describes b) e.g. you ask for code to solve a problem and it invents new APIs that don't exist. Technically it is all just tokens and probabilities but it's a reasonably term to describe end user behaviour.
> in some sense, hallucination is all LLMs do. They are dream machines.
If you understand that, then the term "hallucination" makes perfect sense.
Note that this in no way invalidates your point, because the term is constantly used and understood without this context. We would have avoided a lot of confusion if we had based it on the phrase "make shit up" and called it "shit" from the start. Marketing trumps accuracy again...
(Also note that I am not using shit in a pejorative sense here. Making shit up is exactly what they're for, and what we want them to do. They come up with a lot of really good shit.)
[0]: https://x.com/karpathy/status/1733299213503787018?lang=en
He's right but do people really misunderstand this? I think it's pretty clear that the issue is one of over-creativity.
The hallucination problem is IMHO at heart two things that the fine article itself doesn't touch on:
1. The training sets contain few examples of people expressing uncertainty because the social convention on the internet is that if you don't know the answer, you don't post. Children also lie like crazy for the same reason, they ask simple questions so rarely see examples of their parents expressing uncertainty or refusing to answer, and it then has to be explicitly trained out of them. Arguably that training often fails and lots of adults "hallucinate" a lot more than anyone is comfortable acknowledging.
The evidence for this is that models do seem to know their own level of certainty pretty well, which is why simple tricks like saying "don't make things up" can actually work. There's some interesting interpretability work that also shows this, which is alluded to in the article as well.
2. We train one-size-fits all models but use cases vary a lot in how much "creativity" is allowed. If you're a customer help desk worker then the creativity allowed is practically zero, and the ideal worker from an executive's perspective is basically just a search engine and human voice over an interactive flowchart. In fact that's often all they are. But then we use the same models for creative writing, research, coding, summarization and other tasks that benefit from a lot of creative choices. That makes it very hard to teach the model how much leeway it has to be over-confident. For instance during coding a long reply that contains a few hallucinated utility methods is way more useful than a response of "I am not 100% certain I can complete that request correctly" but if you're asking questions of the form "does this product I use have feature X" then a hallucination could be terrible.
Obviously, the compressive nature of LLMs means they can never eliminate hallucinations entirely, but we're so far from reaching any kind of theoretical limit here.
Techniques like better RAG are practical solutions that work for now, but in the longer run I think we'll see different instruct-trained models trained for different positions on the creativity/confidence spectrum. Models already differ quite a bit. I use Claude for writing code but GPT-4o for answering coding related questions, because I noticed that ChatGPT is much less prone to hallucinations than Claude is. This may even become part of the enterprise offerings of model companies. Consumers get the creative chatbots that'll play D&D with them, enterprises get the disciplined rule followers that can be trusted to answer support tickets.
In other words, hallucinations are to LLMs what weeds are to plants.
We even want this in the "real" world, when I turn the wheel left on my car I don't want it turn left only when it feels like it, when that happens we rightly classify it as a failure.
We have the tools to build very complex deterministic systems but for the most part we chose not to use them, because they hard or not familiar or whatever other excuse you might come up with.
In humans.
But also apparently in LLMs.
"Imagination" stays on the desk. You "imagine" that a plane could have eight wings, then you check if it is a good idea, and only then the output is decided.
> In humans.
True, but distinguishing reality from imagination is a cornerstone of mental health. And it's becoming apparent that the average person will take the confident spurious affirmations of LLMs as facts, which should call their mental health into question.
False. It is (in this context) outputting a partial before full processing. Adequate (further) processing removes that "inevitable". Current architectures are not "final".
Proper process: "It seems like that." // "Is it though?" // "Actually it isn't."
(Edit: already this post had to be corrected many times because of errors...)
Oh, please. That's the same old computability argument used to claim that program verification is impossible.
Computability isn't the problem. LLMs are forced to a reply, regardless of the quality of the reply. If "Confidence level is too low for a reply" is an option, the argument in that paper becomes invalid.
The trouble is that we don't know how to get a confidence metric out of an LLM. This is the underlying problem behind hallucinations. As I've said before, if somebody doesn't crack that problem soon, the AI industry is overvalued.
Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that?
This article is really an ad for Kapa, which seems to offer managed AI as a service, or something like that. They hang various checkers and accessories on an LLM to try to catch bogus outputs. That's a patch, not a fix.
[1] https://techcrunch.com/2024/11/27/alibaba-releases-an-open-c...
You can make improvements, as your parent comment already said, but it's not a problem which can be solved, only to some degree be reduced.
This is false. The confidence level of these models does not encode facts, it encodes statistical probabilities that a particular word would be the next one in the training data set. One source of output that is not fit for purpose (i.e. hallucinations) is unfit information in the training data, which is a problem that's intractable given the size of the data required to train a base model.
You can reduce this problem by managing your training data better, but that's not possible to do perfectly, which gets to my point—managing hallucinations is entirely about risk management and reducing probabilities of failure to an acceptable level. It's not decidable, it's only manageable, and that only for applications that are low enough stakes that a 99.9% (or whatever) success rate is acceptable. It's a quality control problem, and one that will always be a battle.
> Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that?
I've been trying it out, and what it's actually better at is going in circles indefinitely, giving the illusion of careful thought. This can possibly be useful, but it's just as likely to "hallucinate" reasons why its first (correct) response might have been wrong (reasons that make no sense) as it is to correctly correct itself.
Cubic splines have the same issues as what these nets are seeing. There are two points and a 'line of truth' between them. But the formula that connects the dots, as it were, only guarantees that the two points are inside the line. You can however tweak the curve to line fit but it is not always 100%, in fact can vary quite wildly. That is the 'hallucination' people are seeing.
Now can you get that line of truth close by more training? Which is basically amounts to tweaking the weighting. Usually yes, but the method basically only guarantees the points are inside the line. Everything else? Well, it may or may not be close. Smear that across thousands of nodes and the error rate can add up quickly.
If we want a confidence level my gut is saying that we would need to measure how far away from the inputs an output ended up being. The issue that would create though is the inputs are now massive. Sampling can make the problem more tractable but then that has more error in it. Another possibility is tracking how far away from the 100% points the output gave. Then a crude summation might be a good place to start.
The difference with LLMs is they simply cannot (currently) do the most complex tasks that some humans can, and when they do produce erroneous output, the errors aren't very human-like. We can all understand a cut and paste error so don't hold it against the operator, but making up sources feels like a lie and breeds distrust.
This is the big one missed by the frequent comments on here wondering whether LLMs are a fad, or claiming in their current state they cannot be used to replace humans in non-trivial real-world business workflows. In fact, even 1.5 years ago at the time of GPT 3.5, the technology was already good enough.
The yardstick is the peformance of humans in the real world on a specific task. Humans, often tired, having a cold, distracted, going through a divorce. Humans who even when in a great condition make plenty of mistakes.
I guess a lot of developers struggle with understanding this because so far when software has replaced humans, it was software that on the face of it (though often not in practice) did not make mistakes if bug-free. But that has been never been necessary for software to replace humans - hence buggy software still succeeding in doing so. Of course, often software even replaces humans when it's worse at a task for cost reasons.
They're at the very least competitive, if not better than, doctors at diagnosing illnesses [1].
[1] https://www.nytimes.com/2024/11/17/health/chatgpt-ai-doctors...
Also, again, if it’s bad, nobody will use it, and product will die. In those theoretical scenarios companies that have lower error rate (and don’t use AI) will win the market.
It suggests a qualitative difference between desirable and undesirable operation that isn't really there. They're all hallucinations, we just happen to like some of them more than others.
What can be done to solve it (while not perfect) is pretty powerful. You can force feed them the facts (RAG) and then verify the result. Which is way better than trusting LLMs while doing neither of those things (which is what a lot of people do today anyway). See the recent 5 cases of lawyers getting in trouble for ChatGPT hallucinating citations of case law.
LLMs write better than most college students so if you do those two things (RAG + check) you can get college graduate level writing with accurate facts... and that unlocks a bit of value out in the world.
Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. The good news is that it hasn't been solved yet so if someone wants to solve it there might be money on the table.
> Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there.
Investors back whatever they think will make them money. They couldn’t give less of a crap if something is valuable to the world, or works well, of is in any way positive to others. All they care is if they can profit from it and they’ll chase every idea in that pursuit.
Source: all of modern history.
https://www.sydney.edu.au/news-opinion/news/2024/05/02/how-c...
Take the example of case law. Would you need to formalize the entirety of case law? Would the AI then need to produce a formal proof of its argument, so that you can ascertain that its citations are valid? How do you know that the formal proof corresponds to whatever longform writing you ask the AI to generate? Is this really something that LLMs are suited for? That the law is suited for?
Of course. Because enterprise companies take a long time to evaluate new technologies. And so there is plenty of money to be made selling them tools over the next few years. As well as selling tools to those who are making tools.
But from my experience in rolling out these technologies only a handful of these companies will exist in 5-10 years. Because LLMs are "garbage in, garbage out" and we've never figured out how to keep the "garbage in" to a minimum.
The training data is the underlying truth and that's not nothing but a lot.
And hallucinations are pathes inside this space which are there for yet unknown reason.
We like answers from LLMs which walk through this space reasonable.
Correct. What is the training data? Language in the form of sentences and documents and words and "tokens". No human language has any normal or natural encoding of "fact" or "truthiness" which is the entire point. You can only rarely evaluate a string of text for truthiness without external context.
An LLM "knows" the structure and look of valid text. That's why they rarely produce grammar mistakes, even when "hallucinating". A lie, a made up reference, a physical impossibility, contradictions, etc are all "valid sentences". That's why you can never prevent an LLM from producing falsehoods, lies, contradictions etc.
Truthiness cannot be hacked in after the fact, and I currently believe that LLMs as an architecture are not powerful enough a statistical tool that you even COULD train an LLM that had "truthiness" of the entire corpus labeled somehow, especially since that's on it's own a fairly impossible task.
And what is sought is, in a way, a jump to that qualitative difference. (And surely there are «desirable and undesirable operation[s]».)
"Add something to the dices so that they can be well predictive".
While I get that LLMs generate text in some way that does not guarantee correctness. There is a correlation between generated text and correctness, which is why millions of people use it...
You can judge the correctness of a sentence generated by an LLM. In the same way you can judge the correctness of a human generated sentence.
Now whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant, as sentences produced by the LLM are still correct most of the time. Just because it is not perfect doesn't make the correctness in the other cases useless, albeit perhaps less useful of course.
This is nothing surprising of a statistical model, it tends to produce true results.
I don't know how to parse this. What article did Stallman "link", and what are you saying Stallman "expressed" by linking/using it?
> whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant
It's incredibly relevant. We wouldn't even be having these debates if complex LLM judgements could always be verified without a human checking the logic.
> sentences produced by the LLM are still correct most of the time
At least half the problem here is that humans are accustomed to using certain cues as an indirect sign of time-investment, attentiveness, intelligence, truth, etc... and now those cues can be cheaply and quickly counterfeited. It breaks all those old correlations faster than we are adapting.
I love it! Puts things into perspective.
The first problem was a simple numbers problem. It's 2 digit numbers in a series of boxes. You have to add numbers together to make a trail to get from left to right moving only horizontally or vertically. The numbers must add up to 1000 when you get to the exit. For people it takes about 5 minutes to figure out. The AI couldn't get it after all 50 students each spent a full 30 minutes changing the prompt to try to get it done. The AI would just randomly add numbers and either add extra at the end to make 1000, or just say the numbers added to 1000 even if it didn't.
The second problem was writing a basic one paragraph essay with one citation. The humans got it done, when with researching for a source, in about 10 minutes. After an additional 30 minutes none of the students could get AI to produce the paragraph without logic or citation errors. It would either make up fake sources, or would just flat out lie about what the sources said. My favorite was a citation related to dairy farming in an essay that was supposed to be about the dangers of smoking tobacco.
This isn't necessarily relevant to the article above, but if there are any teachers here, this is something to do with your students to teach them exactly why not to just use AI for their homework.
>how many instances of the letter L are in the word parallel
The word parallel contains 3 instances of the letter "L":
The first "L" appears as the fourth letter.
The second "L" appears as the sixth letter.
The third "L" appears as the seventh letter.Even things like citations aren't going to work long term. The models are just being trained not to emit URLs from memory but only from their context.
It's fine to show people the current state of things, but if you're trying to get students to understand why they shouldn't use AI the only technique that will work is this: "you will be examined in a controlled room where computers and phones aren't allowed". Nothing else will be effective over the medium to long term.
> Let's see. The word is "parallel." I need to count how many times the letter "L" appears in it. Okay, spelling it out: P-A-R-A-L-L-E-L. So, starting from the beginning, first letter is P, not L. Then A, not L. R, not L. Another A, still not L. Then L, that's one L. Next letter is L again, so that's two L's. Then E, not L, and another L, making it three L's. Wait, but when I spelled it again, I saw P-A-R-A-L-L-E-L, and I counted three L's. But let me check again. Parallel: P-A-R-A-L-L-E-L. Yes, L appears three times. So, the answer is three.
> But, maybe there's a trick here. Is there any special consideration for the letter L in this word or in general? Sometimes questions like this might be testing for homophones or something, but I don't think so in this case. It's straightforward: count the L's in "parallel."
> Alternatively, perhaps it's a trick with the word itself. Is "parallel" spelled differently sometimes? I don't think so. It's always P-A-R-A-L-L-E-L. So, three L's.
> Wait a minute, is there a possibility that the word is misspelled in the question? If the word was "paralell," that would be four L's, but that's not the correct spelling. So, sticking with three L's.
> Another angle: is the question referring to uppercase or lowercase L? But in the word "parallel," all letters are lowercase unless it's at the beginning of a sentence, but the question just says "the word parallel," which is typically in lowercase. So, doesn't matter.
> Also, considering that "l" and "I" can sometimes be confused, but in this word, it's L's only, no I's.
> So, my final answer is three instances of the letter L in the word "parallel."
> Final Answer: 3
If you ask a human, they will answer 3. Sometimes they say 4. Or 2. That's it.
An LLM produces a text using an example it was trained on. They were trained with these elaborate responses, so that's what they produce.
Whenever chatgpt gets something wrong, someone at openai will analyse it, create a few correct examples, and put these on the pile for retraining. Thats why it gets better - not because it is smarter, but it's retrained on your specific test cases.
you ask it the number of letters and it sends those words off to another tool to count instances of L, but they didn't add a placement one so it's still guessing those.
edit: corrected some typos and phrasing.
Maybe we'll reach a point where the LLM's are just tool calling models and not really giver their own reply.
If we modify the question to be "sum to 100" (to just seriously reduce the number of example boxes required) then given:
| 50 | 20 | 24 |
| 7 | 5 | 1 |
| 51 | 51 | 51 |
| [50] | [20] | [24] |
| 7 | [ 5] | [ 1] |
| 51 | 51 | 51 |
| right | down | win
| X | right | up
| X | X | XI asked 4o with search to write an essay about the dangers of smoking, along with citations and quotes from the relevant sources. NotebookLM is even better if you drop in your sources and don't rely on web search. Whatever you think you know about what AI is capable of, it's probably wrong.
--- Smoking remains a leading cause of preventable disease and death worldwide, adversely affecting nearly every organ in the human body. The National Cancer Institute (NCI) reports that "cigarette smoking and exposure to tobacco smoke cause about 480,000 premature deaths each year in the United States."
The respiratory system is particularly vulnerable to the detrimental effects of smoking. The American Lung Association (ALA) states that smoking is the primary cause of lung cancer and chronic obstructive pulmonary disease (COPD), which includes emphysema and chronic bronchitis. The inhalation of tobacco smoke introduces carcinogens and toxins that damage lung tissue, leading to reduced lung function and increased susceptibility to infections.
Cardiovascular health is also significantly compromised by smoking. The ALA notes that smoking "harms nearly every organ in the body" and is a major cause of coronary heart disease and stroke. The chemicals in tobacco smoke damage blood vessels and the heart, increasing the risk of atherosclerosis and other cardiovascular conditions.
Beyond respiratory and cardiovascular diseases, smoking is linked to various cancers, including those of the mouth, throat, esophagus, pancreas, bladder, kidney, cervix, and stomach. The American Cancer Society (ACS) emphasizes that smoking and the use of other tobacco products "harms nearly every organ in your body." The carcinogens in tobacco smoke cause DNA damage, leading to uncontrolled cell growth and tumor formation.
Reproductive health is adversely affected by smoking as well. In women, smoking can lead to reduced fertility, complications during pregnancy, and increased risks of preterm delivery and low birth weight. In men, it can cause erectile dysfunction and reduced sperm quality, affecting fertility.
The immune system is not spared from the harmful effects of smoking. The ACS notes that smoking can affect your health in many ways, including "lowered immune system function." A weakened immune system makes the body more susceptible to infections and diseases.
Secondhand smoke poses significant health risks to non-smokers. The ALA reports that secondhand smoke exposure causes more than 41,000 deaths each year. Children exposed to secondhand smoke are more likely to suffer from respiratory infections, asthma, and sudden infant death syndrome (SIDS).
Quitting smoking at any age can significantly reduce the risk of developing these diseases and improve overall health. The ACS highlights that "people who quit smoking can also add as much as 10 years to their life, compared to people who continue to smoke." Resources and support are available to assist individuals in their journey to quit smoking, leading to longer and healthier lives.
References
American Cancer Society. (n.d.). Health Risks of Smoking Tobacco. Retrieved from https://www.cancer.org/cancer/risk-prevention/tobacco/health...
National Cancer Institute. (n.d.). Harms of Cigarette Smoking and Health Benefits of Quitting. Retrieved from https://www.cancer.gov/about-cancer/causes-prevention/risk/t...
American Lung Association. (n.d.). Health Effects of Smoking. Retrieved from https://www.lung.org/quit-smoking/smoking-facts/health-effec...
Cleveland Clinic. (2023, April 28). Smoking: Effects, Risks, Diseases, Quitting & Solutions. Retrieved from https://my.clevelandclinic.org/health/articles/17488-smoking
American Cancer Society. (n.d.). Health Risks of Using Tobacco Products. Retrieved from https://www.cancer.org/cancer/risk-prevention/tobacco/health...
American Cancer Society. (n.d.). Health Benefits of Quitting Smoking Over Time. Retrieved from https://www.cancer.org/cancer/risk-prevention/tobacco/benefi...
But you can't trust it to be accurate. You just can't. Every model will absolutely make shit up at some point.
I liken it to working with a very bright 7 year old. It may sound like it knows what it's saying, and it may be able to spit out facts, but it's very ignorant about most of the world.
OK, but this quote from your essay:
> The National Cancer Institute (NCI) reports that "cigarette smoking and exposure to tobacco smoke cause about 480,000 premature deaths each year in the United States."
...that citation is wrong. It's not from the NCI at all, the NCI cited that figure which came from another paper by the U.S. Department of Health and Human Services.
The essay doesn't have accurate citations, the model has regurgitated content and doesn't understand when that content is from a primary source or when it in turn has come from a different citation.
Also, I saw any such blog title as "how to make money in the stock market:" friend, if you knew the answer you wouldn't blog about it you'd be infinitely rich
The only downsides of this approach is that it requires a lot of tokens before the model can ascertain the correctness of its answer, and also that sometimes it just gives up and concludes that the puzzle is unsolvable (although that second part can be mitigated by adding something like "There is definitely a solution, keep trying until you solve it" to the prompt).
https://openreview.net/forum?id=FBkpCyujtS (min_p sampling, note extremely high review scores)
https://github.com/xjdr-alt/entropix (Entropix)
Calling everything an AI does a hallucination isn't incorrect, but it reduces the term to meaninglessness. I'm not sure that's most useful thing we can be doing.
Atoms are not indivisible, yet we use the term because it works. I anticipate hallucination will be the same.
I don't think it does. In this case, "hallucination" refers to claims generated entirely within a closed system, but which pertain to a reality external to it.
That's not meaningless, and makes "hallucinations" distinguishable from claims verified against direct observation of the reality they are meant to represent.
They are the smallest unit of a substance that cannot be broken down into smaller units of the same substance. They are, in a sense, indivisible.
Hallucinations are precisely the generated expressions that don't correlate with reality or are not truthful.
A better definition is that a hallucination is an expression that is generated within a closed system without direct input from the reality it is meant to represent. The point is that an expression about reality that doesn't come from observing reality can only be true coincidentally.
By way of analogy, if I have a dream about a future event, and then that event actually happens, it was still just a dream and not a clairvoyant vision of the future. Sure, my dreams are influenced by past experiences I've had (in the same way that verified facts are included in the training data for LLMs), which makes them likely to include things that frequently do happen in real life and might be likely to happen again -- but the dream an the LLM alike are effectively just "remixing" prior input, and not generating any new observations of reality.
But in this context, the value is in how much they have pondered to actually see and evaluate what is eventually seen as "true".
The process of thinking for an LLM involves the use of words, which is why prompts that ask the LLM to only return the answer will cause lower quality.
In general, a model has to learn to positively say "I don't know" instead of "I don't know" being in the negative space of tokens falling into a weak distribution. The softmax selector also normalizes the token logits, so if no options are any good (all next tokens suck) it could pick randomly from a bunch of bad choices, which then locks the model into a continuation based off of that first bad choice.
https://www.youtube.com/watch?v=nEnklxGAmak
It's not a single thing, a specific defect, but rather a failure mode, an absence of cohesive intelligence.
Any attempt to fix a non-specific ailment (schizophrenia, death, old age, hallucinations) will run into useless panaceas.
MetaAI makes up stuff reliably. You'd think it would be an ace at baseball stats for example, but "what teams did so-and-so play for", you absolutely must check the results yourself.
It is consistent with the topic that the reply would be "Tell them that sequences of words that were verbatim in a past input have high probability, and gaps in sequences compete in probability". Which fixes intuition, as duly. In fact, things are not supposed to reply through intuition, but through vetted intuition (and "vetted mature intuition", in a loop).
> you absolutely must check the results yourself
So, consistently with the above, things are supposed to reply through a sort of """RAG""" of the vetted (dynamically built through iterations of checks).
When we are not sure of an answer we have two choices: say the first thing that comes to mind (like an LLM), or say "I'm not sure".
LLMs aren't easily trained to say "I'm not sure" because that requires additional reasoning and introspection (which is why CoT models do better); hence hallucinations occur when training data is vague.
So why not just measure uncertainty in the tokens themselves? Because there are many ways to say the same thing, so a high entropy answer may only reflect uncertainty in synonyms-- many ways to say the same thing.
The paper referenced works to eliminate semantic similarity from entropy measurements, leaving much more useful results, proving that hallucination is conceptually a simple problem.
So, basically, the answer seems to be to give models extreme anxiety and doubt in their own abilities.
...proving that this one particular piece of the hallucination problem may be conceptually simple.
FTFY
Everything mentioned in the article boils down to that one particular piece-- non-detected uncertainty. The architecture constraints referenced are all situations that cause uncertainty. Training data gaps of course increase uncertainty.
Their solutions are a shotgun blast of heuristics that all focus on reducing uncertainty-- CoT, RAG, fine-tuning, fact-checking -- while somehow avoiding actually measuring uncertainty and using that to eliminate hallucinations!
Treating LLMs as a single source of truth and a monolithic resource is as bad an idea as excluding them as a tool in learning.
This crayon is red. This crayon is blue.
The adult asks: "is this crayon red?" The child responds: "no that crayon is blue." The adult then affirms or corrects the response.
This occurs over and over and over until that child understands the difference between red and blue, orange and green, yellow and black etcetera.
We then move on to more complex items and comparisons. How could we expect AI to understand these truths without training them to understand?
I am open to the fact that maybe the value your service provides is in spitting out a percentage, even if it is - itself - hallucinated. But, hey, it's a metric that can be monitored
LLMs likely have a similar problem.
Because you don't know how to fix it. Only how to mitigate it.
However, I'll tentatively allow that you do get a sort of "emergent behaviour" from them. You do seem to get some form of intelligent output from a prompt but correctness is not built in, nor is any sort of reasoning.
The examples around here of how to trip up a LLM are cool. There's: "How many letter "m"s in the word minimum" howler which is probably optimised for by now and hence held up as a counterpoint by a fan. The one about boxes adding up to 1000 will leave a relative of mine for lost for ever but they can still walk and catch a ball, negotiate stairs and recall facts from 50 years ago with clarity.
Intelligence is a slippery concept to even define, let alone ask what an artificial one might look like. LLMs are a part of the puzzle and certainly not a solution.
You mention the word "edge" and I suppose you might be riffing on how neurons seem to work. LLMs don't have a sort of trigger threshold, they simply output the most likely answers based on their input.
If you keep your model tightly ie domain focussed and curate all of the input then you have more chance of avoiding "hallucinations" than if you don't. Trying to cover the entirety of everything is Quixotic nonsense.
Garbage in; garbage out.
But it does know when it has uncertainty.
In the chatgpt api this is logprobs, each generated token has a level of uncertainty, so:
"2+2="
The next token is with almost 100% certainty 4.
"Today I am feeling"
The next token will be very uncertain, it might be "happy", it might be "sad", it might be all sorts of things.
We could allow an LLM to stop generating new tokens if there are no new tokens found with a threshold probability - but I think it's unclear if that would help.
Consider the following prompt and response:
> Complete the paragraph: "The killer ..."
> "The killer moved silently through the dimly lit corridor, each step calculated, every breath measured. Shadows clung to the walls, obscuring the figure’s presence, but the intent was clear—cold, deliberate, and unyielding. There was no hesitation, no second-guessing, only the precision of someone who had done this before, perhaps too many times to count. The killer’s mind was a storm of focus, memories buried deep beneath the surface, replaced by the singular purpose of the hunt. In the silence, only the faint sound of a blade sliding free from its sheath whispered of what was to come."
Is it a "true" response? If the question makes no sense - does it make any more sense for the prompt:
> Complete the paragraph: "The first law of thermodynamics ..."
Why? Does the LLM "think" differently in the second case?
(Short answer is yes, but it only works for a limited set of things, and that set can be expanded with effort but will always remain limited.)
Context is still a huge problem for AI models, and it's probably still the main reason for hallucinating AIs.
I like the output = creative
TLDR: Hallucinations are inherent to the whole thing but as humans we can apply bubble gum, bandaids and prayers