undefined | Better HN

0 pointsTerrifiedMouse2y ago0 comments

> I feel like using LLM today is like using search 15 years ago - you get a feel for getting results you want.

I don't think it's quite the same.

With search results, aka web sites, you can compare between them and get a "majority opinion" if you have doubts - it doesn't guarantee correctness but it does improve the odds.

Some sites are also more reputable and reliable than others - e.g. if the information is from Reuters, a university's courseware, official government agencies, ... etc. it's probably correct.

With LLMs you get one answer and that's it - although some like Bard provide alternate drafts but they are all from the same source and can all be hallucinations ...

0 comments

famouswaffles2y ago

>although some like Bard provide alternate drafts but they are all from the same source and can all be hallucinations ...

Yes and no. If the LLM is repeating the same thing on multiple drafts then it's very unlikely to be a hallucination.

It's when multiple generations are all saying different things that you need to take notice.

LLMs hallucinate yes but getting the same hallucination multiple times is incredibly rare.

TerrifiedMouseOP2y ago

Then why aren’t hallucinations being eliminated by comparing drafts?

famouswaffles2y ago

automatically comparing drafts for every single query would be expensive.

and that wouldn't eliminate hallucinations just tell you if large details have likely been hallucinated.

But it's a method some research has used.

https://arxiv.org/abs/2303.08896

1 more reply

PoignardAzur2y ago

Wait, is that true? I feel like that claim needs a lot of disclaimers.

famouswaffles2y ago

https://arxiv.org/abs/2305.18248

"In particular, we find that LMs often hallucinate differing authors of hallucinated references when queried in independent sessions, while consistently identify authors of real references. This suggests that the hallucination may be more a generation issue than inherent to current training techniques or representation."

https://arxiv.org/abs/2303.08896

"SelfCheckGPT leverages the simple idea that if a LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another."

j / k navigate · click thread line to collapse

0 pointsTerrifiedMouse2y ago0 comments

> I feel like using LLM today is like using search 15 years ago - you get a feel for getting results you want.

I don't think it's quite the same.

With search results, aka web sites, you can compare between them and get a "majority opinion" if you have doubts - it doesn't guarantee correctness but it does improve the odds.

Some sites are also more reputable and reliable than others - e.g. if the information is from Reuters, a university's courseware, official government agencies, ... etc. it's probably correct.

With LLMs you get one answer and that's it - although some like Bard provide alternate drafts but they are all from the same source and can all be hallucinations ...

0 comments

famouswaffles2y ago

>although some like Bard provide alternate drafts but they are all from the same source and can all be hallucinations ...

Yes and no. If the LLM is repeating the same thing on multiple drafts then it's very unlikely to be a hallucination.

It's when multiple generations are all saying different things that you need to take notice.

LLMs hallucinate yes but getting the same hallucination multiple times is incredibly rare.

TerrifiedMouseOP2y ago

Then why aren’t hallucinations being eliminated by comparing drafts?

famouswaffles2y ago

automatically comparing drafts for every single query would be expensive.

and that wouldn't eliminate hallucinations just tell you if large details have likely been hallucinated.

But it's a method some research has used.

https://arxiv.org/abs/2303.08896

1 more reply

PoignardAzur2y ago

Wait, is that true? I feel like that claim needs a lot of disclaimers.

famouswaffles2y ago

https://arxiv.org/abs/2305.18248

https://arxiv.org/abs/2303.08896

j / k navigate · click thread line to collapse