undefined | Better HN

0 pointsLeonardoTolstoy2y ago0 comments

The first is maybe a good example of where an LLM integrated search engine makes sense. Because "compare two movies" has flickchart as the third option for me. But it is nowhere to be seen for "compare two movie posters" which is how I read your search originally (and indeed flickchart is about ranking movies not the posters themselves ... Obviously).

Anyways an LLM clearly teased that out whereas if you misremember or misread something a straight search is going to be bad.

Most of my searches are the opposite. I was to know about an obscure movie from the 80s with a toy helicopter. Google very neatly suggests Defense Play (correct) but most LLMs I've tried end up just suggesting very popular films with a helicopter and it ends up being quite difficult to get it to give me information about obscure stuff. Also with that same search the LLM suggests a bunch of incorrect films since (and I figured this out later) it turns out that it was all sourced from a single forum thread from the 90s where a bunch of the posts suggested movies that don't have toy helicopters in them. Go figure.

0 comments

SomewhatLikely2y ago

Thanks, this gave some good insight to GPT4. If I provide the entire Wikipedia page contents but blank out the movie name and director name it can't recall it. https://chat.openai.com/share/c499e163-3745-48c3-b00e-11ea42...

However, if I add the director it gets it right: https://chat.openai.com/share/a602b3b0-5c17-4b4d-bed8-124197...

If I only give it 1980s film and the director's name it can still get it. https://chat.openai.com/share/d6cf396b-3199-4c80-84b9-d41d23...

So it's clearly not able to look this movie up semantically and needs a strong key like the director's name.

EDIT: Digging deeper it's clear the model only has a very foggy idea of what the movie is about: https://chat.openai.com/share/d0701f53-1250-421e-aa4b-dc8156... People have described these types of outputs as the text equivalent of a highly compressed JPEG, which seems to fit well with what's going on here. It gets some of the top level elements right and kind of remembers there's some kind of vehicle that's important but it has forgotten all the details, even the date the movie was released. But unlike a human who might indicate their fuzziness (was it a helicopter or a submarine?), GPT4 gladly pretends like it knows what it's talking about. I think it's likely a solvable problem, the model probably has the information to know when it's confident and when it's in a fuzzy JPEG region but the current alignment isn't doing a great job of surfacing that.

c-hendricks2y ago

"compare two movies" on Google doesn't even have flickchart for me, and rewording my question to not involve posters also doesn't have flickchart in the results.

So, get as pedantic as you'd like, but Google really dropped the ball. ChatGPT gave flickchart with both questions.

j / k navigate · click thread line to collapse