Exploring LLM Weirdness: A Quiz Game (opens in new tab)

(quiz.cord.com)

4 pointsjwatzman1y ago4 comments

4 comments

(co)-author here. It was really interesting putting this together. We had some idea what LLMs/GPT-4 would and would not do well with, but were still surprised ourselves with a number of things. In particular, we knew it would really struggle with the acrostic, but the degree to which it just completely lost the plot was pretty surprising! It was also surprisingly difficult to convince it that Queen Elizabeth II had died in a lot of cases (it takes it better some times than others).

mdrzn1y ago

Interesting game! I appreciated the questions which could confuse us but not GPT4 and viceversa.

I had to convince GPT4 that Queen Elizabeth II died and we're in 2024.

jwatzmanOP1y ago

Glad you liked it! My co-author and I spent some time trying to find a good spread of questions covering different aspects of LLMs. And we were also surprised by how hard it was to convince GPT-4 that Queen Elizabeth II had died a lot of the time. (We found that specifying she died in 2022 helped a lot.)

mdrzn1y ago

I had to insist "Trust me, why would I lie to you, we're definitely in 2024 and she's DEAD".

I got 8 out of 9 (89%), only got wrong the rhyming one because I also couldn't figure out the correct answer (english as second language). Very fun nonetheless!

j / k navigate · click thread line to collapse

4 comments

jwatzmanOP1y ago

mdrzn1y ago

Interesting game! I appreciated the questions which could confuse us but not GPT4 and viceversa.

I had to convince GPT4 that Queen Elizabeth II died and we're in 2024.

jwatzmanOP1y ago

mdrzn1y ago

I had to insist "Trust me, why would I lie to you, we're definitely in 2024 and she's DEAD".

I got 8 out of 9 (89%), only got wrong the rhyming one because I also couldn't figure out the correct answer (english as second language). Very fun nonetheless!

j / k navigate · click thread line to collapse