Evaluating Multimodal LLMs Using the Google IO 2024 Puzzle (opens in new tab)

(hiresynth.ai)

3 pointssimonbutt2y ago2 comments

2 comments

Surprising to see these models stumbling on what at first glance seems like a simple task, it would be interesting to see how the non-vision models fare if you convert the problems to ascii art

simonbuttOP2y ago

GPT-4V, Claude 3 Opus and Gemini Ultra go head to head in solving GoogleIO Puzzle 2024

j / k navigate · click thread line to collapse

2 comments

malet2y ago

Surprising to see these models stumbling on what at first glance seems like a simple task, it would be interesting to see how the non-vision models fare if you convert the problems to ascii art

simonbuttOP2y ago

GPT-4V, Claude 3 Opus and Gemini Ultra go head to head in solving GoogleIO Puzzle 2024

j / k navigate · click thread line to collapse