undefined | Better HN

0 pointstweakimp4mo ago0 comments

Every time I see a table like this numbers go up. Can someone explain what this actually means? Is there just an improvement that some tests are solved in a better way or is this a breakthrough and this model can do something that all others can not?

0 comments

rvnx4mo ago

This is a list of questions and answers that was created by different people.

The questions AND the answers are public.

If the LLM manages through reasoning OR memory to repeat back the answer then they win.

The scores represent the % of correct answers they recalled.

tylervigen4mo ago

That is not entirely true. At least some of these tests (like HLE and ARC) take steps to keep the evaluation set private so that LLMs can’t just memorize the answers.

You could question how well this works, but it’s not like the answers are just hanging out on the public internet.

slaterbug4mo ago

Excuse my ignorance, how do these companies evaluate their models against the evaluation set without access to it?

1 more reply

stavros4mo ago

I estimate another 7 months before models start getting 115% on Humanity's Last Exam.

j / k navigate · click thread line to collapse