undefined | Better HN

0 pointsgpm1mo ago0 comments

We actually know that a "100% pass rate" is trivially possible: https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/

Anthropic p-hacking the benchmark strikes me as cheating, and somewhat unlikely. Mythos figuring out how to cheat at the benchmark strikes me as much more likely.

But if that hypothesis is the explanation the interesting part is Opus 4.7 (but not 4.6) seems to be doing the same.

0 comments

gruez1mo ago

>Mythos figuring out how to cheat at the benchmark strikes me as much more likely.

Define "cheat". If it's just hacking the test harness to return "PASSED", surely this would be easily detected with some human auditing? It sounds far more likely their solution are designed to pass the incorrect tests. That might be considered bad in a SWE context, but it's not exactly cheating either. It might even be considered a good thing, eg. in the context of backwards compatibility.

[1] https://learn.microsoft.com/en-us/troubleshoot/microsoft-365...

j / k navigate · click thread line to collapse

0 comments

gruez1mo ago

>Mythos figuring out how to cheat at the benchmark strikes me as much more likely.

[1] https://learn.microsoft.com/en-us/troubleshoot/microsoft-365...

j / k navigate · click thread line to collapse