undefined | Better HN

0 pointsskywhopper1y ago0 comments

"The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning."

Not sure I understand how this follows. The fact that a certain type of model does well on a certain benchmark means that the benchmark is relevant for a real-world reasoning? That doesn't make sense.

0 comments

munchler1y ago

It shows objectively that the models are getting better at some form of reasoning, which is at least worth noting. Whether that improved reasoning is relevant for the real world is a different question.

moffkalast1y ago

It shows objectively that one model got better at this specific kind of weird puzzle that doesn't translate to anything because it is just a pointless pattern matching puzzle that can be trained for, just like anything else. In fact they specifically trained for it, they say so upfront.

It's like the modern equivalent of saying "oh when AI solves chess it'll be as smart as a person, so it's a good benchmark" and we all know how that nonsense went.

munchler1y ago

Hmm, you could be right, but you could also be very wrong. Jury's still out, so the next few years will be interesting.

Regarding the value of "pointless pattern matching" in particular, I would refer you to Douglas Hofstadter's discussion of Bongard problems starting on page 652 of _Godel, Escher, Bach_. Money quote: "I believe that the skill of solving Bongard [pattern recognition] problems lies very close to the core of 'pure' intelligence, if there is such a thing."

1 more reply

bagels1y ago

It doesn't follow, faulty logic. The two are probably correlated though.

j / k navigate · click thread line to collapse

0 pointsskywhopper1y ago0 comments

"The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning."

0 comments

munchler1y ago

moffkalast1y ago

It's like the modern equivalent of saying "oh when AI solves chess it'll be as smart as a person, so it's a good benchmark" and we all know how that nonsense went.

munchler1y ago

Hmm, you could be right, but you could also be very wrong. Jury's still out, so the next few years will be interesting.

1 more reply

bagels1y ago

It doesn't follow, faulty logic. The two are probably correlated though.

j / k navigate · click thread line to collapse