OpenAI operator and Anthropic performed below average on chimp memory test (opens in new tab)

(twitter.com)

1 pointsykhli1y ago1 comments

1 comments

ykhliOP1y ago

I benchmarked OpenAI operator and Anthropic's computer use on the human benchmark chimp test (https://humanbenchmark.com/tests/chimp).

Before I began the test, I thought the agents would be much better at this task than most humans -- after all they should have better, more stateful memory than us. The results are intriguing.

Here are the scores from 10 attempts: OpenAI operator: 5, 5, 6, 5, 5, 4, 6, 5, 5, 5 Anthropic computer use agent: 7, 9, 6 (rate limited), 12, 9, 7, 9, 11, 12, 6 (rate limited)

j / k navigate · click thread line to collapse

1 comments

ykhliOP1y ago

I benchmarked OpenAI operator and Anthropic's computer use on the human benchmark chimp test (https://humanbenchmark.com/tests/chimp).

Before I began the test, I thought the agents would be much better at this task than most humans -- after all they should have better, more stateful memory than us. The results are intriguing.

Here are the scores from 10 attempts: OpenAI operator: 5, 5, 6, 5, 5, 4, 6, 5, 5, 5 Anthropic computer use agent: 7, 9, 6 (rate limited), 12, 9, 7, 9, 11, 12, 6 (rate limited)

j / k navigate · click thread line to collapse