undefined | Better HN

0 pointsnopinsight1y ago0 comments

> Currently, all LLMs are so limited that they struggle with journeys longer than four edges, even when given a full itinerary of all edges in the graph.

This is probably not the case for LLMs in the o1 series and possibly Claude 3.5 Sonnet. Have you tested them on this claim?

0 comments

llm_trw1y ago

Yes, they also fail. I've found the original gpt4 to be the most consistent. One of these days I'll spend the couple of thousands needed to benchmark all the top models and see how they actually perform on a task which can't be gamed.

nopinsightOP1y ago

What kinds of problems in what domains did you test o1 models with?

I found that they are good at logic and math problems but still hallucinate. I didn’t try to stretch test them with hard problems though.

llm_trw1y ago

Finding a path between two vertices when given an itinerary of all the edges in a general graph, exactly what I said in the OP.

1 more reply

j / k navigate · click thread line to collapse

0 pointsnopinsight1y ago0 comments

> Currently, all LLMs are so limited that they struggle with journeys longer than four edges, even when given a full itinerary of all edges in the graph.

This is probably not the case for LLMs in the o1 series and possibly Claude 3.5 Sonnet. Have you tested them on this claim?

0 comments

llm_trw1y ago

nopinsightOP1y ago

What kinds of problems in what domains did you test o1 models with?

I found that they are good at logic and math problems but still hallucinate. I didn’t try to stretch test them with hard problems though.

llm_trw1y ago

Finding a path between two vertices when given an itinerary of all the edges in a general graph, exactly what I said in the OP.

1 more reply

j / k navigate · click thread line to collapse