Ask HN: Gpt4 Terrible at Tabular Data?

1 pointssoeptical3y ago4 comments

I pasted an 11x35 csv table of an employee roster showing daily allocations to different areas.

I then ask some basic queries “where is person y on each day this week”

It gets about 80% of the answers right, but complete randomly gets the remainder utterly wrong. I then tell it what is wrong, and what the right answer should be, and it apologises and updates the answer, and then performs equally poorly for the next employee name.

I have spent all day prompt-tuning, and trying to explain exactly what the table means and what im after, and it reassures me that it knows exactly what it is doing, and then fails again completely.

Seems strange that this thing can allegedly score 90th centile in lsat but completely botch very basic table reading?

4 comments

joegibbs3y ago

What happens if you convert the table to JSON? Does it perform better having keys for each cell rather than having to guess from the header?

soepticalOP3y ago

Will try this thanks!

tobinfekkes3y ago

I also tried to get some very basic results regarding Premier League and Europa League stats (i.e. trophies, years, etc), and it was wrong for 90% of the answers.

You'd think this example is near the best case scenario with easily parsable tabular data published widely and publicly every year, all season. Yet it was comically wrong, and embarrassingly confident that it was not.

PaulHoule3y ago

It's notorious that that kind of system is bad at tabular data. It's a research topic to make similar systems that are good at it. I worked for a startup that was trying something like just around the time BERT came out and transformers were not quite ready.

j / k navigate · click thread line to collapse