undefined | Better HN

0 pointsthatcherthorn2y ago0 comments

They've reported surpassing GPT4 on several benchmarks. Does anyone know of these are hand picked examples or is this the new SOTA?

0 comments

xiphias22y ago

It will be SOTA maybe when Gemini Ultra is available. GPT-4 is still SOTA.

philomath_mn2y ago

Usually SOTA status is established when the benchmark paper is released (probably after some review). But GPT4 is the current generally-available-SOTA

silveraxe932y ago

They also compare to RLHFed GPT-4, which reduces capabilities, while their model seems to be pre-RLHF. So I'd expect those numbers to be a bit inflated compared to public release.

williamstein2y ago

They certainly claim it is SOTA for multimodal tasks: “Gemini surpasses SOTA performance on all multimodal tasks.”

j / k navigate · click thread line to collapse