Author here. I think our approach to AI benchmarks might be too human-centric. We keep creating harder and harder problems that humans can solve (like expert-level math in FrontierMath), using human intelligence as the gold standard.
But maybe we need simpler examples that demonstrate fundamentally different ways of processing information. The dice prediction isn't important - what matters is finding clean examples where all information is visible, but humans are cognitively limited in processing it, regardless of time or expertise.
It's about moving beyond human performance as our primary reference point for measuring AI capabilities.
