No doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data for most LLMs yet.
That's actually a really great point. I'm guessing we need to keep modifying attributes of the questions while maintaining the underlying structure. Instead of "Sally (a girl)", it's "Sal (a guy)" and then tweak the numbers.
Although, part of me is convinced it's almost a fluke that MPT 7B gets it right because MPT 30B doesn't.
