more interesting question: if some of those "digits" are so hard to recognize even by humans then how can we ever label them with the one true "correct" answer?

Do we know a person who draw those digits and ask "what artist had in mind when making this masterpiece" ? And even then someone might have been trying to draw the "2" but end effect looks more like "3".

I think that some of the test cases simply don't have definitive answer and trying to reach 100% accuracy is just misguided effort.