Something I think about frequently is that 20 years ago, there weren’t machines that could do visual object recognition/categorization and we didn’t really have a clue how humans did it either. We knew that neuron built fancier and fancier receptive fields that became “feature detectors”, but h the ere was a sense of “is that all it takes? There has to be something more sophisticated in order to handle illumination changes of out of plane rotation?”

But then we got a neural wr that was big enough and it turns out that feedforward receptive fields ARE enough. We don’t know whether this is how our brains do it, but it’s a humbling moment to realize that you just overthought how complex the problem was.

So ive become skeptical when people start claiming that some class of problem is fundamentally too hard for machines.