2.) "a team of computer-vision researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL)" may have "described a method for peering into" the not-black box of a convnet two years ago, but Oxford researchers published on in in 2013.
3.) Gushing about how understanding convolutional networks can help confirm the grandmother cell hypothesis in real brains is embarrassing under all circumstances but should be particularly so when thorough examinations from real brains just came out to the considerable detriment of said hypothesis. http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X
Nothing wrong with making visualizations of your nets, but I'm less than impressed by the reporting.
I get that they mean something like 'is difficult to understand' too, which is in many ways completely uncontroversial. (I'm certainly not going to claim it's a solved problem to quickly and effectively come understand intellectually how an arbitrary ANN does what it does. I doubt there are many people who would.) If that's what they mean, then they can say that, or make up their own picture language that isn't already busy meaning the polar opposite of the situation they're alluding to. A black box is characterized by observability only at the edges and unknown inner workings. It is by definition an inappropriate term for a convolutional network where every single weight and operation and intermediate result is trivially inspectable and you can do things like follow the effects of an experimental perturbation along every step of every path through the network. 'But it's really hard to get an intuitive understanding of what that means in the big picture' is a perfectly legitimately concern for something to improve on, but it isn't remotely good enough as an excuse for effectively claiming we have no observability or control where both are clearly abundant.
Abusing the term like this detracts from its established role in engineering, systems theory, etcetera and in my opinion also from communicating the actual problems of understanding how ANN:s do their thing. I really wish they'd stop making that claim.
Now I'm going to leave my computer before I get started on the ######s talking about steep learning curves as if they were an obstacle.
For example, "Many recent advances in NLP problems have come from formulating and training expressive and elabo- rate neural models. This includes models for senti- ment classification, parsing, and machine translation among many others. The gains in accuracy have, however, come at the cost of interpretability since complex neural models offer little transparency con- cerning their inner workings."
Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola
Also: "When people say, "Neural networks are black boxes", what they mean is that it is hard to look "into" the network and figure out exactly what it has learnt.
"In a hand-crafted pipeline, you know precisely what you are building. So you may say, "my face detector will first look for eyebrows, then the mouth, and only if both are present, it will say 'face' ".
"For simple machine learning models, like linear SVMs or linear regression, you can "look" inside the model and see what it is actually doing (well, sort of). The model in this case is just a weighted linear combination of the features and so a feature the model weights highly is probably important to the task, while a feature not weighted highly is probably irrelevant (this is a big oversimplification).
"However if your model is learnt, end-to-end, as well as highly non-linear, as in a neural network, you can't do this. You know the model is some non-linear combination of some neurons, each of which is some non-linear combination of some other neurons, but it is near impossible to say what each neuron is doing."
https://www.quora.com/Why-are-artificial-neural-nets-black-b...
I disagree that this work is detrimental to the idea of a grandmother cell. What this work shows is how a brain "understands" a face, i.e. how it extracts the meaningful features from a given face. But recognizing individuals is an orthogonal concern to "understanding" the face. This work shows how to decode/encode pixel intensity arrays into facial features. It doesn't show how the brain then maps a given set of facial features to individuals.
"Why did it do that?"
and
"Make it stop doing that!"
The first time a self-driving car accident results in a court case, these things are going to come up. I very much doubt that people are going to be satisfied without clear explanations, and they shouldn't be. When these systems take on roles of increasing importance to society, some level of accountability is going to be necessary.
This one's cool too: http://scs.ryerson.ca/~aharley/vis/conv/
I just scan papers that come up in the Reddit group[1]. I've seen:
"Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks" by Rajarshi Das, Arvind Neelakantan, David Belanger, Andrew McCallum
"Rationalizing Neural Predictions" by Tao Lei, Regina Barzilay and Tommi Jaakkola
"'Why Should I Trust You?' Explaining the Predictions of Any Classifier by Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin
You might be able to chase down the works of these various authors to find more.
