The network's hidden layers. I can elaborate, but looking at your profile, you've implemented an LSTM before, so I shouldn't need to delve deeply into how that works. I'm honestly not sure where your confusion or aversion to the idea that much models can learn to encode semantically meaningful concepts is.
The concept of "color" is never explicitly encoded anywhere by a human. It infers clusterings based on the correlations between which cards are chosen. Unsurprisingly, given reasonable training data, those clusters form along useful boundaries in the game world, one of which is color. If you similarly passed a pack containing every card into the model, you'd likely get out what the model's opinion on the best limited card is. No one ever told it that, but based on the training data, the model "figures it out".
My comments on understanding can be summarized as such: either
1. You're of the opinion that nothing that isn't "strong AI" can have understanding, because understanding is some concept unique to conscious entities (or some reasonably similar opinion). This is an almost completely semantic argument, and isn't particularly interesting. Its an argument about definitions that avoids any actual useful academic questions.
2. You think that non-conscious entities can "understand" concepts, but deny that implicit understandings based on learned clusterings is "understanding". This is marginally more interesting, but wrong: if an implicit understanding can pass a "turing test" whereby I mean that the statistical/learned model can perform as well as whatever you're comparing against, whether it is a human or an expert system, at some task, the two things have the same understanding when confined to that domain.
In other words, sure saying a model doesn't "understand language" might be reasonable because language is multifaceted. But suggesting a model that outperform humans on the synonym portion of the LSAT doesn't understand synonyms is silly. Of course it does. Better than humans. Sure it can't express its understanding of synonyms as music or dance, but that's not because it lacks understanding of synonyms, that's because it lacks other basic faculties that we take for granted.
The question of whether or not you or I can introspect the model to see how its understanding is structured doesn't matter. I can't look inside your head to see how your understanding of language is structure. There's no ArrayList<WordDefinition> I can see in your mind. But I think anyone would agree that you and I both "understand" synonyms despite that lack of transparency. Why would you expect anything different from a statistical model?
Please don't do this. Too many assumptions about what and how I think leave a bad taste.
Yes, a model that can identify synonyms accurately lacks human faculties, including understanding. That's what modern machine learning boils down to. There are many tasks we thought would require human intelligence or reasoning, that can, after all, be reduced to dumb classification. In other words, there is no need to claim "understanding" to explain the output of a classification model, just because a human can perform the same task _and_ can understand it.
As to the representation- that is the only thing that matters. If you want to claim a model represents a concept, you have to be able to show where in the model's structure that concept is represented. If there is a representation- where is it?
> If there is a representation- where is it?
Where is your understanding of language?
>> For this model, its that the vector space clusters similar colored cards.
I mean- what is the representation you speak of in the previous comment. What data structure holds the model's understanding of M:tG colour? The source code is available online.
