Yes, but people don’t reason about language, they just do it. I know you think I’m confused about this but I’m not. I mean reason here quite explicitly because what we’re talking about is understanding. No one thinks that they ... uh, well ... “understand” language ... okay, we need a new word here because “understand” has two different meanings here. Let’s use “perform” for when you make correct choices from an inexplicit model, that’s what the NN does, and hold “understand” for what a linguist (maybe) does per language, and what a physicist does per orbital mechanics. What we are hoping a GAI will do is the latter. Any old animal can perform. Only humans, as far as we know, and perhaps a few others in relatively barrow cases, understand in the sense that a physicist understands OM. No NN trained on language is gonna have the present argument. Ever.

I don’t understand your last point, but the point about it being hard to be clear about what a model means is exactly right. But it’s not because it’s not clear what a model is, but rather because it’s not clear what the modeling language of thought is. Here’s where the algebra analogy breaks down. Pretty obviously, the model or models that we are reasoning with in this discussion aren’t simple algebraic equations, but some sort of rich representations of cognitive science and computer science concepts. And, sure, there are NNs running those models, and NNs running the reasoning over them, but they have almost nothing to do with language in the sense of the syntax of sentences. Also, we didn’t get trained with eleventy zillion examples of AI discussions in order to form the models we are employing at this very moment.