Things have names, but things are what we perceive about them and largely about what we can and do, do with them.
This is what any agent embodied in an environment must do and do do.
Many of the criticisms about LLM will evaporate for multi-modal models, as they become multimodal, and gain (or infer from) agency.
Rather, embodied agents start with reference and indices. The hard problem is actually ungrounding — which takes work — to eventually get to things that approach what people typically think of “symbols”.
Discrete objects give integer arithmetic. Correspondence gives equality. Spatio-temporal behavior gives basic logic: concurrent AND, choice OR, inside/outside, under/over, up/down, more/less... Properties and behaviors cluster to give categories in a context. Action frames give role bindings for actors...
It's Lakoff&Johnson all the way down.
The symbol grounding problem, as discussed in the paper snippet, addresses the challenge of how to connect the semantic interpretation of symbols within a formal symbol system to the real world, making it intrinsic to the system rather than relying on external interpretations. In the context of Large Language Models (LLMs) and AI, this problem is relevant as it raises questions about the meaning and understanding of symbols or tokens generated by these systems.
The paper suggests a potential solution to the symbol grounding problem: grounding symbolic representations in nonsymbolic representations of two kinds - “iconic representations” (analogous to sensory projections) and “categorical representations” (feature detectors for object and event categories). Elementary symbols are then the names of these categories, assigned based on their categorical representations.
Connectionism, a neural network-based approach, is proposed as a mechanism to learn the invariant features underlying categorical representations, thus connecting symbols to the sensory world. This hybrid model combines symbolic and connectionist elements to address the symbol grounding problem.
In the context of LLMs, they operate primarily based on symbolic manipulation of text, and while they can generate text that appears semantically meaningful, the challenge lies in grounding this meaning in real-world understanding. LLMs lack the sensory perceptions and cognitive mechanisms that humans have for grounding symbols. Addressing the symbol grounding problem in LLMs would require incorporating mechanisms to connect their symbolic outputs to real-world understanding, similar to the proposed hybrid model discussed in the paper.
In summary, the symbol grounding problem is a significant challenge in AI and LLMs, as it questions how symbolic representations can be tied to real-world semantics. Addressing this problem would require developing mechanisms to bridge the gap between symbolic manipulation and true understanding of the world.
