undefined | Better HN

0 pointsdTal10mo ago0 comments

A "Chinese Room" absolutely will, because the original thought experiment proposed no performance limits on the setup - the Room is said to pass the Turing Test flawlessly.

People keep using "Chinese Room" to mean something it isn't and it's getting annoying. It is nothing more than a (flawed) intuition pump and should not be used as an analogy for anything, let alone LLMs. "It's a Chinese Room" is nonsensical unless there is literally an ACTUAL HUMAN in the setup somewhere - its argument, invalid as it is, is meaningless in its absence.

0 comments

CamperBob210mo ago

A Chinese Room has no attention model. The operator can look up symbolic and syntactical equivalences in both directions, English to Chinese and Chinese back to English, but they can't associate Chinese words with each other or arrive at broader inferences from doing so. An LLM can.

If I were to ask a Chinese room operator, "What would happen if gravity suddenly became half as strong while I'm drinking tea?," what would you expect as an answer?

Another question: if I were to ask "What would be an example of something a Chinese room's operator could not handle, that an actual Chinese human could?", what would you expect in response?

Claude gave me the first question in response to the second. That alone takes Chinese Rooms out of the realm of any discussion regarding LLMs, and vice versa. The thought experiment didn't prove anything when Searle came up with it, and it hasn't exactly aged well. Neither Searle nor Chomsky had any earthly idea that language was this powerful.

dTalOP10mo ago

Where are you getting all this (wrong) detail about the internals of the Chinese Room? The thought experiment merely says that the operator consults "books" and follows "instructions" (no doubt Turing-complete but otherwise unspecified) for manipulating symbols they they explicitly DO NOT understand - they do NOT have access to "symbolic and syntactical equivalences" - that is the POINT of the thought experiment. But the instructions in the books in a Chinese Room could perfectly well have an attention model. The details are irrelevant, because - I stress again - Searle's Chinese Room is not cognitively limited, by definition. Its hypothetical output is indistinguishable from a Chinese human.

I tend to agree that Chinese Rooms should be kept out of LLM discussions. In addition to it being a flawed thought experiment, of all the dozens of times I've seen them brought up, not a single example has demonstrated understanding of what a Chinese Room is anyway.

CamperBob210mo ago

The details are irrelevant, because - I stress again - Searle's Chinese Room is not cognitively limited, by definition.

So said Searle. But without specifying what he meant, it was a circular statement at best. Punting to "it passes a Turing Test" just turns it into a different debate about a different flawed test.

The operator has no idea what he's doing. He doesn't know Chinese. He has a Borges-scale library of Chinese books and a symbol-to-symbol translation guide. He can do nothing but manipulate symbols he doesn't understand. How anyone can pass a well-administered Turing test without state retention and context-based reflection, I don't know, but we've already put more thought into this than Searle did.

j / k navigate · click thread line to collapse

0 comments

CamperBob210mo ago

If I were to ask a Chinese room operator, "What would happen if gravity suddenly became half as strong while I'm drinking tea?," what would you expect as an answer?

Another question: if I were to ask "What would be an example of something a Chinese room's operator could not handle, that an actual Chinese human could?", what would you expect in response?

dTalOP10mo ago

CamperBob210mo ago

The details are irrelevant, because - I stress again - Searle's Chinese Room is not cognitively limited, by definition.

So said Searle. But without specifying what he meant, it was a circular statement at best. Punting to "it passes a Turing Test" just turns it into a different debate about a different flawed test.

j / k navigate · click thread line to collapse