> is qualitatively different from a robot
yes.
> To reduce a system to its inputs and outputs is fine if those are all that matter in a given context
we argue that this indeed is all that matters
> but in doing so you may fail to understand its internal mechanics
the internal mechanics are what we call "conscious" it is the grouping of internal mechanics into one unified concept, but we don't care exactly what they are.
> Those matter if you're trying to really understand the system, no?
since we cannot directly observe consciousness, we are forced to concede that we will never really "understand" it outside of observing its effects.
In the same way that a mechanical turk human and a robot can "play chess", a human and an LLM are "conscious". That is, consciousness is the ability to play chess, by some mechanism. The exact mechanism is irrelevant for the purposes of yes/no conscious.
We now enter a discussion on how much these two consciousnesses differ.