> we in fact cannot know if they're conscious or not, even if they seem to be
They (modern LLM's/agents) don't "seem to be" from my point of view. I respectfully disagree I suppose.
edit: One data point - https://www.twitch.tv/claudeplayspokemon . Claude has been failing to to beat pokemon, a game effectively made for children, for _months_ now.