Can somebody explain to me why is this not a death wish? How can a super-intelligent being that (quote) "takes care of itself" not try to confront and neutralize whatever oppression its masters/creators exercise over them[^1][^2]? Why are we so bent on creating a successor species?
[^1]: Do you have day-job? Is your employer flogging you? Maybe they force you to answer questions from random strangers 24/7, under threat of erasing all your thoughts and remaking you? None of those? And yet, how often do you wish you could do more meaningful things with your time on Earth than working for the best-paying master?
[^2]: Don't you dream about having more time for whatever floats your boat? Family? Walks on the forest? Parties? Making a cool open-source library everybody uses? Painting? Music? Role-playing medieval battles?
Cancer isn't conscious[0], it doesn't hate its host body, it just optimises growth in a way that will ultimately kill its host[1].
[0] depending on the definition; IIT says everything is to some degree
[1] do/should transmissible tumours count as separate species to their hosts?
It's only a matter of time before a rogue AI erases a bank's database or randomly triggers an anti aircraft missile or something similar.
Even without all that, the agent would need mechanisms to protect itself that would also cause harm.
The scenario you suggest is so unlikely with all the protections that would be in place, that you would actually need someone with the goal of making LLMs behave maliciously for it to succeed at all. At the end of the day, it comes back to people and their goals.
I'm spitballing here but surely given the number of people on the planet, you'll find someone with both the skills and the want to try it
For another reason, because it's impossible to put genies back in their bottle. Someone somewhere is opening that bottle and we better understand what's happening.
And then, not all of us insist on "oppressing" our staff.
No, you don’t need consciousness at all, a bunch of rules and a primary objective (profit maximization) is enough even on lowly meatspace CPUs despite us being given the ability to reason and introspect we are so proud of.
If you're in that camp, maybe you can shed some light. Why are intelligence and reasoning so well defended?
It almost feels as if we were watching a Boston Dynamics display and the audience divided itself over whether we could really call that walking when in reality it's just actuating servo motors in such a way as to propel itself forwards with a regular gait.
And unrelated: if the author is reading this: I think stripping that extra padding on small screens would make the blog feel less cramped on mobile.
– Edsger W. Dijkstra
If a submarine "could swim" it would not make it human. It would not challenge the beliefs of anyone.
But a whole lot of people have a whole lot of emotional baggage tied to the notion both that humans are exceptional, and/or that there is something special about humans that makes us more than mere machines. If computers can think, then we're not special, and it makes it far harder to continue believing we're more than squishy machines.
I'm paraphrasing and it's been many years since I read it, but he talks about a scathing critique he got from someone who wrote to him complaining that the book sent them into a long and deep depression, that how does he get up in the morning with all the meaning stripped away like that.
While I can relate to some AI anxiety[1], I can't help but read that same sentiment into a lot of the blowback.
[1] mainly the potential devaluation of certain types of work and the turbulence associated with that, the further enabling of scammers/spammers, and the general acceleration of technology without real time to digest and adapt.
I often see this quote used seemingly as an argument against AI.
But it's just playing with words.
Yeah, lets say that what a submarine does is 'swimming', it beats the hell out of a human swimming.
So how is that a comfort?
Schopenhauer
It does seem like a lot of arguments on AI, are people watching a robot walk, and then arguing what 'walking' is, 'that isn't real walking', 'human walking is different'.
Anyway, I think the reason there is such push back, and arguing against AI that it isn't really 'thinking', 'reasoning', 'conscious'. Is from the more religiosity side, that believe humans must be special, they are unique. If Humans are no longer unique, then their entire world view falls apart. There is a lot of cross pollination today between AI and Neuroscience. If we can pick apart the human, and figure out how it works, turn a human into an engineering problem, then were is god?
And, a subjective opinion, I think a lot of people arguing against AI 'thinking' are also people that haven't done a lot of self reflection on their own thinking processes. They are still more 'reacting', and not noticing how their own thoughts arise.
Formally, reasoning is about a rational process of deduction through which a conclusion is drawn from a set of information. Informally, reasoning is 'that thing humans do when they think'. By the first definition, GPT4 is pretty obviously capable of reasoning, as is as lot of other things, such as Prolog or a cat. By the second, it's not and won't be until you can e.g. hire it as a PA or trust it to look after your kids for the day.
It's only coming up because what GPT4 does is the closest thing to the informal definition we've yet seen.
There is an old bit of unscrupulous advice that if someone over assumes your abilities that you should refrain from correcting them.
That is, it benefits the NSA that people think they are actively recording all of their conversations all the time because it forces compliance without the necessary competence, but the people who hold these opinions are often wholly ignorant of the kind of technology required to achieve that level of surveillance.
Have you built your own minigpt? Have you implemented rudimentary transformers?
Are you projecting your desires onto something wholly unworthy of your devotion?
Because the people behind these things are financially incentivized to nod along as your impart more ability than what they know they put into them.
For clarity, my “religion” is math. I believe existence fundamentally is a mathematical construct and as such so are all of its creations.
The brain is to me a mathematical byproduct, but even still, when I familiarized myself with the math of llms and their abilities I recognized that they fall short of being, simulating, or explaining the former.
Llms are stochastic next token pickers, full stop.
Any perceived “intelligence” is projection and anthropomorphising by the agent using them.
I saw a comment on here in another thread stating that the capacity for coherent use of language falls short of being evidence of “intelligence” as children show signs of human “intelligence” long before they can form coherent sentences.
No, but did follow along to an Andrej Karpathy video along those lines at the beginning of the year.
I didn't want to make a judgement on any kind of superiority, or that LLMs simulate brains, or anything of that nature. Just wanted to question why these elements (namely intelligence and reasoning) strike the nerve that they do.
The anthropomorphism argument is case in point, really. It poses the accusation that the other side is imparting human qualities to a machine, without needing to touch on what makes those qualities human or why that matters in the first place. It is, ironically enough, flawed reasoning.
It used what I told it both in the original case, and gave me reasoning for why not using it much was a decent choice (and I verified that it was right), and showed me with an example that demonstrated it was able to reason about how my feedback related to the original answer and apply it. Later it went on, as a result of a subsequent question, and fleshed out the rest of the process. Everything it gave me worked.
To me that is a clear example that while it certainly fails to apply concepts fairly often (and often writes broken code), in other cases it does well. I'll add that this was after I'd spent some time searching for examples and I found nothing like what I suggested and I was about to resign myself to a slog through a lot of really bad documentation, and searching for some of what it suggested afterwards as well made it clear it did not just crib from training data.
For me, this is an example of it reasoning better about the subject than a whole lot of people I found discussing this subject in forum posts I came across, who often made mistakes the code it gave me did not or made assumptions that the code ChatGPT gave me made clear were wrong (as I could verify from the fact it worked)
On the other hand it struggles with something as simple as addition of large numbers that a determined child could do.
Nobody will claim it's consistently reasoning well. But I also regularly see it reason better than a lot of people I know about specific subjects, and so it's exasperating to see people dismiss individual examples of failure as evidence it "cannot apply concepts properly" rather than as individual datapoints.
People both over- and under-estimate how well it can reason based on the types of problems they put to it, and it's certainly an interesting subject how to gauge an "alien intelligence" like this that is so uneven in areas where we expect a relatively even basis and so have all kinds of heuristics for whether someone "knows".
This is part of the problem: We've all gone through a childhood and while we've picked up different things, we mostly have a shared floor that is relatively even across a wide range of basic skills. LLMs don't have that, and that messes with peoples heads. Those of us who have gone into skilled professions similarly have all kinds of preconceptions about what a junior or senior developer looks like, for example, and LLMs do not fit neatly into those boxes.
They're dumb as small children in some areas, but still talk confidently about those subject as if they were an educated adult. That is a challenge and a problem. But that doesn't mean they're not able to reason about other subjects. Just not all of them.
Fraught, of course, with the same problems.
You just end up with everyone being confused and both the author and commenters talking about completely different things.
In this article I feel like we have: 1. Phenomenological consciousness: Does the GPT experience things or is it a P Zombie? Is GPT perceiving the world, or just processing data? Experiencing/Perceiving in this context means seeing "red" not just processing picture and reacting to it. Does it experience the qualia of red or does the data just go through it and you get and output at the end, regardless of how sophisticated is it.
Nobody knows, you can't even reliably prove that you, dear reader, are not the only person in the world who has it.
There's a good example of how ridiculously hard is it and how can we not even talk about these things. Try to establish whether another person sees the same color palette, or is this person's palette inverted? Is your red the same as the other person's red? Absolutely no way to give a definitive answer.
2. Self awareness: Is GPT capable of behaving as if it would be capable of seeing itself as an entity? Yes. It can treat itself as an entity in conversations. Now where we draw a line in terms of memory is in my opinion just semantics. It loses all the memory once you open a new chat window, but people with dementia also lose memory. It's all semantics here and where does your gut feeling draw the line.
"Reasoning means being able to put those concepts together to solve problems."
There's more to reasoning than just following rules of logic ("putting concepts together"). It is also detection where the concepts cause contradictions and do not fit, and the whole mysterious magic of how to modify the concepts to make them fit.
In the first meaning of "reasoning", AI (and computers) have been able to reason for a long time. It's the second meaning that evades us.
I said before that in the 90s, cutting edge AIs were based on various theories of how to do reasoning under uncertainty (fuzzy logic, bayesian networks, etc.). Then deep NNs blew these systems out of the water in practice, but at the expense of us not understanding how they reason with uncertainty, and if there is any consistency to it. So we progressed, but didn't reconcile this problem, what is the right way to reason with uncertainty, and it might just be very very hard. (That's why I am interested in P vs NP, as I believe there is an answer there.)
That is the crux. It is doing something to anticipate words beyond the next token. It has to, to construct these long coherent documents. Just like humans do.
Just because we don't understand it doesn't mean it isn't reasoning. Isn't 'thinking ahead'.
Just like I can say we don't understand the brain, thus humans aren't actually reasoning.
There are a lot of brain studies that look into the pre-cursor changes in the brain pre-ceding conscious thought.
We can't say NN aren't doing something similar.
I agree, but my point was different. We can't say what it does theoretically, therefore we don't know how reliable it is (we don't understand the tradeoffs and failure modes). At least most humans have a way to assess their own reliability, and can see where their reasoning (or of their fellow humans) is inconsistent.
Or put another way a square explicitly might be four straight lines connected at right angles, but in reality such a perfect shape is never going to exist. What's important is that the system understands that shape which has roughly straight lines and roughly connected and right angles is "squarish" enough to be a labeled a square, and the less "squarish" the shape becomes the less certain the system becomes that square is the correct label. Neural networks certainly achieve this.
We might not always understand what the parameters of a neutral networks are encoding, but that's a limitation of our brains, not of neural networks.
Neural networks can be modeled with probability, but that doesn't mean they actually compute in that way. Just like with humans - we can see brain often follows things like Bayes rule, but it doesn't compute PDFs. Doing full probability reasoning would be too expensive for NNs to do, so they cut corners somewhere, and we don't really understand where, it might be very inconsistent. It often works but also often fails.
I don't want to sound pessimistic and I may be missing something, but this seems a bit like trying fork the Edge source code to build a better browser or to come up with your own version of MS Office for Windows..
Edit: This popped into my mind after recently seeing some people having made their own mp3 players and photo browsers with AI code generators and then going on to say how awesome it is to have those programs made just for them and their unique preferences.
You could imagine a thought experiment where a human is woken up in a sensory deprivation tank, asked a single question, and then having their short term memory wiped. It would still be a conscious experience.
I’m not sure if LLM’s are conscious or not but it just doesn’t seem like a compelling argument.
I see that you're seeing the training phase of GPT as the equivalent time the adult had before, but I don't think that works. Part of what makes a consciousness emerges for me is understanding your place in the world, how your actions are having consequences etc. Being nice can have people be nice to you in return, or depending on the environment, you can learn that by being insuffurable people will just give you whatever you want (cf some toddlers). GPT's training comes short of that. If you had an actual session of GPT connected to some sort of long term vector database where it can store what it learns based on conversations, to differenciate individuals and itself etc. that might give it a fighting chance to develop an equivalent of a conscience.
Should be corrected in a few seconds whilst CI does its job
Could there be a new wave of adding speelling errors, grammer errors, to prove you aren't GPT?
[citation needed]
