It seems like an incredibly bad outcome if we accept "AI" that's fundamentally flawed in a way similar to if not worse than humans and try to work around it rather than relegating it to unimportant tasks while we work towards a standard of intelligence we'd otherwise expect from a computer.
LLMs certainly appear to be the closest to real AI that we've gotten so far. But I think a lot of that is due to the human bias that language is a sign of intelligence and our measuring stick is unsuited to evaluate software specifically designed to mimic the human ability to string words together. We now have the unreliability of human language processes without most of the benefits that comes from actual human level intelligence. Managing that unreliability with systems designed for humans bakes in all the downsides without further pursuing the potential upsides from legitimate computer intelligence.
Also, I think it’s apparent that the world won’t wait for correct AI, whatever that even is, whether or not it even can exist, before it adopts AI. It sure looks like some employers are hurtling towards replacing (or, at least, reducing) human headcount with AI that performs below average at best, and expecting whoever’s left standing to clean up the mess. This will free up a lot of talent, both the people who are cut and the people who aren’t willing to clean up the resulting mess, for other shops that take a more human-based approach to staffing.
I’m looking forward to seeing which side wins. I don’t expect it to be cut-and-dry. But I do expect it to be interesting.
Just tried it:
tell me the current date please
Today's date is October 3, 2023.
Most real actual human people would either know the date, check their phone or their watch or be like "Oh, that's a good question lol!". But somehow GPTs always be the 1% of people that will lie to know the answer to whatever question you ask them. You know, the kind that evening talk shows will ask ask. Questions like "how do do chickens lay eggs" and you get all sorts of totally completely b0nkers but entirely "confidently told" answers. And of course they only show the ones that give the b0nkers con-man answers. Or the obviously funnily stupid people.
Of course absent access to a "get the current date" function it makes sense why an LLM would behave like it does. But it also means: not AGI, sorry.
We like to pretend humans can reliably execute basic tasks like telling left from right or counting to ten, or reading a four digit number, and we assume that anyone who fails at these tasks is "not even trying"
But people do make these kinds of mistakes all the time, and some of them lead to patients having the wrong leg amputated.
A lot of people seem to see fault tolerance as cheating or relying on crutches, it's almost like they actively want mistakes to result in major problems.
If we make it so that AI failing to count the Rs doesn't kill anyone, that same attitude might help us build our equipment so that connecting the red wire to R2 instead of R3 results in a self test warning instead of a funeral announcement.
Obviously I'm all for improving the underlying AI tech itself ("Maintain Competence" is a rule in crew resource management), but I'm not a super big fan of unnecessary single points of failure.
You've just explained "race to the bottom". We've had enough of this race, and it has left us with so many poor services and products.
People’s unawareness of their own personification bias with LLMs is wild.
Compare that to the weight we place on "experts" many of whom are hopelessly compromised or dragged by mountains of baggage.
So I'll leave it to Skeeter to explain.
On the computer side of things, I think at a minimum I'd want intelligence capable of taking advantage of the fact that it's a deterministic machine capable of unerringly performing various operations with perfect accuracy absent a stray cosmic ray or programming bug. Star Trek's Data struggled with human emotions and things like that, but at least he typically got the warp core calculations correct. Accepting LLMs with the accuracy of a particularly lazy intern feels like it misses the point of computers entirely.
What is most characteristic about human intelligence is the ability to abstract from particular, concrete instances of things we experience. This allows us to form general concepts which are the foundation of reason. Analysis requires concepts (as concepts are what are analyzed), inference requires concepts (as we determine logical relations between them).
We could say that computers might simulate intelligent behavior in some way or other, but this is observer relative not an objective property of the machine, and it is a category mistake to call computers intelligent in any way that is coherent and not the result of projecting qualities onto things that do not possess them.
What makes all of this even more mystifying is that, first, the very founding papers of computer science speak of effective methods, which is by definition about methods that are completely mechanical and formal, and this stripped of the substantive conceptual content it can be applied to. Historically, this practically meant instructions given to human computers who merely completed them without any comprehension of what they were participating in. Second, computers are formal models, not physical machines. Physical machines simulate the computer formalism, but are not identical with the formalism. And as Kripke and Searle showed, there is no way in which you can say that a computer is objectively calculating anything! When we use a computer to add two numbers, you cannot say that the computer is objectively adding two numbers. It isn’t. The addition is merely an interpretation of a totally mechanistic and formal process that has been designed to be interpretable in such ways. It is analogous to reading a book. A book does not objectively contains words. It contains shaped blots of pigment on sheets of cellulose that have been assigned a conventional meaning in a culture and language. In other words, you being the words, the concepts, to the book. You bring the grammar. The book itself doesn’t have them.
So we must stop confusing figurative language with literal language. AI, LLMs, whatever can be very useful, but it isn’t even wrong to call them intelligent in any literal sense.
One of my first teachers said to me that a computer won't ever output anything wrong, it will produce a result according to the instructions it was given.
LLMs do follow this principle as well, it's just that when we are assessing the quality of output we are incorrectly comparing it to the deterministic alternative, and this isn't really a valid comparison.
5 is exactly halfway, that's not random enough either, that's out.
2, 4, 6, 8 are even and even numbers are round and friendly and comfortable, those are out too.
9 feels too close to the boundary, it's out.
That leaves 3 and 7, and 7 is more than 3 so it's got more room for randomness in it right?
Therefore 7 is the most random number between 1 and 10.
People tend to avoid extremes, too. If you ask for a number between 1 and 10, people tend to pick something in the middle. Somehow, the ordinal values of the range seem less likely.
Additionally, people tend to avoid numbers that are in other ranges. Ask for a number from 1 to 100, and it just feels wrong to pick a number between 1 and 10. They asked for a number between 1 and 100. Not this much smaller range. You don't want to give them a number they can't use. There must be a reason they said 100. I wonder if the human RNG would improve if we started asking for numbers between 21 and 114.
My favorite is:
No one is as dumb as all of us.
* Pseudo Intelligence
From that follows that LLMs fit to produce all kinds of human biases. Like preferring the first choice out of many, and the last our of many (primacy biases). Funnily the LLM might replicate the biases slightly wrong and by doing so produce new derived biases.
In most cases, The LLM itself is a name-less and ego-less clockwork Document-Maker-Bigger. It is being run against a hidden theater-play script. The "AI assistant" (of whatever brand-name) is a fictional character seeded into the script, and the human unwittingly provides lines for a "User" character to "speak". Fresh lines for the other character are parsed and "acted out" by conventional computer code.
That character is "helpful and kind and patient" in much the same way way that another character named Dracula is a "devious bloodsucker". Even when form is really good, it isn't quite the same as substance.
The author/character difference may seem subtle, but I believe it's important: We are not training LLMs to be people we like, we are training them to emit text describing characters and lines that we like. It also helps in understanding prompt injection and "hallucinations", which are both much closer to mandatory features than bugs.
But human expectations are also not bias-free (e.g. from the preferring-the-first-choice phenomenon)
How can the RLHF phase eliminate bias if it uses a process(human input) that has the same biases as the pre-training(human input)?
Together? It would be, 1. AI programmers, 2. AI techbros and a distant 3. AI fiction/history/literature. Foo who never used the internet: not responsible. Bar who posted pictures on Facebook: not responsible. Baz who wrote machine learning, limited dataset algorithms (webmd): not responsible. Etc.
> spits out chunks of words in an order that parrots some of their training data.
So, if the data was created by humans then how is that different from "emulating human behavior?"
Genuinely curious as this is my rough interpretation as well.
Hardly a shocker. I think this say more about the experimental design then it does about AI & humans.
The authors discuss the person 1 / doc 1 bias and the need to always evaluate each pair of items twice.
If you want to play around with this method there is a nice python tool here: https://github.com/vagos/llm-sort
* Comparing all possible pair permutations eliminates any bias since all pairs are compared both ways, but is exceedingly computationally expensive. * Using a sorting algorithm such as Quicksort and Heapsort is more computationally efficient, and in practice doesn't seem to suffer much from bias. * Sliding window sorting has the lowest computation requirement, but is mildly biased.
The paper doesn't seem to do any exploration of the prompt and whether it has any impact on the input ordering bias. I think that would be nice to know. Maybe assigning the options random names instead of ordinals would reduce the bias. That said, I doubt there's some magic prompt that will reduce the bias to 0. So we're definitely stuck with the options above until the LLM itself gets debiased correctly.
The experiment itself is so fundamentally flawed it's hard to begin criticizing it. HN comments as a predictor of good hiring material is just as valid as social media profile artifacts or sleep patterns.
Just because you produce something with statistics (with or without LLMs) and have nice visuals and narratives doesn't mean is valid or rigorous or "better than nothing" for decision making.
Articles like this keep making it to the top of HN because HN is behaving like reddit where the article is read by few and the gist of the title debated by many.
Although of course that behavior may be a signal that the model is sort of guessing randomly rather than actually producing a signal.
The LLM isn't performing the desired task.
It sounds possible to cancel out the comments where reversing the labels swaps the outcome because of bias. That will leave the more "extreme" HN comments that it consistently scored regardless of the label. But that may not solve for the intended task still.
The LLM isn't performing the desired task.
I wouldn't treat the output as complete garbage, just because it's somewhat biased by an irrelevant signal.
Yes and no.
Yes, this is really problem, because at current level of technologies, some thing are inexpensive only if done in large numbers (factor of scale), so for example, just could not exist one person who could be accountable for machine like Boeing-747 (~500 human-years of work per plane).
Unfortunately, modern automobile is considered large system, made from thousands parts, so again, not exist one person to know everything.
And no, Germans said "Ordnung muss sein", which in modern management mean, constant clear organization of the game of the whole team is more important than the success of individual players.
Or, in simple words, right organization, controlled by rules is considered enough reliable to be accountable.
And for example in automobile industry, now normal to consider accountable whole organization.
And for example, Daimler officials few years ago said, Daimler safety systems will use Daimler view on robotic laws - priority will be safety of people inside vehicle. You may know, traditionally used Lem robotic laws, which have totally different view, separated from inside vs outside approach. In civil aviation using approach, to just use simple designs or design with evidence of reliability.
Sure, government regulators could decide something even more original, will see.
Any way, as technology emerge, accountability of machines will be sure subject of many discussions.
Is this a universal phenomenon where you've worked? Consider yourself very lucky.
To me it’s literally the same as testing one Markov chain against another.
It can be incredibly hard to get a person to acknowledge that they might be remotely wrong on a topic they really care about.
Or, for some people, the thought that they might be wrong about anything attall is just like blasphemy to them.
"Acknowledging they might be wrong" makes them sound like more than token predictors trained on polite sounding text.
When you don't do that sufficiently you run the risk of producing the "Sydney" personality that Bing Chat had, which would argue back, and could go totally feral defending its incorrect beliefs about the world, to the point of insulting and belittling the user.
Also, often less capable of carrying on a decent conversation.
I’ve noticed an periconcious urge when talking to people to judge them against various models and quants, or to decide they are truly SOTA.
I need to touch grass a bit more, I think.
TL;DR: the author found a very, very specific bias that is prevalent in both humans and LLMs. That is it.
Now: some people can't count. Some people hum between words. Some people set fire to national monuments. Reply: "Yes we knew", and "No, it's not necessary".
And: if people could lift the tons, we would not have invented cranes.
Very, very often in these pages I meet people repeating "how bad people are". That is "how bad people can be", and "and we would have guessed these pages are especially visited by engineers, who must be already aware of the importance of technical boosts" - so, besides the point relevant to the fact that the median does not represent the whole set, the other point relevant to the fact that tools are not measured on reaching mediocre results.
Those who insist that "all humans are <slur>" are "racist" against humanity (against the "human race", if you wish).
That spirit is in the refusal to see exceptions and to recognize that there can be exceptions.