If you mean "guessing without grounds", that is exactly the phenomenon which is expressed by bad thinkers in both the carbon and the silicon realms, and that is what we are countering.
> predict[ing] "facts"
It's called "Science". In a broader way, it's called "intelligence" ("Intelligence is being able to predict the outcomes of an experience you never had" ~~ Prof. Patrick Winston)
> Getting facts right is a matter of
It is a matter of procedurally adhering to an attitude of iterative quality refinement of ideas, and LLMs seem to be dramatically bad at "procedures".
"Science" is coming up with, and testing, theories -- they may be true, they may be false, and you can't know, and shouldn't hold a very strong position, until you test them. It's true that a more intelligent person will come up with better hypotheses and more inventive ways to put them to the test, but that's not what you seemed to be talking about, nor are we in any disagreement on that point.
A more intelligent cat will also catch mice more effectively -- it'll have a more accurate mental model of the mouse and of its own physical capabilities in time and space. Still, the outcome of the hunt is never perfectly predictable. Some outcomes are statistical -- and, intriguingly, LLMs mirror this in how they predict tokens.
> LLMs seem to be dramatically bad at "procedures".
How do you figure, and how did you reach this conclusion?
And Michelson and Morley did through Einstein's theory. And Jack did when he said "if my theory is correct, that falling brick will break my skull more probably than not". And it's a matter in which LLMs tend to fail, when they go "surely your operating system will have a `scratchmyback` command to allow you to work more hours sitting in front of it, it just makes sense".
> How do you figure [that «LLMs seem to be dramatically bad at "procedures"»], and how did you reach this conclusion?
I just tried with a main widespread engine, and it failed. And it showed that it still seemed to be guessing an output instead of actually checking to build the output (as if remembering that very often "2+2=4" instead of checking "1 and 1, and 1 and 1: 1, 2, 3, 4").
Forget Einstein for a minute. When you drive a car, you hold a mental model of your position and velocity in time and space, of the expected behaviors of other drivers, of the conditions of the road, and you continually adjust your behavior in accordance with that model. Almost anything that requires attention is something that requires us to build a mental model of the future -- and predict that future.
So, yeah, you can hew closely to validated scientific theories and "predict" how things will happen in that sense. But, as you walk home from your meeting at the astronomical society, you stop at a crosswalk, look both ways, and you're back to making essentially probabilistic predictions about how crossing the road is going to go.
I get the sense that you dislike them, but really LLMs are not so different. How they handle probability and prediction is different in degree, but I don't think that it's entirely different in kind.
> And it showed that it still seemed to be guessing an output instead of actually checking to build the output (as if remembering that very often "2+2=4" instead of checking "1 and 1, and 1 and 1: 1, 2, 3, 4").
You've never memorized your multiplication tables?
Boss Terry Tao has a reasonably high opinion of the abilities of LLMs as mathematicians, which is remarkable -- really astounding -- considering how they're built and trained, as essentially language prediction and manipulation machines.
I must stress that the idea of "Science predicting facts" is a consolidated formula in Philosophy of Science.
And there has never been a doubt that prediction is probabilistic. But, see the example in in the parallel additional post about "dreaming and wake", the predicting activities of a junkie under psychedelics and that of a lucid thinker are substantially different.
> You've never memorized
You have the framework very very wrong: the point is not that we memorize, the point is that those LLMs don't check. When you state an idea, you are supposed to have checked it in other occasions before memorization.
Procedural operations, of which counting is just an example, can fail in those LLMs, which means they are simulating it instead of doing it, which suggests that they «seem to be guessing an output instead of actually checking to build the output», which makes them structurally untrustworthy, unreliable - broken by design.
Being black boxes (bad), they must be stress tested to see whether proper functioning is present or just simulated: the chief problem is not that they can't count, it is that they must be missing the roots of counting: procedural lucid thinking.
Check the parallel submission about the detective game ("Temporal Clue")*: an algorithm that cannot fully reason with a lucid world model, solving logic puzzles, is unreliable. The probabilistic nature of the architecture in this case is below the intelligent, as opposed to the sophistication of considering less probable unexpected branches of possibilities.
current Neural Network architectures seem to perform in a dreamlike state in which "oh in that area there should be a piece of finger this way oriented";
humans also have a wake state module in which they count them fingers.
These NNs seem to dream; we can be awake.
