However, I do think we'll continue to see impressive advances in the areas of media consumption and production, with complex reasoning on hard problems being a likely area of improvement in the near (1 decade) future. While I once never expected to see something like HAL in my lifetime, I feel that many aspects of HAL (voice recognition, ship automation, and chess-playing) have been achieved, if not fully integrated into a single agent. We can expect most applications to be banal- the giants who have the largest data piles will train models that continue to optimize the addictivity of social media, and click-thru rates of ads.
I am also quite impressed at the recall of information by language models for highly factual and well-supported things (computer reviews in particular).
I think it's impressive that we went from LLMs not being useful at all to GPT3.5 shocking the world to GPT4 becoming super useful for many things in around 7 months time.
LLM progress have slowed down a bit. But I think we're just getting started. It's still really early. It's only been 1 year since GPT4 came out. Even at the level of GPT4, scaling it would have immense benefits. But my sense is that we'll have a few more levels of great leaps in LLM capabilities that will shock people in the next 3-4 years.
I'm referring to the OpenAI white paper on GPT4 that shows exam results. https://cdn.openai.com/papers/gpt-4.pdf figure 4, and surrounding text.
Clearly not superintelligence, as I would define it (see my other comment about scientific discovery, which I consider the best test), but these are tests actual humans take, and we know how most humans would score on these tests (thru some amount of memorization, along with parsing word problems and doing some amount of calculation). But many people who looked at the testing results concluded that GPT-4 was actually a reasoning agent, or that reasoning agents were just around the corner.
The press picked up on that, and my LinkedIn stream was absolutely filled with second-class influencers who thought that superhuman capabilities were not far away. For a while there, looking at some of the test results specifically on moderately challenging math problems, I suspected that LLMs had some sort of reasoning ability.
Why? Can you see the future? No one (serious) was claiming that GPT-4 is superintelligence, it’s about the rate of improvement.
There has only been 6 years between GPT-1 and GPT-4, and each iteration brought more and more crazy emergent behaviour. We still don’t see any sign of the scaling laws slowing down.
I work in ML research, and personally don’t believe ASI is just around, but I talk everyday to researcher that believe so, they don’t say that to swindle anyone’s money (they have extremely well paid 9 to 5 jobs at Goog/MS/OAI, they aren’t trying to raise money from VCs), they only believe so due to the rate of improvement.
Claiming, barely 18 months after GPT-4, when we haven’t yet seen any result from the next jump in scale, that it’s all baloney is a bit premature.
Btw in research time, 10 years from now is « around the corner ».
Now for the VC-chasing folks, their motivation is an entirely different story.
I just continue to think that Vinge was a bit optimistic both on the timeline and acceleration rate. Everybody who cares about this should read https://edoras.sdsu.edu/~vinge/misc/singularity.html and consider whether we will reach the point where ML is actively improving its own hardware (after all, we do use ML to design next gen ML hardware, but with humans in the loop).
is a bit vague as to how long round the corner is and which folk you are thinking of but there have been very non charlatan predictions that you'd be getting something like HAL around now based on the Moore's law like improvements in hardware performance which have kept going for a century and are currently accelerating due to the vast amounts of cash being thrown in.
Probably the best of them in terms of reasoned thinking and being ahead of the curve is Hans Moravec, a robotics guy at the Robotics Institute of Carnegie Mellon who argued computers would be reaching this point around now in his 1988 book - graph here https://imgur.com/a/moravec-graph-V3S2XoK and there's more detail in his 1998 paper https://jetpress.org/volume1/moravec.pdf
The reasoning is very down to earth based on his research attempts at robot vision and comparing the hardware needed to that of the retina - not much hucksterism.
In the graph in the paper on page 5 or so he has computer power roughly going from the equivalent of a lizard to a monkey to a human over about two decades so going by that and assuming they are around human level now they should be about monkey to human level better than us in a decade or so. Not sure if that counts as superintelligence around the corner? This is all independent of which particular algorithms are used.
In actuality, humans are still needed for the 10% the robots can't do well, or serve to enhance the productivity of humans.
I predict AI is like this and going to be for a while - it can clearly do some stuff well and sometimes better than humans, but humans will have their niches for a while.
Personally, my area of interest is scientific discovery. Could a model not dissimilar from what we have today, if asked a cogent question, not answer it with an experiment that could be carried out? For example, one of the most important experiments, Avery-MacCleod, which proved (to the extent that you can prove anything in biology) that DNA, not protein, was the primary element of heredity, is not all that complicated, and the mechanical details seem nearly in reach of modern ML techniques. Similarly, could the ML model provide a significant advance in the area of understanding the molecular function in intimate detail of proteins as determined by their structure (which AlphaFold does not do, yet), complete with experimental instructions on how to verify these hypotheses? As of this time, my review of modern ML methods for science suggest we have made some advances, but still have not passed the "phase transition" demonstrating superscientist-level understanding of any field. But perhaps it will just fall out naturally from improved methods for media generation/parsing and ad targeting.
I continue to remain hopeful that within my remaining 20-40 or so years (I'm a typical american male, age 51, with a genome that contains no known risk factors) I will see something like what Vinge describes in https://edoras.sdsu.edu/~vinge/misc/singularity.html in a way that is demonstrable and safe, but honestly, I think it could go in any number of directions from "grim meat-hook future" to "unexpected asteroid takes out human life on the planet, leaving tardigrades to inherit the earth" to "kardyshev-scale civilization".
Do you think Sutskever,Hinton or Sutter are charlatans?
It's as if the easiest people to fool are the researchers themselves
If I wanted to predict the next ten years, I'd bring in Demis Hassabis, Noam Shazeer, and Vincent Vanhoucke, from what I've read of Demis's work, and my interactions with the latter, they seem to have very realistic understanding and are not prone to hype (Demis being the most ambitious of the three, Vincent being the one who actually cracked voice recognition, and Noam because ... his brain is unmatched).
Similar to containers, my feeling is that the truth is LLMs are wildly overkill for almost everything going on today. You don’t need an LLM to sort some already structured data when a basic python library or something will work equally fast, predictably, and with less black box magic. There’s probably a small number of use cases that it makes sense for but for everyone else it’s just silly to try and force the technology. It doesn’t help that the people who are selling the shovels in this gold rush are extremely good at extending the hype train every few months, but eventually when these models stop being sold at a loss and businesses have to start facing down with the bill to run them and/or make these API calls, it will correct itself real fast.
I don't have much belief in fully autonomous generative AI agents performing more complex tasks any time soon. It's a significant productivity boost for some jobs, but not a total replacement for humans who do more than read from a script, or write clickbait articles for media.
Right now there are many small but decent models available for free, and cheap to use. If it wasn't for the hype, it would never have reached that level of optimization. Now we can make decent home assistants, text parsers and a bunch of other stuff you already mentioned.
But someone paid for that. The companies who believed this would be revolutionary will eventually have a really hard reality check. Not that they won't try and use it for critical stuff, but once they do and it fails spectacularly they will realize a lot of money went down the drain.
The newer models are 10x faster and cheaper, therefore synthetic data is 10x cheaper to make now.
If the ARC challenge makes an impact, there's a good chance the next generation AI will need a lot less data.
LLMs and AGI might be hogwash, but processing multimedia is where Gen AI and especially diffusion models shine.
Furthermore text-to-{whatever} models might produce slop, but Gen AI "exoskeletons" (spatial domain, temporal domain editors) are Photoshop and Blender from next century. These turbocharge creatives.
Hearing and vision are simple operations relative to reasoning. They're naturally occurring physical signals that the animal kingdom has evolved, on several different occasions, to process. This is likely why they're such a low hanging fruit to replicate with Gen AI.
Perhaps the difference between insanity and visionary, "scam" and genius is simply the outcome.
When someone like Sam Altman declares optimistically that we will get AGI and talks about what kind of society we will need to build... It's kind of hard to tell what mix of those 4 is at work. But certainly it will be perceived differently based upon the outcome not the sincerity of the effort.
I'm not convinced this is true. What is irrational about the possibility of e.g. scientific progress, inventing new products, or creating a viable business?
Irrational belief may be one way to motivate yourself to try those things. But it's not the only way. Calculated risk-taking isn't irrational, is it?
Please don't post wayback links unnecessarily. Content still fresh and available.
Discussion here: https://news.ycombinator.com/item?id=40856329
Although, I'd be shorting 80% of everyone else spending money with NVidia without a clear path to recover. However, given that most are (likely?) not listed, there isn't that much to short?
It's mostly speculative narrative with a fair number of data-driven charts. I wouldn't spend much time on it unless you like financial analysis with hand-waving.
