undefined | Better HN

0 pointsgonehome5y ago0 comments

> More likely it has memorised some partial multiplication tables.

Did you read what I linked? (I don't intend this to be hostile, but the paper explicitly discusses this.) They control for memorization and the errors are off by one which suggest doing arithmetic poorly (which is pretty nuts for a model designed only to predict the next character).

(pg. 23): ”To spot-check whether the model is simply memorizing specific arithmetic problems, we took the 3-digit arithmetic problems in our test set and searched for them in our training data in both the forms "<NUM1> + <NUM2> =" and "<NUM1> plus <NUM2>". Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000 subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers could have been memorized. In addition, inspection of incorrect answers reveals that the model often makes mistakes such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than memorizing a table.”

> The paperclip maximizer idea still assumes that the AI has an extremely intricate understanding of the physical and human worlds - much better than any human's. My point was that there is no way at the moment to know if this is possible or not.

It seems less likely to me that biological intelligence (which is bounded by things like head size, energy constraints, and other selective pressures) would happen to be the theoretical max. The paperclip idea is that if you can figure out AGI and it has goals it can scale up in pursuit of those goals.

> I'm not claiming that AGI risk in some far future won't be a real problem. My claim is that it is as silly for us to worry about it as it would have been for Socrates to worry about the effects of 5G antennas.

I think this is a hard claim to make confidently. Maybe it's right, but maybe it's the people saying the heavier than air flight is impossible two years after the Wright brothers flew. I think it's really hard to be confident in this prediction either way, people are famously bad at this.

Would you have predicted gpt-3 kind of success ten years ago? I wouldn't have. Is gpt-3 what you'd expect to see in a world where AGI progress is failing? What would you expect to see?

I do agree that given the lack of clarity of what should be done it makes sense for a core group of people to keep working on it. If it ends up being 100yrs out or more we'll probably need whatever technology is developed in that time to help.

0 comments

tsimionescu5y ago

> Did you read what I linked? (I don't intend this to be hostile, but the paper explicitly discusses this.) They control for memorization and the errors are off by one which suggest doing arithmetic poorly (which is pretty nuts for a model designed only to predict the next character).

I read about this before. I must confess that I had incorrectly remembered that they had only checked a few of their computations for their presence in the corpus, not all of them. Still, they only check for two possible representations, so there is still a possibility that it picked up other examples (e.g. "adding 10 with 11 results in 21" would not be caught - though it's still somewhat impressive if it recognizes it as 10 + 11 = 21).

> It seems less likely to me that biological intelligence (which is bounded by things like head size, energy constraints, and other selective pressures) would happen to be the theoretical max. The paperclip idea is that if you can figure out AGI and it has goals it can scale up in pursuit of those goals.

Well, intelligence doesn't seem to be so clearly correlated with some of those things - for example, crows seem to have significantly more advanced capabilities than elephants, whales or lions (tool use, human face memorization). Regardless, I agree that it is unlikely that humans are a theoretical maximum. However, I also believe that the distribution of animal intelligence to brain size may suggest that intelligence is not simply dependent on the amount of computing power available, but on other properties of the computing system. So perhaps "scaling up" is not going to be a massive growth in the amount of intelligence - that you need entirely different architectures for that.

> Would you have predicted gpt-3 kind of success ten years ago? I wouldn't have. Is gpt-3 what you'd expect to see in a world where AGI progress is failing? What would you expect to see?

I don't think GPT-3 is particularly impressive. I can't claim that I would have predicted it specifically, but the idea that we could ape human writing significantly better wouldn't have seemed that alien to me I think. GPT-3 is still extremely limited in what it can actually "say", I'm even curious if it will find any real uses that we don't already outsource as brain-dead jobs (such as writing fluff pieces).

And yes, I do agree that this is a problem worth pursuing, don't get me wrong. I don't think lots of AI research is going in the right way necessarily, but some is, and some neuroscience is also making advances in this area.

j / k navigate · click thread line to collapse