undefined | Better HN

0 pointsalexb_2y ago0 comments

I'm saying more that it's "interesting" in an argumentative way, not that it's actually of interest about how this is about EY. It should be interesting to followers of EY because all of these traits are things that cult leaders do, and are things that EY does, because EY is (intentionally or not) running a doomsday cult. It's impressive how much reach he's been able to get, to the point where his ideas of control and morality have seeped into the most important technology of our time, but it's still a cult founded on shaky conjecture nonetheless. I think it's pretty clear if you are interested in AI who this article is about, although I do agree it should have been much more explicitely followed up with "Here's how EY is these things".

It's made me very sad to see extremely smart people, who I once looked up to and really found very interesting (you're included in here!) fall victim to the irrational fear of AI spurred on by Eliezer. You can create as much literature as you want, and you can create as many hypothetical scenarios as possible, but it's not going to change the fact that dedicating your life into controlling AI so that you can "logically" justify your own life is nonsense.

It's not that I don't get it. I just disagree. I don't think AI will create a science fiction horror show to end all of humanity. I don't think AI will cause humans to stop being humans. I don't think AI will create the end of the world. Yes, I am not 100% sure. There is a 0.01% chance of this all happening. But that doesn't mean that we need to pay attention to it just because multiplying 0.0001 by 1,000,000 yields a big number. I think basing your morality and existence on trolley problems and mental experiments is abhorrent. And no, I can't give you an exact spelled out reason as to why. Do I have to justify basic human morality?

You're an extremely smart person. I love your blog posts, and I love the depth and time you put into them. It's incredible how disappointing it is for me to see you become another "rationalist" obsessed with a fantasy of AI doom.

Edit for your edit: Someone who is wealthy and has nothing to lose quitting his job to join the cult of AI doomerism is, in no way, proof that AI xrisk is meaningful. You're using someone else coming to a conclusion as a reason you are correct. I trust you are smart enough to know why this is awful logic.

0 comments

gwern2y ago

> It should be interesting to followers of EY because all of these traits are things that cult leaders do

It's not really because it's glib pattern-matching. It's roughly "Is the NIH a cult?" https://www.michaeleisen.org/blog/?p=1217 level. Go compare Eliezer to say, NXIVM, to see what a real cult looks like.

> It's made me very sad to see extremely smart people, who I once looked up to and really found very interesting (you're included in here!) fall victim to the irrational fear of AI spurred on by Eliezer. You can create as much literature as you want, and you can create as many hypothetical scenarios as possible, but it's not going to change the fact that dedicating your life into controlling AI so that you can "logically" justify your own life is nonsense...You're an extremely smart person. I love your blog posts, and I love the depth and time you put into them. It's incredible how disappointing it is for me to see you become another "rationalist" obsessed with a fantasy of AI doom.

I feel I should point out that you have not seen me 'become another rationalist' because this is what I have been since around 2004, long before I started any writing or even was 'gwern'. 'AI doom' has always driven my writing (even if I sometimes wander quite far afield before eg. concluding that nootropics are a dead end for intelligence augmentation that might help with AI risk - on the bright side, I think DL scaling may have helped answer why nootropics have been such a bust in general). I just thought we'd have way more time.

> Someone who is wealthy and has nothing to lose quitting his job to join the cult of AI doomerism is, in no way, proof that AI xrisk is meaningful. I trust you are smart enough to know why this is awful logic.

'some wealthy dude' is not the relevant description here for Geoff Hinton. I trust you are smart enough to see the problem with your even glibber comment there.

rolisz2y ago

Yes, a 100 times this! Thank you for putting this so nicely into words! Too bad there's no HN gold or whatever for me to award you!

j / k navigate · click thread line to collapse