Turns out that when resources are scarce, the optimal move is to knock the opponent away. I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem.
But if advanced AI is being developed in a capitalist economy by independent actors, it seems most likely the incentives will be anything other than optimizing the output for the individual outcome.
If that AI finds a way to "hurt" the other actor, there could be some major boat load of unintended consequences.
My reading is that the point of this research is to find out what problem spaces are conducive to cooperation, not to find out details of how these particular agents work.
[0] http://www.sciencealert.com/google-s-new-ai-has-learned-to-b...
Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate survival, it needs to have higher ideals than winning, just like humans. It needs to be able to see and be aware of the big picture. You don't need to look at AI for that, you can just look at the world.
Development of AI's of this nature will just lead to a super-powered Moloch. Cooperative ethics is a highly advanced concept, it's not going to show up on its own from mere game theory without a lot of time.
I think we shouldn't confuse efficient strategies with the chosen strategies. What causes Moloch is the inability to see the big picture, to see outside of the self in the collective (maybe Buddhism has a point).
An efficient strategy may very well be something we'd prefer, such as tit-for-tat. But is that the strategy we choose? Looking at the long history of evolution, I'd say no.
I'm not saying that Trade Is The Answer. I would be somewhat surprised if it doesn't form some of the solution eventually, but that's not the argument I'm making today. The argument I'm making is that if the simulation can't simulate trade at all, that's a sign that it may have been too simplified to be useful. There are probably other things you could say that about; "communication" being another one. The only mechanism for communication being the result of iteration is questionable too, for instance. Obviously in the real world, most cooperation doesn't involve human speech, but a lot of ecology can be seen to involve communication, if for no other reason than you can't have the very popular strategy of "deception" if you don't have "communication" with which to deceive.
Which may also explain the in-my-opinion overpopular and excessively studied "Prisoner's Dilemma", since it has the convenient characteristic of explicitly writing communication out of it. I fear its popularity may blind us to the fact that it wasn't ever really meant to be the focus of study of social science, but more a simplified word problem for game theory. Studying a word problem over and over and over may be like trying to understand the real world of train transportation systems by repeatedly studying "A train leaves from Albuquerque headed towards Boston at 1pm on Tuesday and a train leaves from Boston headed towards Albuquerque at 3pm on Wednesday, when do they pass each other?" over and over again.
(Or to put it really simply in machine learning terms, what's the point of trying to study cooperation in systems whose bias does not encompass cooperation behaviors in the first place?)
In iterated prisoners dilemma and other similar games, the "API" with which agents interact with the world is extremely simple. The statement of the problem is also very simple. The agent itself can be any computable algorithm for deciding to cooperate or defect based on the past history of game rounds. I find it interesting to see agents learn recognizable behaviours like "communication" or "trade" when they aren't explicitly programmed to do those things.
Whenever I think I've finally gotten a handle on the state-of-the-art in AI research, they come up with something new that looks really interesting.
They're now training deep-reinforcement-learning agents to co-evolve in increasingly more complex settings, to see if, how, and when the agents learn to cooperate (or not). Should they find that agents learn to behave in ways that, say, contradict widely accepted economic theory, this line of work could easily lead to a Nobel prize in Economics.
Very cool.
It's just a matter of time before it floods the Enrichment Center with deadly neurotoxin.
But I do wonder if an even more intelligent AI (perhaps in a more complex environment) would take the long view instead and find a reason to co-habitate.
It's kind of like rocks, paper scissors - when you attempt to think several levels deeper than your opponent and guess which level they stopped at. At some intelligence level for AI, cohabitation seems optimal - at the next level, not so much, and so on.
We're probably going to end up building something so complex that we don't quite understand it and end up hurting somebody.
I think this is a kind of strong statement to take as a given, especially as an opening. This is taking social darwinism as law, and could use more scrutiny.
> sequential social dilemmas, and us[ing] artificial agents trained by deep multi-agent reinforcement learning to study [them]
Don't expect any crazy deep insights, but it's a useful read if you want to set up a similar experiment or understand the research methodology.
In a game where you are given the choice of killing 10,000 people or be killed yourself, which is the most rewarding outcome?
Basically, it learned that it didn't need to fight until there was resource scarcity in a simulation.
Two racers are competing to see who runs faster, but if one pulls out a laser gun and shoots the other, that's aggressive.
Actually, it's an interesting word. Dictionary definitions of aggression frequently revolve around emotions - it's a very human word, probably not suitable for AI.
Edit: unfortunately you've been doing this a lot. We ban accounts that do this, so please stop.
"Understanding Agent Cooperation" https://news.ycombinator.com/edit?id=13635218