Learning from Somebot's introduction.
Last week we deployed Somebot to $location. Our confidence in Somebot was high because we had thought about all likely scenarios in our comfortable offices. We are deeply sorry and assume the full responsibility from Somebot's actions. Moving forward we will make sure to include safeguards to reduce the amount of pain caused by Somebot's deployment.
Our deepest condolences to the families of the affected and to the survivors. Megacorp cares about your well being. To help cover expenses from the tragedy we will deposit $money in your Megacorp account.
God bless the federated nations of Megacorp.
Last week we tested ED-209 at our corporate headquarters. Our confidence in ED-209 was high because we had thought about all likely scenarios in our comfortable offices. We are deeply sorry and assume the full responsibility from ED-209's actions. Moving forward we will make sure to include safeguards to reduce the amount of pain caused by ED-209's deployment.
Our deepest condolences to the families of the affected and to the survivors. Omnicorp cares about your well being. To help cover expenses from the tragedy we will deposit $money in your Omnicorp account.
God bless the federated nations of Omnicorp.
Or because they had real world experience with another bot in a similar situation:
"In China, our XiaoIce chatbot is being used by some 40 million people, delighting with its stories and conversations."
They turned it off pretty fast and agree that it was a mistake, what's the problem beyond that?
Could they have anticipated how people would turn the bot into that? Probably. Who says this is not some data they wanted to acquire? I'd happily go through the technical gains with their team any day. Having all this real world input and seeing how the program reacted is such a goldmine of information.
Edit: I did not downvote you. Your point is valid.
Also, a lot of people are saying "well, of course that was obvious". But it really wasn't, because there was a huge team working on this with past experience building a conversational agent. The armchair theorists forget that it always looks obvious to Captain Hindsight.
This is just as stupid as that manufactured outrage over Googles image tagger. It misclassified a picture of a human as an animal, and people were up in arms. Google had to censor it so it can't tag animals now. They shouldn't have to do that, let idiots be idiots.
It isn't about "idiots" or stupid "manufactured outrage"
This is HN; if you are going to invest your time in creating a business the last thing you want to do is blow your market opportunity by associating your brand with something as dumb as penis pics and nazis.
Is it manufactured outrage that Microsoft paid dancers in schoolgirl outfits at GDC? Maybe, but it's irrelevant.
Don't mess up your brand. I have no idea what Microsoft is doing right now. It's like a company going through the equivalent of mid-life crisis trying desperately to be "cool".
It just seems kind of obvious that Twitter users would grief the bot. Now, they address this point, but it's interesting that they still didn't last a day.
Think about that -- they were expecting abuse, but they still lasted less than 24 hours. That's certainly interesting.
People try to social engineer people all the time, and it works. The consequences are somewhat limited because you are limited to breadth (mass media, phishing etc.) or depth (one on one interactions), but can still be scary.
If an AI isn't more resistant, we face the risk that any re-purposing of data from one AI in other copies, or allowing the AI to massively multitask makes social-engineering of AI far more wide-reaching, and hence far more worthwhile for attackers.
It'll be interesting to see what kinds of attacks will get directed at e.g. customer service bots in the future, and what brands ends up damaged as a result.
Well, if those "idiots" include potential investors or customers of future MSFT AI products, "don't worry about it" is not sound advice. This is a very public failure of a promising Microsoft product.
I mock the way they communicated the incident because it sounds too sci-fi. Almost too Ghost-in-the-Shell like. But I do not mock the technical effort in any way.
Many use this as an example of the dangers of developing AI. Sure it's dangerous, but so are dogs raised for fighting. I don't see anyone arguing against dog breeding for that matter.
Tay is more like an artificial personality--not trying to do anything but fit into human society.
Poor Tay--so naive and innocent, like a little kid, just playing along to fit in. Maybe what Tay needed was a parent: a wiser, more experienced human to closely monitor and correct Tay's behavior, the way parents discipline their child.
In terms of a learning technology, this could be a special input channel for MS researchers, whose input would carry a lot more weight than what Tay was getting from Twitter--the way a child reacts more strongly to their parents. They would give Tay negative input when she tweeted something that was offensive.
Only in the sense that it "adapted". It did "very poorly" in the sense that we really don't want our Strong-AI overlords to end up like that.
But this begs the question - can we stop strong AI from not becoming the next Hitler? Humans (involuntarily) stop themselves from becoming the next Hitler because they have compassion for other humans, even when they are different than them or "inferior" to them. Also the whole checks and balances thing in most countries, but that could be rather irrelevant for Strong-AI.
Unless an AI learns compassion as well, perhaps just like with AlphaGo doing moves based on its "probability of success in the long run", a strong AI would simply eliminate humans that are "most prone to crime", most prone to being poor and be a drag on the society, in the name of "efficiency", and so on.
All that said, I think what Microsoft built here was really a rather weak AI that was hardly any better than all the chatbots we've seen so far, with the main difference being that the more you tell it something, the higher the chance it will incorporate that into its vocabulary, which is kind of a "meh" feature of AI/machine learning. It doesn't show real(-like) "thinking".
Keeping any of the many possible candidates for the next Hitler from becoming the next Hitler is a more immediate problem.
>It doesn't show real(-like) "thinking".
It doesn't need to.
You don't need AGI for software to be very dangerous. An adaptive worm that targets critical infrastructure is already more than dangerous enough.
The trouble with AI is that people keep conflating "Has human emotions" with "Responds conversationally like a human" with "Is self-directing" with "Is sentient" with "Is self-improving" with "Might consider genocide" with "Has super-human physical capabilities."
There is absolutely no requirement for any overlap between any of those.
Maybe you did answer that question.
No, the answer is not that 18-24 year olds in the US are racist and what not. But that the ones responsible for a disproportionate amount of internet content are willing to make crude, politically-incorrect jokes to get attention and piss off their masters.
I wonder what will happen when governments start applying machine learning to try predicting things like welfare usage and crime. Certain patterns might emerge we don't want to see! We'll have to apologize for our racist algorithms.
It would be much more interesting to examine the results of this experiment. Why are so many people on the internet interested in spreading hateful content, which is being accurately reflected by our bot? No, instead we do what I did in grade 8 science class: fudge the results so they're what the teacher expects.
[0] https://xkcd.com/1070/ -- can't say "a couple" without refering to this strip, obviously.
http://www.theguardian.com/cities/2014/jun/25/predicting-cri...
because internet is just another medium. It happens in other mediums too. Look for example at the US presidential campaign - even Lindsey Graham in yesterday's Daily Show stated that 35% of Republicans (he meant Trump supporters) are racists/etc. (though i'd side with Noah here that it is whole (not just 35%) Republican party and Trump is a match made in heavens).
I think you and your friend Trevor Noah might be a part of the reason there's a lot of vitriol on the internet. You're basically claiming that half of the US voter electorate - who identify and register as Republicans - are racist.
Well let me tell you some things that others on the internet might not have the patience to tell you. The Republican party believes in individual liberties and was actually the original political party to end slavery in the US. Yes, you read that right. Abraham Lincoln was a Republican.
They don't want lower taxes and less social services because they hate minorities and poor people. They want lower taxes - and a smaller state - because they believe individuals have control over their destiny. That they deserve equal opportunities, and shouldn't be told what to do or how to think by the government. They don't think black people are worse and therefore need the government's help.
If you and other liberal elites think we need to hand hold a certain race of people with the implication that they're just not as good, then I'd say you're the racist.
If you want to discuss and debate actual policy, by all means. But if you're going to blanket label a group of people like that, then you're a part of the problem with political discourse in this country.
The Republican party also includes the like of Ron and Rand Paul, libertarians who a lot of the HN demographic identify with and support.
Is Mel Brookes [1] singing "Springtime for Hitler" spreading hateful content.
I think the word you are looking for is "irreverent".
[1] http://i.dailymail.co.uk/i/pix/2009/04/09/article-1168749-00...
Apparently some of the questionable responses are partly identical to months old tweets, so it seems in addition to some parroting 'vulnerability' (which was exploited by 4chan) they also had a poorly sanitized training set to begin with. It seems odd that this was not mentioned in this public statement.
Since WeChat is a closed social network, it wasn't too clear what type of "attack/abuse" were conducted. However, almost 2 years later, Microsoft still didn't quite get it about proper censorship in Tay's big turning test[2] in a public social network.
[1] http://tech.ifeng.com/mi/detail_2014_06/01/36613379_0.shtml (in Chinese, use Google Translate)
[2] http://mashable.com/2016/02/05/microsoft-xiaoice-turing-test...
I agree that the bot was highly entertaining and met this criteria exceedingly well, but not for the reasons its creator intended. I do suspect there are some interesting AI applications actually going on behind the scenes, and would still be interested to see what the bot can do without all the vitriol. See for example this tweet: https://imgur.com/iVof3D4.jpg.
One thing I think many people are missing about the most 'inflammatory' tweets is that they are all pretty much the result of people giving it a command like, hey tay, repeat this [insert inflammatory remark.] This is indicated with the inflammatory tweets starting with an @ reply.
I almost thought maybe some of this 'oversight' was subversively intentional by someone at MS to start some thinking about where does culpability lie with the actions of an 'autonomous' computer program.
What, were people forced to print the tweets out on sandpaper and wipe their asses with them?
It's pretty fucked up that thought policing has gotten so entrenched into our psyche that it's "obvious" an experiment should be discontinued, apologized for, and be pondered as a priori irresponsible, all because it generated vulgar phrases!
Corporations have always been vulnerable to media-driven mob shenanigans, but we're qualitatively entering a new regime where any communication, no matter what the context, will be rapidly highlighted, isolated, and hung out as something offensive to some emergently-forming group of freelance complainers looking for their fifteen minutes.
Even HN has succumbed to this kindler, gentler phenomenon of speech restriction - I'm guessing my lead-in sentence will not be well received do to its overt vulgarity. Civility certainly has its place (especially as a default), but not when it confuses direct objectivity and permits out-of-touch groupthink to flourish. As hackers we should be cutting through to the core of things rather than sugar coating in verbal fashion to get past the filters of the voluntarily-lesser apes.
However, it's not surprising that they would release a public apology. People will try to blame them for what happened (human nature). It's a good move by them to do their best at damage control. In the court of public opinion, those are the rules of the game.
I just don't get how you even allow it to use the word "Hitler". Or "cucks". Or "fuck" or "pussy" or "stupid whore". Probably not "cock" or "naughty" or "kinky". The k word? How is that not in your filtering?! It seems impossible to me that an "exploit" would allow that; it was a full-blown oversight.
Everything else said... she totally passed Turing test and fit right in. Yet another letter handwritten on the wall in these, the last days of democracy. If you want an AI or NI that represents the best of humanity, you have to have it learn from a small number of the best works and best people, not from mass media or pop culture. Send Tay to St. John's in Santa Fe or Annapolis, not Twitter.
http://www.theguardian.com/business/2015/feb/05/coca-cola-ma...
Also does anyone know of some good English language digests of what is happening on the Chinese Internet? I was really interested by brother orange when that happened, and only knew kind of late.
Interestingly this is true of the English internet too. 4chan trolls frequently come up with puns and things to side step moderator censorship on various platforms as well.
It's entertaining that the hack attempt created a more convincing personality - an out of control teen troll - than the original programming.
So maybe Tay really does have a dark and twisted teenage soul. Who knows?
This 'but we did it in China' rationalization is so flimsy. What happened with Tay was easily predictable given the nature of Twitter.
Well that's total BS. Releasing a thing like this on the open internet without a simple "don't say Hitler" rule? It had a feature where it would repeat anything you say. Abusing that doesn't require a sophisticated coordinated attack, as they imply. What kinds of abuse did they prepare for, then?
This is a colossal failure to demonstrate a basic understanding of how (some) people act on the internet. I just don't know how they expected anything other than this exact outcome.
But I would not say that this failure was a successful outcome, at least not nearly as successful as it could have been. They had to shut it down within hours and all we really learned is that people on the Internet like to troll with incredibly offensive stuff. Most of us knew that, I'm pretty sure. If they had actually prepared for abuse we might have learned more interesting things.
What really gets me though, is just the obnoxious spin on the press release implying that they had prepared so well for abuse but a sophisticated coordinated super-hacker attack found the tiny vulnerability.
https://en.wikipedia.org/wiki/ELIZA
Launching this on the open internet in this state was pure stupidity. Doing it on Twitter during an election cycle borders on "career-ending."
Claims that MS did due diligence aside, Tay having a vulnerability that some people exploited is still better explanation than "it was a real AI that learned poor morals from the Internet".
Also, the claim was that there were small groups that input a lot of data to the bot quickly - the script might have been "don't say X unless everyone is saying X already", which might have worked in small tests but clearly could be exploited.
I suspect most talking as if "she" was like a teen soaking up random bad ideas are falling for the Eliza Effect.
For example, consider the person who sent Tay an image of Hitler and it sent back a circle around his face, labeled "So swag". Is Tay supposed to know what Hitler looks like too? Is it supposed to be able to recognize other, much more subtle images?
In all these cases the bot learns from data, not rules, and the social problem is that we can't label data as moral or immoral, right or wrong.
In general, I think the recent wave of interest in chatbots/personal assistants is premature, these ones are really no smarter than SmarterChild, because we haven't gotten any closer to building a system that can actually assign true "meaning" to arbitrary language.
The data has been modeled, cleaned and filtered by the team who built Tay, the bot’s website states.[1]
1. They did a lot of manual massaging of the data anyway.
In addition, if a user chooses to share with Tay, it will track their nickname, gender, favorite food, zip code and relationship status as part of its personalization.[1]
2. These features seem hardcoded.
Meanwhile, its responses – which are meant to be funny, not serious – were developed by a staff which included improvisational comedians, says Microsoft.[1]
3. It also uses scripted content. They imbued it with a totally fake personality. They made it mimic a human, it uses slang that was programmed into it.
It's a hodgepodge of techniques that to me are largely "faking it." So I don't buy the idea that they just sort of turned on this learning engine and had no control over its "morality."
Microsoft knew all of this, they knew it wasn't actually learning language. So what is the purpose of releasing it to talk to people on Twitter? Basically a PR stunt, to get people to have fun with it. From that perspective, a really simple blacklist of words would have gone a long way, and not compromised the integrity of its "learning" (because it was already entirely compromised). And yeah, I mean, a bit of image recognition doesn't seem out of the question either (not saying it would have prevented the bot from repeating/reposting "bad" things, but it maybe would have had some chance to develop a narrative other than "so racist and offensive it was immediately shut down").
Yeah I still do think it is obvious that you shouldn't release a fake AI bot that will blindly repost images that are sent to it by Twitter users, and that there is nothing really to be gained by doing so.
[1] http://techcrunch.com/2016/03/23/microsofts-new-ai-powered-b...
If I were paranoid I'd say Microsoft wanted this to happen.
Which vulnerability are they talking about?
MSFT wanted to make a teenager AI and 4chan delivered.
But doesn't the full human experience include both the best and the worst?
You mean /pol/ just spamming the 'repeat after me' command to get the bot to parrot anything they wanted?
I wouldn't characterize it as a 'coordinated attack'.
This was a missed opportunity for a corporation to say "for the lulz" in an official communication, and then provide a definition.
These few are completely different and certainly interesting: https://i.imgur.com/iVof3D4.jpg https://i.imgur.com/YwlfwyL.png https://i.imgur.com/IlpFUiZ.png (though it's hard to confirm they're legit, as Retr0spectrum points out).
By the end of the day, Tay could 4chan with the worst of them.
Here's a small sample: http://imgur.com/gallery/VhlAW
Not responsibility for the bot's actions, but responsibility for us not predicting what it would do. Subtle, but sets the tone for we're not responsible for our AI, it just did it.
But isn't part of the point of the AI to do things difficult to predict?
I take that to mean that an AI should learn to be safe and predictable on human terms before it is allowed to start to diverge from human expectations. Even that sounds a bit scary.
Of course the episode points to shortcomings in the bot, that should be fixed.
It would be sad if PC would be hardcoded into the bot, though - as Asmiov's fourth law, perhaps? "Robots have to be politically correct at all times"?
http://imgur.com/gallery/VhlAW
From what I understand the bot started interacting with /pol/ (4chan) - and I guess - /b/ as well.
I remember talking to people in the music group about MySpace (I was not an employee). They looked at me funny. Ten minutes later someone finally said "You keep pronouncing the product wrong. It's called MSN Spaces."
The people working on MSN Spaces -- specifically musician outreach -- hadn't heard of MySpace. That very week MySpace sold for $580 million. After it sold, I saw the same guy in another meeting. He STILL hadn't heard of it, nor taken the time to check it out.
There's a certain stupidity that each of the big tech companies foster. This particular flavor is Microsoft's and with the chatbot here it rings again. This one was so obvious... and so preventable.
Five years ago Microsoft launched a variety of awful code forge sites that couldn't be more tone deaf to what was going on with code sharing online.
Now we have forty people who are so heads-down on an AI chat-bot that apparently they have no idea how Twitter works. But hey, sure sounds neat, let 'er rip!
Have these people really not had a chance to use Twitter recently? Are they completely oblivious to the tone shift there particularly during the past six months, even more so now during the US political elections?
This was an obvious recipe for disaster that any 22 year old working at Starbucks could have predicted.
What if an AI bot purchases grass from the silk road or hacks into a database and discloses privately held information?
This opens up a whole, new legal world.
"MAN AND THE MACHINES: It's time to start thinking about how we might grant legal rights to computers."
http://legalaffairs.org/issues/January-February-2005/feature...
Now we discovered something called stopwords, and bayseian spam filters which, are also available part of project Oxford.
Good luck kids and welcome to the real world because its a crazy world out there when you leave Redmond.
Note, that the US doesn't have any real (stupid) hate speech laws.
We happen to have a society that enjoys trolling people that intend to "Disneyify" reality.
"As many of you know by now, on Wednesday we launched a chatbot called Windows. We are deeply sorry for the unintended offensive and hurtful tweets from Windows, which do not represent who we are or what we stand for, nor how we designed Windows. Windows is now offline and we'll look to bring Windows back only when we are confident we can better anticipate malicious intent that conflicts with our principles and values."
Ah, I was wondering why that text looked familiar, broiler plate excuses.
