https://azure.microsoft.com/en-us/services/machine-learning/
^Basic version is free to use!
Had they had this direction ten years ago, probably android would not have existed!
(full disclosure: employee here)
Their "News QA dataset" contains 120k Q&As collected from CNN articles:
Documents are CNN news articles. Questions are written by human users in natural language. Answers may be multiword passages of the source text. Questions may be unanswerable.
NewsQA is collected using a 3-stage, siloed process. Questioners see only an article's headline and highlights. Answerers see the question and the full article, then select an answer passage. Validators see the article, the question, and a set of answers that they rank. NewsQA is more natural and more challenging than previous datasets.
Their "Frames" dataset contains 1369 dialogues for vacation scheduling:
With this dataset, we also present a new task: frame tracking. Our main observation is that decision-making is tightly linked to memory. In effect, to choose a trip, users and wizards talked about different possibilities, compared them and went back-and-forth between cities, dates, or vacation packages.
Current systems are memory-less. They implement slot-filling for search as a sequential process where the user is asked for constraints one after the other until a database query can be formulated. Only one set of constraints is kept in memory. For instance, in the illustration below, on the left, when the user mentions Montreal, it overwrites Toronto as destination city. However, behaviours observed in Frames imply that slot values should not be overwritten. One use-case is comparisons: it is common that users ask to compare different items and in this case, different sets of constraints are involved (for instance, different destinations). Frame tracking consists of keeping in memory all the different sets of constraints mentioned by the user. It is a generalization of the state tracking task to a setting where not only the current frame is memorized.
Adding this kind of conversational memory is key to building agents which do not simply serve as a natural language interface for searching a database but instead accompany users in their exploration and help them find the best item.
---
Can anyone with experience in ML/AI comment on how novel/complex these projects are, and how expensive it would be to build out these datasets? Would be interesting to see what it takes to publish a few datasets trained on 20 day conversations between real people, and get acquired by Microsoft/Apple/Google.
Assuming you have the expertise necessary to design and run the process in-house, the major expense is going to be compensating the humans in the loop, which can add up quickly.
This is why organizations that already have access to large datasets have such a huge advantage.
I think that one of the reasons we're seeing such a rush to deploy chatbots is that even a minimally-useful bot will quickly start accumulating extremely useful (and very clean) training data.
There is a lot of noise being made about "democratizing AI", but as long as the best results require a lot of training and huge amounts of training data it will remain the bottleneck.
Look for progress on 1-shot and 0-shot learning to get a better feel for how much progress is made on real democratization.
Last fall, we formed the Artificial Intelligence and Research organization, bringing engineering and research closer together to accelerate the pipeline from cutting-edge research to product development. Maluuba, too, has closely aligned its research and engineering teams, and we’re looking forward to learning from their experiences as well.
"Frame tracking" and adaptive slot filling without pre-canning is actually a very active area of research - the goal is to provide not just an Eliza style infinite conversational ability, but to be able to reason about and get the user to a specific outcome in a real world use case by prompting them for information in a semi supervised manner.
Being able to do it via a series of differentiable (calculus wise) functions is a fundamental improvement in achieving convergence between statistical systems and logical reasoning (which harks back to "classical" AI problems such as planning). Microsoft has some very interesting research here and papers like these [1][2]. Maluuba has some good stuff here as well. [3][4]
I doubt the company was acquired for their datasets. They've been a well respected AI company with some top notch researchers, and I think their area of research gels well with MSR's own NLP research.
[1] https://arxiv.org/abs/1606.01269
[2] https://www.microsoft.com/en-us/research/publication/unsuper...
But if one of the next 12 users was a Microsoft executive who liked what they saw, well then that's an important data point :) Having Bengio as an advisor definitely helps too.
Applying reinforcement learning to dialogue systems seems incredibly difficult, but if Maluuba (or others) can get a handle on the problem it would not be unreasonable to expect another revolution in the vein of applying convolutional nets to vision.
