Amazon Machine Learning – Make Data-Driven Decisions at Scale (opens in new tab)

The hardest part in machine learning is not training the model but debugging the model. How do you improve precision/recall after the first cut? Do you need more training data? Is some of your training data bad? Is it properly distributed? Does your feature have bug? Are you missing features to cover some cases? Is your feature selection effective? Did you tuned parameters carefully?

All these scenarios are difficult to debug because it's "statistical debugging". There are no breakpoints to put or watch windows to look at. There is no stack trace and there are no exceptions. Any Joe can train a model given training data, it takes fair bit of genius to debug these issues and push model performance to next level. Unfortunately all these new and old "frameworks" almost completely ignore this debugging part. I think the first framework that has great debugging tools will revolutionize ML like Borland revolutionized programming with its visual IDEs.

This. The pity is that immediately we get the results after a week the project is over and we move back to data wrangling hell!

You hit the nail on the head. Completely agrees with all my experience at Kaggle and applying machine learning across a broad number of industries

Maybe it's just me, but the "tedious" feature design and extraction IS the fun part. Am I the only one?

I mean, it's time consuming and frustrating, but it's also the essence of ML work and the place where I get to apply creativity and gain insight.

Agree 100%, in that light, anyone know how far we are away from having data wrangling be more automated? I saw a demo for a product called Paxata a few weeks ago, it looked like a good start. Anyone know more about things like that?

I think the "1 part fun 9 parts of perspiration" ratio is typical of most software fields - especially fields working in established industries. That's why dealing with software in a professional context is called a job and not an enjoyable hobby which it otherwise would be :)

I think this is one of advertised advantages of deep learning: it will find useful and unobvious features in your data corpus without much effort from your side.

Isn't the point here that you can do it on huge datasets that don't work nicely with R

do you mean that it takes more work to do the stuff surrouding the machine learning like gathering data to build a dataset that takes months and other resources where as the fun stuff is actually very short and easy to do.

I smell commoditization.

No... it's Amazon ML and Azure ML trying to catch up with BigML. They copied many things from our service but forgot to copy the ease of use. Services like Azure ML, Amazon ML and even Google Predict API work like a black box, and lock your model away, making you extremely dependent on their proprietary service. With BigML, you can easily export your models and use them anywhere for free. If the goal is to democratize machine learning, then the ability to extract your models and use them as you see fit is essential, and only BigML offers that level of freedom.

I would be less worried about that and more worried about cost. I know of two different startups that aren't profitable but would be if they hadn't put their entire platform on amazon services. One of those startups was lucky enough to be acquired but it's going to take them many unprofitable years to migrate away.

BigML cofounder here. Most BigML customers doing machine learning at scale use either BigML subscriptions (starting $30/mo) or private deployments – both of which provide unlimited model training and predictions and are suitable for developers and large enterprises alike. In addition, with BigML you can export your models (for cluster analysis and anomaly detection and not just classification/regression) to run locally and/or to be incorporated in related systems and services.

No -- see https://aws.amazon.com/machine-learning/faqs/ --

"Q: What algorithm does Amazon Machine Learning use to generate models?

Amazon Machine Learning currently uses an industry-standard logistic regression algorithm to generate models."

But disappointingly:

"Q: Can I export my models out of Amazon Machine Learning?

No.

Q: Can I import existing models into Amazon Machine Learning?

No."

Note that they are doing classification and regression on iid feature vectors. Of course, ML is much larger than this setting, but this setting is generic enough that it has some applicability to lots of problems.

Or Weka.

> Maybe a few companies/startups who have cash in their pocket will use it/try it out

Honestly, I'd see it the other way around. Small companies without a DS team might be drawn to this. I don't see how any company with a lick of sense would lock down their prediction model into AWS. They very clearly won't let you export your model once the training is done.

> Maybe a few companies/startups who have cash in their pocket will use it/try it out

This would be really nice to use at my startup, but its cost prohibitive even on a very large budget.

I am setting up Spark Streaming to handle model creation and updates for recommendations based on what a user interacts with. If I were to even attempt something similar with this AWS service, its $10 for every 1 million predictions which isn't sustainable (not including the costs to create and update the model).

> but the audience is really limited beyond that in my opinion.

Definitely, largely as a result of cost. I would love to not have to worry about Spark in my infrastructure (its another piece...) but at this price the AWS service is just too expensive.

The system also uses logistic regression and is limited to 100 gb dataset. Prediction with LR isn't that expensive and training can be done online with something like stochastic gradient descent. That can be done on a single computer. Given that the models aren't exportable and you can import a model, I'm hard pressed to see the immediate value. Long term, though, I'm sure there's plenty of growth.

It's kind of unclear, but it looks from the screenshots as if AWS is doing feature selection behind the scenes. But it seems that unless AWS does feature selection or model selection really efficiently behind the scenes, the cost of that extra work time is placed on the user.

What differences did you notice beyond Google's Prediction API?

go to the datasources tab and see if there's an error message from data source creation. i had the same error due to an issue with variable names.