How we run Spark and Sqoop in production (opens in new tab)

(thumbtack.com)

58 pointsnatekupp9y ago18 comments

18 comments

Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.

technofiend9y ago

I feel like I've recommended it enough times I'm turning into a shill but Pentaho is an open-source and commercially supported ETL tool that will natively do what you want, or call sqoop when you discover that's kinda slow. :-) And no I definitely don't work there.

dn59y ago

Have a look at Kafka Connect (http://docs.confluent.io/2.0.0/connect). The JDBC Connector will poll for database changes changes and push them to a Kafka topic. Means you should see all the changes in the database rather than a snapshot say once a day

rathboma9y ago

Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi

You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline

machbio9y ago

Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?

2 more replies

natekuppOP9y ago

+1 to this, we kick off our Sqoop jobs using Airflow - http://airbnb.io/projects/airflow/

Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.

Joeri9y ago

We've been trying out goldengate to get streaming replication, but it has proven rather unreliable. Stops replicating if you sneeze in its general vicinity. I wonder whether the alternatives like shareplex and tungsten are more reliable.

capkutay9y ago

You can try out Striim for streaming data integration (full disclosure, I work there):

http://www.striim.com/download-striim/

falaki9y ago

Ever since using Apache Spark's Data Sources API was released, I have been relying no different Spark Data Source packages for my ETL jobs.

sciurus9y ago

Their earlier post about rebuilding their data infrastructure is more interesting imho: https://news.ycombinator.com/item?id=11474284

natekuppOP9y ago

hey all, feel free to reach out to me either on this thread, or directly at nate[at]thumbtack.com if I can answer any questions!

rahij9y ago

I had to connect to a US VPN to access the jobs page. Is that intentional?

natekuppOP9y ago

thanks for flagging, I'll look into it!

lazywizard9y ago

Thanks for sharing with all scripts. Great help.

lobster_johnson9y ago

Speaking of Spark, has anyone used it with Go? Is such a thing even possible?

mcrad9y ago

Thumbtack is great for lazy consumers. But word on the street is they contribute to too much price pressure on the market, therefore drive the overall quality of services down. Good work Thumbtack! You have figured out a convenient way to sacrifice long term value in favor of short term profit.

j / k navigate · click thread line to collapse

18 comments

ktamura9y ago

Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.

technofiend9y ago

dn59y ago

rathboma9y ago

Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi

You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline

machbio9y ago

Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?

2 more replies

natekuppOP9y ago

+1 to this, we kick off our Sqoop jobs using Airflow - http://airbnb.io/projects/airflow/

Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.

Joeri9y ago

capkutay9y ago

You can try out Striim for streaming data integration (full disclosure, I work there):

http://www.striim.com/download-striim/

falaki9y ago

Ever since using Apache Spark's Data Sources API was released, I have been relying no different Spark Data Source packages for my ETL jobs.

sciurus9y ago

Their earlier post about rebuilding their data infrastructure is more interesting imho: https://news.ycombinator.com/item?id=11474284

natekuppOP9y ago

hey all, feel free to reach out to me either on this thread, or directly at nate[at]thumbtack.com if I can answer any questions!

rahij9y ago

I had to connect to a US VPN to access the jobs page. Is that intentional?

natekuppOP9y ago

thanks for flagging, I'll look into it!

lazywizard9y ago

Thanks for sharing with all scripts. Great help.

lobster_johnson9y ago

Speaking of Spark, has anyone used it with Go? Is such a thing even possible?

mcrad9y ago

j / k navigate · click thread line to collapse