Pandas on Ray – Early Lessons from Parallelizing Pandas (opens in new tab)

I don't know what Yarn is, but Ray is a distributed heterogeneous computing framework. It's meant to make it easy to take fairly arbitrary computational programs (with machine learning being an important test/use case) and run/debug them in parallel across lots of machines with high performance in a natural fashion and without drastic changes. [1] The advantage compared to (say) Hadoop is that it allows for heterogeneous programming models and isn't limited to or designed for (say) MapReduce; you can invoke functions in parallel in a dynamic fashion pretty easily.

[1] You can get an idea of that here: https://ray.readthedocs.io/en/latest/

It feels to me like more of a spark competitor, aiming for the people who want more flexibility than Spark and not needing as much cluster setup (a redis cluster is the hardest part). It might even be able to use Yarn to do some of the resource management.

Ray is, like YARN, a resource manager - but it's lightweight and fast. Kind of like Erlang is for a lightweight process model, Ray can start tens of thousands of containers with much lower latency than YARN or Mesos or Kubernetes. It has found a niche in distributed Reinforcement Learning (deep RL), and is establishing a beach-head there. Now it faces the chasm....

For those of you confused like me, "Pandas on Ray has moved into the Modin project"

http://ray.readthedocs.io/en/latest/pandas_on_ray.html

It is a WIP distributed Pandas implementation. Allows you to spread massive dataframes, their data and computations, over multiple machines vs. Pandas which is only local to your machine. It's not quite there yet [1]

[1] http://modin.readthedocs.io/en/latest/pandas_on_ray.html#usi...