And I don't think it matters anymore. I predict that the JVM will eventually be out of the equation. We're already seeing query engines being replaced by proprietary or open source equivalents in C++ or Rust. Large scale distribution is less of a selling point with modern cloud computing. Do you really need 100 executors when you can get a bare metal instance with 192, 256 or 384 cores?
People want a dataframe API in Python because that's what the the ML/DS/AI crowd knows. Queries and processing will be done in C++ or Rust, with little or even zero need for a distributed runtime. The JVM and Scala solve a problem that simply won't exist anymore.
Still my favorite language I had the privilege to work with professionally for over a decade. However, in this post-JVM world, I'm actually excited to see a lot more OCaml discussion on here lately. The Jane Street work on OxCaml is terrific after a long period of stagnation with the language. I'm using it for most of my projects these days.
