Haven't seen any memory consumption benchmarks but suspect that it's lower than Spark for same jobs since datafusion is designsd from the ground up to be columnar-first.
For companies spending 100s of thousands if not millions on compute this would mean substantial savings with little effort.
[1] https://datafusion.apache.org/comet/contributor-guide/benchm...
Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.
I say theoretically, because I have no idea how Comet works with the memory limits on Spark executors. If you have to rebalance the memory between regular memory and memory overhead or provision some off-heap memory for Comet, then the migration won't be so simple.
I want to be able to connect to interact with the full services from GCS, Azure, AWS, OpenAI etc none of which DataFusion supports.
As well as use libraries such as SynapseML, SparkNLP etc.
And do all of this with full support from my cloud provider.
How does it compare to Blaze[1] and Gluten[2]?
I'm interested in running some benchmarks soon against all three for my project to see how they all go.
I live in a dream world :)
Databricks' terms prevent(ed?) publishing benchmarks, it would be interesting to see how Comet performs relative to it over time.
Photon comes at a higher cost, so one big advantage of Comet is being able to deploy it on a standard Databricks cluster that doesn't have Photon, at a lower running cost.
