undefined | Better HN

0 pointsdrawturkey4y ago0 comments

This is very true. They make the lowest common denominator parts "open source" but control all of the commits. Also the query engine used for this benchmark is proprietary, closed source (Photon)

0 comments

feqgmmr24y ago

The 'open' here refers to the data. Delta lake can be read/written by multiple open source engines, not just Spark. Not to mention, if you want you can use Databricks with Parquet, though the experience won't be as good.

But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

feqgmmr24y ago

Do you have to pay to export data out of Snowflake? Yes. They have a nice guide on how to spend money doing it (https://docs.snowflake.com/en/user-guide/data-unload-overvie...).

Do you have to pay to export data out of Databricks? No, it's already sitting where you want it.

Which one is open? I wonder

geoduck144y ago

I used Snowflake in my previous company. When we loaded data into Snowflake, we loaded it FROM S3/Blob where we also kept it.

1 more reply

drawturkeyOP4y ago

Not entirely true. There is a bi-directional Spark connector for Snowflake written by Databricks. And exporting your data in bulk out of Snowflake into any number of open formats is incredibly easy using the COPY INTO command. You can also use Snowflake on top of Parquet and even Delta Lake.

This is the problem. Both Snowflake and Databricks are spreading FUD and otherwise smart people are falling for it.

feqgmmr24y ago

It is not a "small" cost. The cost is proportional to the size of the data exported.

For all intents and purposes, large amounts of data are locked into Snowflake. Is it theoretically possible to export a petabyte out of SF? Sure.

Do I want to spend money on it? Not really. That is what I mean by the "data doesn't come out".

"Exporting" a petabyte out of Databricks is a no-op. I can already read Deltalake from other open source tools.

2 more replies

drawturkeyOP4y ago

Agreed there is a small cost, but it is possible, which is at odds with your statement "with Snowflake, the data never comes out".

j / k navigate · click thread line to collapse

0 comments

feqgmmr24y ago

But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

feqgmmr24y ago

Do you have to pay to export data out of Snowflake? Yes. They have a nice guide on how to spend money doing it (https://docs.snowflake.com/en/user-guide/data-unload-overvie...).

Do you have to pay to export data out of Databricks? No, it's already sitting where you want it.

Which one is open? I wonder

geoduck144y ago

I used Snowflake in my previous company. When we loaded data into Snowflake, we loaded it FROM S3/Blob where we also kept it.

1 more reply

drawturkeyOP4y ago

This is the problem. Both Snowflake and Databricks are spreading FUD and otherwise smart people are falling for it.

feqgmmr24y ago

It is not a "small" cost. The cost is proportional to the size of the data exported.

For all intents and purposes, large amounts of data are locked into Snowflake. Is it theoretically possible to export a petabyte out of SF? Sure.

Do I want to spend money on it? Not really. That is what I mean by the "data doesn't come out".

"Exporting" a petabyte out of Databricks is a no-op. I can already read Deltalake from other open source tools.

2 more replies

drawturkeyOP4y ago

Agreed there is a small cost, but it is possible, which is at odds with your statement "with Snowflake, the data never comes out".

j / k navigate · click thread line to collapse