undefined | Better HN

0 pointsfeqgmmr24y ago0 comments

Spark has always been able to handle way larger scale than any DW.

0 comments

hello_moto4y ago

Handle what though?

Can Spark queries 100Bn structured data performing aggregation on multiple fields (or dimension?)

geoduck144y ago

In my previous company, we had 63 petabytes of data in Snowflake.

hello_moto4y ago

That sounds great: storage problem is solved.

What about large scale read via OLAP queries (y'know, the typical measures and dimensions)

blueglassfish4y ago

That's a respectable amount for a DW, true. Spark and it's ilk are designed for much larger scales though. Multiple FAANG use cases for Spark are in the petabytes per week range.

j / k navigate · click thread line to collapse

0 comments

hello_moto4y ago

Handle what though?

Can Spark queries 100Bn structured data performing aggregation on multiple fields (or dimension?)

geoduck144y ago

In my previous company, we had 63 petabytes of data in Snowflake.

hello_moto4y ago

That sounds great: storage problem is solved.

What about large scale read via OLAP queries (y'know, the typical measures and dimensions)

blueglassfish4y ago

That's a respectable amount for a DW, true. Spark and it's ilk are designed for much larger scales though. Multiple FAANG use cases for Spark are in the petabytes per week range.

j / k navigate · click thread line to collapse