undefined | Better HN

0 pointsdreyfan4y ago0 comments

Databricks is a shit platform that encourages terrible data practices and accretion of technical debt.

0 comments

As with other offerings in this space, the key to managing technical debt is to get functions out of notebooks ASAP, stage intermediate results where appropriate, and turn everything into jobs.

exsmelliarmus4y ago

Seems pretty good to us! Can you give more information?

0x500x794y ago

As people noted elsewhere, you have to be VERY careful with using databricks for a full data warehouse due to the fact that it drives you to notebook driven development and scheduling of those notebooks when data pipelines should follow similar development practices as other software projects.

Great for proof of concepts, but when you start to build out complete pipelines please look into how to make the pipelines more sustainable and maintainable.

fs1114y ago

Finally somebody that has used Databricks! I can't believe all the praise I read elsewhere in the comments here. Databricks is broken in so many ways, it is beyond me how anyone can like using this.

1 more reply

j / k navigate · click thread line to collapse