Show HN: Bender – Let's Standardize Serverless ETL (opens in new tab)

(engblog.nextdoor.com)

52 pointsstlava8y ago21 comments

21 comments

ETL stands for Extract, Transform and Load. So this is a data pipeline framework. Nextdoor please put that in your blog post as I do not believe it is a very common acronym.

techwizrd8y ago

It's not that uncommon, especially in data science/analytics/engineering. I've definitely heard "ETL" more often that data pipeline or analytics pipeline.

This is a nice list[0] of ETL software.

0: https://github.com/pawl/awesome-etl

sbuttgereit8y ago

Not uncommon in the ERP/Enterprise/Business computing either. (I say that while I'm working on just such a project for a food distributor).

gipp8y ago

It is an extremely common acronym.

vidarh8y ago

It's an extremely common acronym in a small niche. So while you're right, sort of, it's still worth clarifying when posted to a more general site.

2 more replies

taude8y ago

I'll leave you with a link to Wikipedia [1]. ETL goes back a long, long way, and is still commonly used all through the industry, except maybe in startups composed of all young, inexperienced people.

[1] https://en.wikipedia.org/wiki/Extract,_transform,_load

throwaway2016a8y ago

Thank you for pointing that out.

I would have myself made the mistakes of assuming the terms is ubiquitous and everyone already knows it. I have heard it used liberally at every job I've done in the last 15 years.

I would have termed it a "very very common acronym." Right up there with API.

But your post reminds me that not everyone's experiences are the same.

manigandham8y ago

ETL as a term has been around for decades and is very well used and understood in pretty much all IT, data, software and enterprise scenarios.

john_teller028y ago

Its nice but AWS released GLUE for similar use case a couple of weeks back

snowpalmer8y ago

I have so far been unimpressed with GLUE.

slagfart8y ago

Gobblin (https://gobblin.apache.org/), which looks like it does something similar, can be packaged up as a single .jar, and run on AWS Lambda.

Anyone had working experience with either?

throwaway2016a8y ago

Great work. I'm in the process of writing an ETL and while I don't think this will suite our needs (so unfortunately I need to keep writing) this article provides a lot of great detail that helped me see the process more clearly.

diranged8y ago

It would be great to know what doesn't fit your needs - can you describe your project in more detail?

rbradk8y ago

Interested in chatting about your ETL project more? We may be able to help.

throway_foo_bar8y ago

Link to the source in the document doesn't work.

diranged8y ago

Can you point me to where -- I scanned through the Blog and havn't found a bad link yet?

throway_foo_bar8y ago

The last link, to github, was to some internal repo. It has since been fixed.

j / k navigate · click thread line to collapse

21 comments

RubenSandwich8y ago

ETL stands for Extract, Transform and Load. So this is a data pipeline framework. Nextdoor please put that in your blog post as I do not believe it is a very common acronym.

techwizrd8y ago

It's not that uncommon, especially in data science/analytics/engineering. I've definitely heard "ETL" more often that data pipeline or analytics pipeline.

This is a nice list[0] of ETL software.

0: https://github.com/pawl/awesome-etl

sbuttgereit8y ago

Not uncommon in the ERP/Enterprise/Business computing either. (I say that while I'm working on just such a project for a food distributor).

gipp8y ago

It is an extremely common acronym.

vidarh8y ago

It's an extremely common acronym in a small niche. So while you're right, sort of, it's still worth clarifying when posted to a more general site.

2 more replies

taude8y ago

I'll leave you with a link to Wikipedia [1]. ETL goes back a long, long way, and is still commonly used all through the industry, except maybe in startups composed of all young, inexperienced people.

[1] https://en.wikipedia.org/wiki/Extract,_transform,_load

throwaway2016a8y ago

Thank you for pointing that out.

I would have myself made the mistakes of assuming the terms is ubiquitous and everyone already knows it. I have heard it used liberally at every job I've done in the last 15 years.

I would have termed it a "very very common acronym." Right up there with API.

But your post reminds me that not everyone's experiences are the same.

manigandham8y ago

ETL as a term has been around for decades and is very well used and understood in pretty much all IT, data, software and enterprise scenarios.

john_teller028y ago

Its nice but AWS released GLUE for similar use case a couple of weeks back

snowpalmer8y ago

I have so far been unimpressed with GLUE.

slagfart8y ago

Gobblin (https://gobblin.apache.org/), which looks like it does something similar, can be packaged up as a single .jar, and run on AWS Lambda.

Anyone had working experience with either?

throwaway2016a8y ago

diranged8y ago

It would be great to know what doesn't fit your needs - can you describe your project in more detail?

rbradk8y ago

Interested in chatting about your ETL project more? We may be able to help.

throway_foo_bar8y ago

Link to the source in the document doesn't work.

diranged8y ago

Can you point me to where -- I scanned through the Blog and havn't found a bad link yet?

throway_foo_bar8y ago

The last link, to github, was to some internal repo. It has since been fixed.

j / k navigate · click thread line to collapse