dwrodri

I deal with a lot of building and debugging data pipelines at work, and the Data Wrangler plugin for VSCode[1] has quickly risen to the top as a quick and easy way for me for inspect and prototype transforms to parquet files and CSVs.

I am submitting this question to HN to perhaps collect more diverse perspectives than I could get from language models and search engines, given the audience here. I have tried to lean into using Visidata[2] a bit, but it has been a steep learning curve so far, and it hasn't exactly been easy to take visidata transformations and apply them to our python codebase.

1: https://devblogs.microsoft.com/python/data-wrangler-release/

2: https://www.visidata.org/

I am an ML Engineer at a Python shop supporting a team of 15-20 data analysts/scientists with a wide range of experience. Most of my gig is building tooling for them and dogfooding that tooling to make sure it works well. All of our Data people know SQL pretty well, but we'd rather not let people run wild trying to write data transformations using two paradigms (Pandas API vs. SQL queries) if we don't have to.

Am I chasing after a rainbow trying to provide a consistent DX here if I consider this to effectively be a solo project at this scale?

DuckDB seems like a promising contender here, but there is a whole new generation of tooling which has emerged to contend with all of the limitation of pandas.

Does anyone here have any positive stories of interacting with this tooling without effectively also signing up for huge maintenance efforts or a new expensive enterprise license? Open to any and all options that don't require my end users to move away from Python.