undefined | Better HN

0 pointsdonall1y ago0 comments

I had a visceral reaction to this comment! I once joined a company doing ETL with Apache camel and a half dozen underpowered pet machines. Ingesting their entire dataset and running a suite of NLP models took 3-6 months (estimated; it was so slow nobody ever reprocessed the data to fix bugs or release improvements). I drew up a simple architecture using Kafka, hbase, and MapReduce to implement a lambda architecture. The CTO very patronizingly told me that just because something is shiny and new it doesn't mean we need to implement it. This was 2017 :laugh-cry:.

0 comments

yyyfb1y ago

But maybe this isn't really what they felt that they needed at the time? I don't mean to defend bad practices, but your comment makes it sound like nobody had tasked you with re-architecting the business, and you took it upon yourself to show them how it should be done (in your opinion), without having earned the necessary trust. This might have also come across as patronizing, or at least antagonistic, and in any case unbeneficial. Not saying that's the case as I obviously wasn't there, just something to think about.

donallOP1y ago

Fair comment. And I'm usually suspicious of young engineers wanting to implement the new hotness and I'm also a fan of "if it ain't broken don't fix it". In this case, though, the system was in very rough shape. Our customers were complaining about data problems which we had no way to fix (short of manually editing the prod db, which was the SOP). I definitely took it upon myself to do something that nobody had asked for, but it was because the people in charge were entirely asleep at the wheel! They did not last long in their positions.

bjconlan1y ago

It's a shame the CTO was patronizing. I've generally found this to be the attitude of many IT workers in similar positions. I would recommend trying to allocate (work) time to prototype and get important back of the envelope metrics that they think are valuable along with those that you think are valuable.

At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)

j / k navigate · click thread line to collapse