My career record (US Navy) for cost savings was something over $50MM. Every time I did something I had to do PowerPoints, present to Flag Officers, etc. Hell, once I almost got hugely punished because I didn't let my boss take the credit (he had no desire to because he had zero idea what it was I even did).
Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects (I hate every single bit of terminology in that entire godforsaken sentence), where it was literally my JOB to save the DOD as much money as possible with the least disruption possible. Those were the absolute worst years of my career. That period of my career was my reward for just going rogue and fixing things that saved millions.
I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.
Never stay at a job where the culture is toxic. Even if the money is better. Nothing eats at the soul like being held down by feckless, petty careerists and control freaks.
Ideally, you should be working for a company that pays well and that actually rewards you for doing things that are good for the business. But it does seem like virtually every company has some kind of dysfunction.
When they are, just move on. Life's too short. Whatever blows you up with $2 million in savings would blow you up at $500K
Paraphrasing, "The great thing about the US Government is it's so big, if the optimal solution would save 30%, but you find a suboptimal solution that only saves 29%, nobody will notice because you're still saving tens or hundreds of millions of dollars of savings."
One of his big projects was on energy efficiently in remote military camps. The cost of fuel per gallon was around $100 delivered to parts of Afghanistan. They had tons of portable generators running at ~20-40% of capacity. If memory serves, ~70% is most efficient. By some combination of building a small or connecting a few tents to a single generator, they were able to very significantly reduce the fuel required and improve the quality of service.
This was one of the most deeply insane parts of the whole twenty-year war. Of course, wars have unlimited budgets, so trucking gasoline thousands of miles through hostile territory with occasional losses and then using it to aircondition tents was going to happen. While back inside the US teachers were having to buy school supplies with their own money, because only wars have unlimited budgets.
I'm an engineer in a FAANG, who worked directly in money flow, and have had experience in diverse areas where it could come handy.
Then I start thinking finding the right person to work with & right area to start at is probably 95% of the job, then give up.
The direct employment option might get better with the recent executive order on AI. USG pays market wages for VA doctor/s; eventually they’ll do the same for tech roles.
I'm so triggered by this phrase. The worst (to work for and performing) company I was at hired SSBBs like crazy and none of them really did anything or made a damn bit of difference, but if you took the courses you were fast-tracked for success as far as promotions went.
That said, it did give me the opportunity to get the hell out of that community and back into IT/CyberSec.
In all honesty, there are absolutely good concepts and tools in that world. The issue is that the people who gravitate to that world have no ability to provide any value themselves so don't actually understand how to USE those tools properly.
Everything is a nail when all you have is an ENTIRE TOOLBOX.
A few jobs ago I was asked to list the names of projects I was actively maintaining for the company, that list stretched out to two entire screens in Excel at default font size! I was burnt out to the point of severe depression.
Even worse is I hate the bullshit ritual of submission that is job hunting.
Let’s rephrase this in a less nihilistic way: Understand your organization’s values and culture. And beware of doing good (or bad) things in a way that goes against your organization’s values and culture.
There isn't a way to make things better in the DOD that doesn't go against the culture. Period. My literal JOB was to make things better and it was the worst time of my career. That was with Flag Officer backing and independent authority. I got a chest full of shiny candy and a pathological distaste for it all after a few years of that.
The takeaway is don't be good at things you don't want to be asked to do.
You do have the option of becoming self employed. If you do, then you might discover a very different perspective. I've been doing it for 23 years now, with two partners and around 20 odd employees. IT company, you wont have heard of us but you will know some of our customers.
I'm still sort of working now (its ~0030 here) keeping an ear to the ground - HN, r/sysadmin (int al), The Register and a few others are some of my canaries. I listen to their songs. I also have a BirdNetPi doing the same for actual birds too.
If my company cocks up it will probably be fatal. I/we don't get the luxury of blaming someone else and I certainly don't get to mumble some sort of bollocks about "Your x is very important to us, soz we failed, lol and that - here have some free credit rating checker thingie".
Why not try running a small company and living on your own wits with no mother to land on - you are mother!
Even then, my kind of self work is just as unstable as my industry. I may in fact be good at it and still fail.
>(I hate every single bit of terminology in that entire godforsaken sentence.
Checks out. :).
A super convenient way to do cool stuff like that in the DoD is to do it for the most senior Flag you can get interested in it directly. Projects at SECDEF or SECNAV offices work...differently than outside, as there isn't really an 'up' left for most of the folks involved (in my experience - most are extremely focused on getting the job done and/or the geo-strategic problems).
Anything that was directly interfacing with that office was great; as soon as I was detailed to do something downline from that office it got painful despite reporting to that Flag.
I wasn't in a position to go higher without spending more time in the service, and that was a non-starter for me.
A part of me finds I might know you. Another part states: this story is way too common.
... I'm mentally tripping over that part. Is that normally expected? ... By recursion, it seems that POTUS should get all the kudos, all the time.
Some of that credit may or may not spill over to you, but the general thought process is that they are responsible for what happens in the command so they get the credit. Good ones avoid that and push the credit to where it belongs, but then they don’t promote. This leads to an incentive to take credit from those under you in order to get promoted.
Preach. 100%.
Likewise with chip software tooling; despite it being standard to outsource tooling to large EDA vendors, we got a lot of mileage out using our own custom tools, generally created or maintained by one person, e.g., while I was there, most simulator cycles were run on a custom simulator that was maintained by one person, which saved millions a year in simulator costs (standard pricing for a simulator at the time was a few thousand dollars per license per year and we had a farm of about a thousand simulation machines). You might think that, if a single person can create or maintain a tool that's worth millions of dollars a year to the company, our competitors would do the same thing, just like you might think that if you can ship faster and at a lower cost by hiring a person who knows how to crack a wafer open, our competitors would do that, but they mostly didn't.
He talks about a "cocktail party version of the EMH". The credentialed econ version of "nothing works" is https://en.wikipedia.org/wiki/The_Market_for_Lemons , which discusses the role of information and its asymmetry in markets. The EMH fails precisely because total knowledge is impossible and adverse selection is real.
There are some companies that represent the opposite of what OP describes. Once you work with one of them, I've found it impossible to settle for one of the bad places.
- managers asked how it was possible we saved that much without help from them
- asked to prepare slides
- asked many times on how it happened
- had to roll it out slowly to make it look like they did it over time incrementally vs one small toggle
- asked for a raise due to impact and did not happen.
Sir, for your sake, apply to a FAANG or something, you'll be at least taken care of better.
Also please implement Twitter card metadata in your blog so it looks better on twitter :)
"hi everyone, I wanted to walk you how we got to a half-a-million dollar savings, basically I spent a day looking at how terrible the original infrastructure was deployed and removed a code test feature that was causing the problems. This was just complete oversight from every aspect of the development, management, testing, and everything. Overall this code is as bad as it can possibly get, and we just launched it. And basically I was told not to say any of this because it makes everyone look bad, so I was to roll this out gradually to make it seem like managers were doing some sort of work."
Then drop the mic and walk off stage.
Honestly the amount of give-a-fucks I would have lost would have been a lot. And this is coming from someone who's done this for almost 2 decades and cares about his job because bills to pay, kids to feed.
About once a year or so one of the stand-out engineers that had the weight of the world on their shoulders would get burnt out and frustrated enough to do exactly what you suggest above.
Literally everyone would just look around awkwardly, leave the meeting and never talk about it again. All of middle management already know all of this, the only way they keep their jobs is by never talking about it, and just ignoring anyone that does. The VPs and President only know what those below them feed them.
But you would get handsomely paid.
Didn’t Xitter recently remove the display of all external site metadata except the image?
See, for example: https://twitter.com/simonw/status/1717768637799706922
“We’ll need the percentage complete as well and a summary of the savings expected. It needs to be sent out every Friday at noon.”
Noooooooooo
Wait, it wasnt the managers that designed the overly complex solution! this is all on the engineers
Large organisations are so woefully inefficient that I'm surprised they're able to compete at all, but they have a ton of money and economy of scale and all that, and along the way there's more than enough money to waste millions on stupid nonsense and inefficiency and nobody really cares.
It's easy to understand: large orgs are competing against other large orgs that are similarly inefficient, because efficiency in a large org is a really, really, really hard human problem that we simply haven't solved.
And remember they're still providing a valuable service/product. For all their inefficiency, they're still more efficient than not existing at all. This is why they would exist if there weren't competition.
And you might ask, what about competition from small orgs? Well, a small org has less efficiencies of scale, but if they are efficient otherwise, they can sometimes compete effectively with the large org. But then they grow into a large org and will wind up with all of the large org inefficiencies.
It's just how things are. It's not that nobody cares -- to the contrary, company owners care hugely. It's that literally nobody has any idea how to solve it.
We used to break up companies that became Too Large. Either they became so large that it was impossible to compete against them (as Large Company can demand way better pricing on volume than Small Startup can or has way better access to financial instruments due to better ratings, or like many automotive companies even run their own goddamn bank), or because they could use a wildly profitable business to price-dump competitors out of existence or because they became so large that they felt free to extort their customers to the tune the people would complain about it in the media/their congresspeople too much, or because they became so large their sheer size represented too much economic risk ("too big to fail").
We should begin doing that again. Something as big as Google/Alphabet, Apple, Amazon, Meta/Facebook, Walmart, the Big Four consultancy shops, virtually all major banks - there's absolutely no reason these should be allowed to exist at their current size. Or, if these companies still wish to exist at their size / their existence as one platform, they at least have to be regulated to mitigate the threat originating from that scale.
Competition is usually eliminated by one or more "moats": huge capital upfront investment, patents, market lockin agreements (e.g. the way Windows is sold to OEMs), vertical integration (e.g. Apple), compliance (it's genuinely hard to start a bank or a hospital), buying out competition, buying out the staff of the competition (why FAANG have so many highly paid staff doing work that gets cancelled) or even less legitimate means.
Large companies tend to end up with similar failure modes to authoritarian state-controlled companies. It's interesting how well China has been able to ride the line between control and growth on this one.
They typically don't. When the government doesn't enforce it's own rules, these companies just buy up competition. They strangle the market and they get more inefficient all at the same time.
Competition is a fairy tale we tell MBAs so they'll start new businesses so it looks like there's competition. They never had a chance.
Inflation is actually very simple: More money is added to the system than value produced. That is, money is printed faster than value is created through labor. Stop and think for a second. How do we have trillions of USD? Banks create money. And they're doing it faster than ever before.
Now you can absolutely have price gouging at the same time, but the two are independent of each other, even though combined the affects are worse for the price gouged.
It's the reason why most central banks really want some minimal level of inflation going on at all times.
The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG.
I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly.
EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.
Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
If you had just let the crisis occur, everyone would get a chance to spring into action like heroes, handshakes and champagne all around on a job well done.
Or find and company that values your talent and knowledge.
Over the next 15 minutes, I kept bumping the priority of the top job in the queue up, an hour later everyone had their work done, and went home. I had the room to myself. Rogue sysadmin for the win. ;-)
I poked around and realized that there was a system that we weren't using anymore that was copying files to the bucket I reached out to the stakeholders and they turned it off and we deleted the files.
The higher-ups didn't seem to really care, My boss's boss told me to reach out to another team that should've caught this and that was about it.
That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading.
I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.
The problem was to calculate bunch of stats from web servers logs (e.g. 10 most popular pages). The original solution was loading it all into Oracle database running on multiple servers since logs were huge. And then running bunch of SQL queries. Rinse and repeat daily.
There is a high likelihood that many other people in the org were lightyears ahead of the author. It's even more likely that an engineer or manager in their group was "Banking" the inefficiencies to use during a cost-cutting period - and the author ruined that chance... which will inevitably cause massive suffering and possibly poor performance reviews when there is nothing to trim.
It really sounds like they stayed in their lane and did their assigned job, just like every other person on their team.
I have worked with someone who was like this. Perhaps unlike the author, they were not in fact the savior in a room full of idiots.
This person was no doubt clever and had interesting experiences to learn from. But they were also dysfunctionally arrogant and utterly resistant to ideas that were not their own, to the point where it repeatedly and painfully got in the way of their own progress and other people's progress around them. At the end of the day they got done what they needed to get done, but they left a big mess behind that took weeks to even stabilize (as in, "prod is broken, prioritize above all other work") before the rest of the team could get back to their regular work.
May such people forever find themselves in solo contributor roles, so as to spare others the pain of being their close collaborators.
In our collaborative environment, it's valuable to maintain a level of respect for diverse perspectives, even if they initially appear confrontational. Instead of dismissing the author's stance outright, we might benefit from conducting a thorough assessment of our current procedures and identifying areas for potential improvement.
Engaging in constructive dialogue allows us to pinpoint inefficiencies and work collectively towards solutions that benefit our organisation. Ultimately, it's our ability to learn from each other and adapt that will contribute to our long-term success.
I'm guessing it's dormant because the level of discourse on HN has devolved to a point where nearly every comment on HN would now qualify.
What really scared me is that I couldn't identify any of the issues raised in my own organization, even though we often run into similar, smaller-magnitude problems caused by a blindness to obvious mistakes. It makes me fear I, too, am blind to massive bleeding wounds. Here's hoping they actually don't exist.
The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was.
The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.
Yep.
If you don't make them realize it was a Hard Problem™ that was only solved by their smart hiring, funding, and task-deciding, you might shatter their whole world view.
Yes, management would have killed me, but I would have been absolved at the Pearly Gates.
Sure you shouldn’t have to do that for every single change. But when something significant comes along it’s wise to document it and socialize it.
If OP had done this they may not have been asked to make PowerPoints or dread conversations or hide anything. They could just point at the public doc that explains it all in technical non political language that also educates other teams how they can improve too. This is leadership and mentorship opportunity.
* "Can I just remove or slowly deprecate this?" I.e this job generates data no one uses, or alerts people just bin-bucket or has been already replaced by faster/ better. * "Can I cache this?" * "Can I run this at a lower-priority/off-peak/less available?" * "Can I reduce the frequency or move processing to deltas?"
This is just the easy stuff you can usually do in a few lines of change without even getting into basic optimization rewrites. Most internal things at $large-co are built because someone thought they might be useful, they might get promoted and using that hypothesis and moved on, but few things are actually continuously validated as still generating value > their cost.
This morning on bluesky I saw a story of a very hasty onprem-to-cloud migration that was facilitated, and accelerated, by the rising floodwaters drowning the premises.
this machine kills imposter syndrome
It is incredibly reassuring that it is not simply my organization that is hamstrung by the pretense of complexity, when really someone just made it complicated to make it seem important.
Then job postings would have to be, "we're looking for competence and adequacy. We want to pay you a normal amount and get our money's worth."
I wrote a tool that saved about two hours a case, in total this saved about £500k. I got a free case off beer.
The only "personal" reward I get from that is: whenever I feel guilty for not having done much in a given day, I remind myself that by this action alone, I've saved my company several times what I would ever cost them.
Helps with self-esteem, but I don't think my company see it that way.
Perm is more "we pay you so fix this", consultant is more the reverse "we need this fixed so we'll pay you".
I always find that dynamic hilarious because in a general sense permanent employees have more value than contractors/consultants as perms usually have a much longer tenure at a company (years and years, vs 6 months to 1 year).
However one thing I noticed after moving to the UK is that the culture is completely different here - everywhere I've worked here there are contractors who stay for years and years like a perm would. It's not necessarily a bad thing, but it does make me reconsider why anybody would ever be a permanent employee beyond a bit more job security (ie long term contractors definitely have to trust that they'll be renewed, even if they usually are it's always possible for the business to decide otherwise).
In fact the team was pretty upset that they'd budgeted that money for infra already and it'd have been better spent instead of waiting till next year to re-budget it.
It wasn't necessary though because we had a committed spend target to reach and we just had to figure out how to legitimately spend the money somewhere else. :(
I showed a demo how easy it is to read private ssh keys to the head of infrastructure, and after some months people could connect to network only using custom credentials (ldap) which was good, but also asked us to install "spyware" that among other things checked the firewall. I never installed the "spyware" but nobody pushed me. I didn't think I somehow prevented a disaster or did some heroic deed because everyone in the company was professional and nobody would exploit this. But of course I didn't tell about this to anyone except the infra because such information should not be disclosed until is fixed. And once is fixed why disclose it?
I really miss the Mac checkbox to enable the firewall. On linux I use nftables which is really powerful, but with so many possibilities it is easy to miss something during configuration.
I observed a lot of senior engineers don't have sufficient network knowledge. A lot of people on linux don't use the firewall which is really bad if you work on shared wifi.
Also when running docker images, if you map a port when using docker run (ex. docker run -p 80:80), docker will automatically add firewall rules and bypass the enabled firewall, exposing that port publicly.
I used to think startups would be one of the few places that actually gave a shit about being lean and efficient - but turns out that's only true if they're bootstrapped.
Success and super efficiency was rewarded with additional work, including hinting I should help other departments (apparently a joke).
I received no extra remuneration despite asking, yet my company continues to hire new staff weekly.
I've learnt my lesson, just like this author.
I've been on the receiving end of middle management because I've been able to fix things in the business that have always been broken but never got fixed until I worked on them.
Management will claim it's their work, will give you just lip service, will not use your name higher up the hierarchy and will actively down play their own mistakes whilst blaming the rest of the department or developers (e.g. like building a project with wrong requirements they actually provided, then missing the deadlines because stakeholders demand changes).
Where that has happened I quit and then they just go back to the status quo.
A company is like a chain, as strong as its weakest link. There is not much gain to have some links much stronger than the others.
The famous "I've learnt my lesson" which means you really reached the same low level of incompetence as everyone but secretly thinking you are more competent than all the others.
The funny thing is that among all those incompetent peoples / idiots mentionned there are probably smart people who just learnt their lesson years/months and adjusted to the weakest links in the chain.
I'm here because there's some path-dependence in careers. I started at a mediocre company due to not having a permanent work visa, and have been clawing my way up. I should write something else on this, but I've also realized clawing my way up was unnecessary - it turns out that while I've had some skill atrophy from working at these places, good engineers recognize someone that isn't going to cause a spreadsheet dumpster fire, so I should have just jumped to one of the top rungs on the ladder years ago.
I love this so much
A colleague of mine and me got the opportunity a while back to basically work as wanted to in one side project. So we just worked truly agile, without any of the "Scrum" bs around it. Everyone involved was blown away how successful it was in terms of implementation. Two companies tried it before in the span of a year, and they didn't get it working at all. We worked just some evenings and weekends outside of our full-time job over the span of six weeks, and we got 90% done, with everything essential included to be able to start working with it. We then implemented everything else the customer wanted within the next weeks by following the same principles. There were no problems at all (apart from the customer basically almost running out of money because they spent most of on the previous companies which couldn't deliver).
At my day job, the managers usually argue against anything we want to do, question every decision from our side, want to have regular big meetings with 16+ people to talk about everything at length until no one wants to do it anymore, and then micro-manage everyone to death. We tried to at least keep these guys out of the daily by pointing to the Scrum guide where it says that only people working on tickets should participate, so one of these guys just created a bs task for himself, then talked every day about it for a lengthy amount of time without providing any real value at all.
I never got those silly shows before, which play in offices where everyone just acts silly and they never really work. But now I know why they are so popular. They just provide comic relief for people who really have to work in those environments and it is really as bad as in the television shows, if not worse.
And you can cut this in half by changing some defaults in instance lifetime?
I’m starting to understand why Snowflake’s market cap is something like $50B. This sounds like a nice money-printing business if you can convince enterprises to use it.
The immense cost was coming from someone writing a query that translates to "I need one row of data" and then we get billed like $10-20 in idling compute. With multiple computers and several full-time SQL modellers, it adds up very, very quickly.
As the guy that just has to keep Snowflake running smoothly and isn't paying for it, it's a really nice product. I would still prefer something else on principle because it isn't open source, but eh, I guess it reduces my stress at work.
> ...
> I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.
Wow, so they're still over budget by 300k dollars. This is a funny story, but the company sounds incompetent.
I've seen "typical" spelled this way several times on Hacker News. Is it a British thing or something like that?
Developers - Always do this as bonus paid by contingency of saved money and a signed off scope if it works out, especially if you’re d looking into it on your own time.
Specifically if you like you can get signatures from everyone on the hierarchy on how much money or time this will save, cost or make them.
Do it enough times and the right kind of CEp/President will tell you to stop bringing the business case to them for approval and just do them if they make sense and you have your backup.
You will enter a side door, bypassing most politics and c-levels (and maybe triggering some new), reserved for people who say let me see what’s possible instead of coming back with reasons why it can’t be done.
From there, paint a picture of what if you built a team of only doers, minus talkers across the organization.
:)
Tech should never report into Finance, the group that can’t even tame spreadsheets.
Yes, business owners pay.
I can only testify that this is not my experience at all, so it might be that you're in the wrong place.
In every company I've joined, I've found issues, often large ones, managed to get assigned to them, and had fun fixing them (or at least the satisfaction of seeing them fixed). Then moved on to the next issue.
I haven't always received much recognition from the powers that be (although me and a few colleagues have once been wined and dined at one of the most expensive restaurants in the city for having saved the company 1M$+ with a few days of work each, that was nice), but I've gained respect from fellow engineers and from myself, which was nice, plus lots of things to tell during job interviews.
So, may I suggest getting out while you can and finding a better place to work?
It was a .net app that I reverse-engineered that was basically calling 10 or so encrypted stored procedures in SQL server. Because the older versions of SQL don't actually use encryption, but a type of encoding, I was able to decode them all and replace the entire app with a script that I wrote.
The whole process with testing took about a month (they thought it would take me a year).
The consultant that made it wasn't very happy when he was told they would no longer be needing his services.
Fire the majority of people leaving the A players, unmanaged, and the world will begin to work near-flawlessly in a few weeks to months.
But this line of thinking hurts feelings which are (presumably) paramount to actually solving the problems people set out to solve.
Am willing to bet it was 'KTMJ'... (Batman reference, but... strangely similar to one of the "big-4")
There was a tale maybe 10+ years ago about someone who automated their job with a script or Excel sheet or macro and didn't tell anyone about it. Having a hard time tracking it down again, anyone remember what that was?
- Last year, there was a post on Reddit, with an HN discussion as well: https://news.ycombinator.com/item?id=29994776
- In 2017, there was a question about whether someone should tell their employer that they've automated their job: https://workplace.stackexchange.com/questions/93696/is-it-un...
This is Dilbertland, at its finest.
I was baffled and urged him to be careful and assume he was wrong that it had to be wrong.
Anyways, he is still fixing it, getting some people to validate it.
I would think he will save at least hundreds of thousands a year.
But seriously, why a company that spends millions of dollars in an area would not hire an expert to try to save some money in it is beyond me.
It doesn't make sense until you realize that they operate in an environment where there is no competition that can disrupt their business model. If their costs go up, their rates go up. There is never an incentive for costs to go down.
In short, I was revamping/enhancing a property insurance policy pricing system as the lead from the business side (i.e. portfolio manager/actuary).
However, much of the development team was truly incapable of writing software in any useful way, so I inevitably dipped into C# and SQL to help diagnose issues as we (as a team) worked on the enhancements.
My business coworker and I found something that remains funny to this day: code littered with references or attempts to use the .NET TPL library to make the platform do more things "in parallel," but without actual knowledge of using such a library in practice.
Nothing worked in parallel! There were blockers all over the place! As someone else said, TPL might well have been a //TPL TO DO.
We suspected that TPL was there as one of those "look what I can do" intrusions that received praise from more senior devs/mgmt. even if it never actually did what it was supposed to do.
I can relate to this.
I think many of the people who disagree have either been serving in non-engineering roles since past couple of years, or have bought in the pitch of rosy world created by managers and so-called leaders.
I threw all of their configuration advice out the window and run all extra small at 1 minute idle. No issues for my use case and we paid 17k for the year.
I have to expect that snow will change their pricing model soon.
To put it another way, to rack up $500k spend for 30 days of constant provisioned time:
$500k/(30*24) = $694/hour
Presumably, with the 10 minute blocks, there was idle time where the spend was zero, so the instantaneous amount would be higher.
I have never used snowflake so I'm not sure how this works.
Well duh...of course you'd want that. But it is not so easy is it? I'd also instantly fire half my staff and hire hard working geniuses instead. But that just isn't an option, unfortunately.
> I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.
I know he doesn’t work the same place I did but damn, it sure feels like it.
Definitely a pattern through the places I've worked. The ones that are smart, have good intentions and go a little bit rogue tend to make the biggest impact.
Executives are too concerned with risk, and honestly, it playing out this way suits them better: if someone goes rogue and it works, they can claim the win, if it doesn't, they can blame and reprimand.
When I find someone very talented, I go out of my way to promote them because good engineers who want to stick around and not jump when the times are good is hard.
The sad reality is that I’m limited to how much I can bump their pay, give them time off, etc.
Open-source: https://github.com/datafuselabs/databend Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059 Cloud: https://app.databend.com/
(Peter Principle writ large.)
Only one thing left to do. Leave 'sleep(600)' calls all over the place and resign.
1. Remember who you work for. Yourself. Not your boss, not your grandpa who always wanted you to become a writer. You only have so many breaths before you get your ticket punched, so make them count.
2. Don't seek work as something you should love. Those should be your loved ones, and hobbies etc that stimulate you. Sure there's a small minority who get to do what they love, but over time love can turn to disgust.
3. When starting a job, figure out the organization's incentives. Some orgs want change, some will fight any change. Figure out what the org wants and your life will be much easier. My job is like the Maytag repairman, kind of waiting for stuff to break. I refer to it as being a digital janitor.
4. Nothing you build digitally will last. Hell, between linkrot, bitrot, and the heat death of the universe, nothing lasts. So don't expect what you've built to last, or to hold value. It'll all be re-factored/re-engineered/re-architected and become obsolete.
5. Large organizations are toxic and often sociopathic. So are small orgs. If you can find a way to start your own business, try to avoid becoming that way. Good luck, most businesses either fail or become that way. But at least you have more choice in the matter.
6. Find a mentor (sometimes referred to as a rabbi) who can provide guidance to you when times get tough. Don't choose a rabbi from your reporting chain. You will find yourself adopting a worldview that might not be aligned to your own interests (in other words, the bastards will manipulate you). This rabbi can be a friend, or just someone you vibe with. Be careful in choosing your rabbi, and take care of this relationship.
7. Illegitmi Non Carborundum
Illegitmi Non Carborundum.
It's important to remember this.
To give an example, most large (multi-billion dollars) companies I worked for where amazing with happy people, whereas the small startups where filled with people backstabbing each other to make sure to be at the top of the food chain IF the company were ever to make money one day.
The moral of the story is: anything can happen anywhere.
Should have just let them burn dude
I'm sorry, but this needs a privilege/gratitude check. You are guaranteed your salary, and you're welcome to take on the same level of risk your company is by starting your own. If you think it's so easy go ahead.
Pain I feel very acutely. As an employee, I saved a FAANG company billions of dollars in potential GDPR fines after they carelessly declared 'mission accomplished', weeks before the deadline, and never even got a 'thank you', much less a raise.
When you find shit like this, people will go "my bad" at the worst, and you all are happy the company has more money. Small teams and small firms have their downside, but honesty and transparency, especially when it comes to cost savings... tend not to be one of them.
Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it.
The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%
They're not usually software engineers. They're tool users not tool makers.
So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.
And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort
was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ).
I promise I'll rewrite it in python at some point.
This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.
Never heard that before, but that’s so on point.
Came for the engineering, stayed for the blisteringly on-point observations of corporate life
It had me stuck reading until the very end - usually that never happens!!
Meanwhile big tech thinks the way to reduce costs is to wholesale fire 1/2 their company.