Engineering from >10 years ago seems like it was a wild west. Some truly stunning pieces of technology, strung together with duct tape. Everything had its own configuration language, workflow engine, monitoring solution, etc. Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. I don't want to disparage engineers who worked there during that time, the systems were amazing, but everything around the edge seemed pretty disparate, and I suspect gave rise to the "promo project" style development meme.
Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID. None of these are perfect by any means, but the convergence is a really good thing. I switched team, PA, and discipline 6 months ago, and it was dead easy to get up and running because I already knew the majority of the tech, and most of that tech is pretty mature and capable.
Maybe Google has become more like other tech companies (although I doubt they have this level of convergence), but I think people glorify the old days at Google and miss that a lot of bad engineering was done. Just one example, but I suspect Google has some of the best internal security of any software company on the planet, and that's a very good thing, but it most certainly didn't have that back in the day.
> Deployments were infrequent, unreliable, and sometimes even done from a dev's machine.
Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.
Additionally, rolling out from a dev machine brings so many risks – security, reproducibility, human error, and so on.
I'm glad this is not the way things work anymore, and for the most part things are more reliable as a result.
On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).
It's been interesting to see the tech and public zeitgeist shift on this.
Lesson: if you're going to be popular for killing a king, be careful how similarly the thing you place on your head starts to look to a crown.
The culture turned on Google around the same time Google lost their innocence and dropped the "Don't be evil" clause (note the dropping of that clause was not the cause, just one of the symptoms).
I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.
That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.
The way I like to look at it is that small things take a while, but big things happen remarkably quickly. For example, rolling out a "hello world" service might take a few days, but having that service serve 1M QPS is pretty much free (in terms of effort). At my previous place a new service might have taken an hour to set up a deployment for, but having it serve 1M QPS would have required overhauling several aspects of our infrastructure over months.
Could you elaborate on what these are for a non-googler? Ironically enough, Google isn't very helpful.
P2020 + Rollouts: This is for intent-driven deployment, where deployment configuration for jobs, networks, pubsub topics, and other resources are declaratively checked into source, and the Annealing system automatically finds diffs against production state and resolves those diffs (aka a rollout).
Automated job sizing: load-trend driven automated capacity planning. Separate from autopilot, which is a more time-sensitive autoscaler. This will actually make configuration changes based on trends in traffic, and request quota for you automatically (with your approval).
BCID: this is for verifiable binaries and configs in prod, where it requires two parties to make source changes, two parties to make config changes, two parties to approve non-automated production changes, and only verified check-in binaries and configs can run in prod, not stuff you build on your desktop machine.
history's biggest yakshave...? not surprising for a programmer-run company
I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more
Unified rollouts, unified versioning, universal dashboards, security compliance, standardized quota management, standard alert configs. It's opinionated, but I can drop into any team and hit the ground running. I don't want to learn your custom dashboards doing the exact same thing with different names.
The issue with PoD is that it's a great concept and implementation that's tight on resources, and doesn't have much of a plan to expand beyond its current paradigm. The P2020 team deserves way more recognition for all the work they have done.
It was a real mistake, SRE is hugely stressful and really unrewarding compared to SWE. Yes you learn some skills and get some occasional glory, but year after year of fighting fires really didn't build any long lasting career.
After switching back to SWE I've finally got promotions and pay rises again, as well as good night sleep and much less stress.
Back to the OP, I raise a glass to your sabbatical. Most SREs end up needing a healing period from repetitive stress injury (AKA burnout).
If I may offer some completely unsolicited advice, don't put too much pressure on yourself in the next few months. People who gravitate towards SRE work tend to thrive under short-term ambiguity and emergency/urgency. However, long/medium term ambiguity without a clear productive goal can quickly feel like a crisis. OP mentions this in closing, so I'm rooting for them to rest and "sit still" for a bit.
I cannot disagree more: our team is healthy, oncall is quite a fine activity to do (and compensated, of course), we have plenty of engineering work to do.
I've had five promotions (and tripled salary) and done so working on plenty rewarding activities over time. I've done from deployment automation, to capacity planning, distributed system design, large data migrations, designing ietf standards for auth protocols, wrote client sdks, now we even do AI for different things (including model development).
I'd recommend to not generalize from "I didn't like it / the experience wasn't a match for me" to "the role is shitty".
> oncall is quite a fine activity to do (and compensated, of course),
Overnight on call is never compensated. I know some tech companies pay but I've never seen it.
> deployment automation, to capacity planning, distributed system design, large data migrations, designing ietf standards for auth protocols, wrote client sdks, now we even do AI for different things (including model development).
To me that is mostly SWE work (capacity planning and migrations perhaps is SRE). in regulated environments SREs are explicitly forbidden from making changes to the code base.
> I'd recommend to not generalize from "I didn't like it / the experience wasn't a match for me" to "the role is shitty".
Agreed.
I miss working in software that also utilizes my skills in infrastructure, but I do not miss the constant escalations, terrible on-call schedules, and only about 20% of my time being spent on the rewarding parts of the job.
I think in general that type of experience would help Developers be more empathetic to the operations side of things. Those fires often come from trade-offs made in development.
I think it made me a better developer because I've seen a lot of what can go wrong. Probably reduces my productivity but ultimate my stuff is more likely to work.
The root cause of this disaster is that, when writing software, interruptions are the death of productivity. Having a software engineer wear too many different hats at one time, especially when some of those hats are largely real-time interrupt driven, can absolutely kill productivity.
To emphasize, I'm not at all in favor of "throwing things over the wall". Software engineers are responsible, for example, for making software that is easy to test and has good observability in place for when production problems show up. But just because you listed a bunch of things that are "critical for software development" doesn't mean that one person or role should be responsible for all of these things.
At the very least, e.g. for smaller teams I recommend that there is a rotating role so devs working on feature development aren't constantly interrupted by production issues, and instead each dev just gets a week-long stint where all they're expected to do is work on support issues and production tooling improvements.
I ask this question here because there seem to be quite some (ex-)Google employees in this thread.
Due to "an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank"?
https://cloud.google.com/blog/products/infrastructure/detail...
If I take another software gig it will certainly be at a small company where my daily work contributes directly to the company’s central goals.
Programming is a superpower that can change the world. Yet the best paying jobs for programmers are at FAANGs building systems to peddle ads. .
It got to a point of being almost farcical, where they were scheduling meetings at 9:30pm multiple times a week. After two years of it I had to leave, I was coming home catatonic and depressed, to a point where my wife was getting concerned.
What you're seeing from Google in the last ten years is a maybe predictable consequence of that culture, where some Googlers really do seem to think they're generically much smarter than everyone else, about everything. You started to see mass scale social engineering via manipulation of search results and products, driven apparently by the immense faith they have in their own wisdom. Is there any claim Googlers cannot immediately resolve as true or false given nothing more than a few ML models and a team of contractors in LatAm? Apparently some of them think that's all it takes.
This quasi-misanthropic culture is miles away from the trusting "make it universally available and useful" culture the company once had, but the seeds of that culture's end were clearly visible even at the start. You can't constantly validate people by telling them they're super smart before some of them come to actually believe it, and that leads naturally to the belief that if they're really the smartest people in the world then surely that means they should be running it.
its the proximity, pedigree, profile that you have to fit to get in to Google
I'm happy for those that made it. Not everybody gets to work for Google. But the work they do are no less challenging or more important than what the rest of us do.
If anything FAANG has contributed greatly to the American Firewall of Algorithms and have destroyed an entire generation's ability to reason and value common sense.
I remember this quote which I can't remember who said but "if they are paying you a large salary, what they take from you is far greater"
Hmm, this seems like a nice thing to tell oneself to avoid feeling underpaid :)
It's like the fourth time I read/hear this. I understand that it's a tricky one to adress.
I’ve worked for European and Asian owned companies, and they seem to be able to handle distributed authority much better. For the “land of the free”, it sure seems like US companies run on a feudal system.
Heaven is an American boss, Chinese cook, Russian wife and English house.
Hell is an English cook, Chinese house, Russian boss, and American wife.
scrum master: “points per sprint is going down guys!1!1 (after moving stories to next sprint). Velocity is gucci”
If you spent the last 9y at Google in SF/NYC (the top comp regions), you'd have a million dollars in stock alone. It doesn't go as far as it did 9 years ago, but it's still ahead of a whole lot of people in this economy.
I am one of those engineers who do not care about culture as long as I am getting paid for the efforts I put in. Google in that sense beat others by HUGE margin.
The engineering work was however very different. We focused on right engineering solutions instead of just business aspect. While that kind of attitude hurts us in short term, it pays big in long term.
Then large companies are not for you. Navigating the culture is key to advancement and long-term satisfaction. Otherwise you will feel like an outcast and likely let go during layoff rounds or kept around at lower compensation rates.
Whoah, it seems fantastic! That alone seems like a good reason to work for Google. Unfortunately, none of the companies I worked for was interested in less than 100%. I told them many times, you can keep your money, I just want to spend 20% or 30% less time at work, but they always insisted on 100%. I have a feeling they would go for 120% if legally allowed.
I'd love to know how in a fast paced office environment you can improve those, none of the trainings or anything are about this (even the leadership like ones feel like just standardized template stuff rather than actually have an environment where you can practice social skills and get the correct timely feedback to improve it)
How much, do you think?
They have a great business that prints money.
The problem Google has is making more money printing business lines - like Microsoft.
I do hope Google turns around and gets back to its roots.
There is nothing all that special about Google. Maybe there was twenty years ago, but that ship has long since sailed. It’s just another large US tech company. Like Microsoft and IBM before it.
This is just a hyperbolic statement that should not be taken seriously at all.
Look, Google isn't some fantasy land that some people might have lauded it as once upon a time, and it isn't unique in terms of pay or talent, but it is certainly at the top echelon.
I did an interview loop for high level IC at both Azure and GCP simultaneously and the different in talent level (as well as pay) was astounding.
IBM has never a company where engineers could rise to the same ranks as directors and higher with a solid IC track.
Is Google special compared to Apple/Netflix/Meta? No. Is it special compared to Microsoft, IBM, and any non FAANG or a company that isn't a top deca-corn? Yes.
It's a similar trajectory is what people are saying. When Google was small and everyone wanted to work there they could take their pick of the top talent. When you run a huge company you inevitably end up with something around the average. I.e. all those huge companies that pay similar wages and do similar work basically end up having similar talent +/- and within that there are likely some great people and some less than great people.
Never? Maybe if you’re talking the last 15-20 years, but IBM has been around a lot longer than that…
I personally know people who moved up the ranks there to director and above, so I can say with confidence you’re absolutely wrong.
This is maybe the third time I've heard this mentioned here on HN, so now I'm curious: What specific kinds of differences?
I imagine there might be a certain kind of prejudice against Microsoft and its employees, especially for "using Windows" or whatever, which I've found often unfairly coloring the opinions of people from Silicon Valley that are used to Linux.
If you don't mind sharing, what specific differences did you notice that gave you a bad impression of the Microsoft team and such a good impression of the Google team?
However, during recent years I have turned into a Google hater. He does not mention any of those aspects. Google is an evil business IMHO. They are an advertising company. The challenges for this planet are sustainability. The goal of advertising is to waste resources. I can type this on a low end phone that soon turns 10. It works perfectly, except that no recent Android version is supported. Google is in the business that cores and memory have been doubled several times since then, for no benefit to mankind. And phones are far from the only category, advertisement is about selling a lot of stuff that does not bring any true improvement in quality of life. Video is one of the worst energy wasters in computing. 90% of Youtube is useless crap, not worth destroying the planet. Nobody would pay a realistic price for it. They are an ugly oligopolist. The list could go on and on...
The claim that more memory/computing power has no benefits at all seems nonsensical.
Video games, supposedly the Killer App for high-end computing hardware, are among the worst offenders: your average modern 2D side-scrolling platformer (Mario clone) locks up my computer (to say nothing of AAA games). Web browsers are the second-worst offenders. Chat clients (secretly web browsers) are third: they barely work, and interoperability has gone the way of the dodo. Operating systems are fourth: they (mostly) still let you use older versions of software, but they have mostly just grown new problems (e.g. built-in adware in Windows; GNOME… well, GNOME) without fixing long-standing ones (e.g. slow domain login in Windows; most systemd misfeatures).
> 90% of Youtube is useless crap
That is the way of all things: https://en.wikipedia.org/wiki/Sturgeon's_law
But the best of YouTube, people are clearly willing to pay for that.
Anyway, congratulations to those Googlers who "picked" GOOG by not selling their GSUs and who made out like bandits. You certainly got lucky, and nobody should expect their own employers' RSUs to do the same.
Googlers from the past ~15 years are comparatively filthy rich. Imagine having had an opportunity to miss out on 8 figures of stock returns and only be left with 7 figures because you followed sound financial advice. Bruce Willis is no doubt dabbing his eyes on Memegen.
The last 15 years were good in tech but the earlier Google employees did even better.
I'm guessing neither of us made money on bitcoin or gamestop... I don't lose sleep over that.
A bit of perspective may be warranted. There are single mothers busting ass raising kids on tips and food stamps, while you're asking for sympathy for making out with only seven figures from your Google stock options.
I'm pretty sure the luck of being born with sufficient intelligence to work at Google is orders of magnitude greater than the luck of making big ROI on Google stock.
> because you followed sound financial advice
Most financial advice is there to help the clueless masses to not lose their savings to get-rich-quick schemes and then default on their debt during their next unemployment period. Anyone smart enough to do Silicon Valley work has probably outgrown those training wheels and can think for themselves when investing.
> It’s a shallow post-mortem
I respectfully disagree. It’s an 8 minute read. Sure, it’s mostly in dot-point form, but personally I’d rather that than some massive 80,000 word blog post that I’m going to drop 1/8 of the way through.
Since when does a personal blog post need to be a well constructed and lengthy document?
And it is worth noting that a lot of the bullet point lists do start with "I made a ton of money" in as many words, which is also just not very interesting to most folks, though it is certainly very relevant and important to the writer.
But a well articulated, technically correct post that’s evenly paced? Heavenly. Its a David Attenborough documentary for tech nerds. It’s exactly what I am after.
I wouldn’t say anything against having an upfront summary for people who don’t have time/patience.
9 years working at a top tech company of bleeding edge work reduced down to “I made money. Oh, I made money. My stocks did well, so I made even more money” is the pinnacle of intellectual laziness. That’s not a postmortem by any stretch of imagination.
Slight OT: what do people recommended for simple server & db monitoring (for a small saas business)?
monit, nagios, victoria metrics, etc?
Lots maybe. But if it's under 8 figures before the dot, it's not tons.
Is Google really different from other companies? I talk to a lot of Amazonians (AWS Hero, FreeBSD/EC2 maintainer) and my general impression is that developers below L7 ought to be classified as "Junior" -- my mapping is basically L4-L6 = Junior Developer, L7/L8 = Developer, and L10 = Senior Developer. Anything which doesn't have L7+ involvement gives me major "these kids need adult supervision" vibes for all the newbie mistakes they make.
Fifteen years ago L5 was actually senior, L4 was a developer, L3 was a junior developer. L6+ = you owned major user-visible features with hundreds of millions of users. L9 = you did something world-class like invent BigTable or Google News, and L10 didn't exist.
This seems extremely surprising. I can believe that the 2010-engineers were more technically capable, but there was also a lot less non-technical complexity involved in getting things done in 2010 than there is today.
Amazon and Microsoft also have less "alignment" at various levels compared to silicon valley due to literal geographic and historical reasons. Principal SWE at AWS is probably ~L6 at G in my experience. Obviously there are always outliers in all directions.
As for L10, that's Distinguished Engineer. I think if they're managers they're also called VPs? I'm not exactly sure what the deal is there; but I know plenty of Amazon L10s who are fantastic engineers.
