I'm not sure the engineers realized despite their secrecy, it would be noticed by the press immediately after deploy.
But the best part is how Google engineers immediately on seeing it figured "oh yeah, we should do that too" (although they apparently got the necessary approvals however that was done at Google, it was easier to do because they figured "well, youtube must have done due dillegence before doing it.")
Amazing!
I don't know how they didn't all get fired. Like, ALL of em, including everyone who set up the special "OldTuber" priv long before.
But... it worked! This is a hacker story for the history books, it sounds like the kind of thing programmers did 20+ years ago for nothing except the reward of doing it right (against their own career interests), that I feel like doesn't happen so much in a more professionalized industry.
I can think of 2 reasons:
1. The gamble paid off. It turned out the time was ripe. Thus, there was really no negative impact on the organization that people had to be punished for.
2. The OldTubers were the experienced backbone of the group. Firing everyone with that privilege would likely have destroyed the group's technical expertise. YouTube as it was would have died.
That latter point is a bit of a strong negotiation position that many engineers don't realize they have. For example in my team we're having a lot of difficulty filling job openings (we're picky and frankly our interviewing pipeline sucks). As such, there's a significant cost/benefit calculation to firing someone. Maybe the person did something egregious, but is it likely to happen again? I know some people in my team are... not great, but are they so bad that we would be better off without them? (I must reluctantly admit, no)
"He had come in on an otherwise normal day to find email from every major tech news publication asking why the second largest website on the planet was threatening to cut off access to nearly a fifth of its user base. Fortunately for us, the publications had already settled on a narrative that this was a major benefit to the Internet. By their call, YouTube was leading the charge towards making the web a faster, safer experience for all of its users."
Go with the Zeitgeist, and you'll have a lot of wind in your sails.
That seems extreme.
I've worked in some organizations where the assumption is that engineers aren't capable of making product decisions. They are merely walking, talking machines that are there to implement the vision of the PM and designers.
In other orgs engineers are very much in tune with how product decisions are made, what the business is optimizing for, and what constraints they need to be aware of. They will generally work with the product owners to shape the user experience.
I'm guessing that YouTube was much more in the latter category.
>They will generally work with the product owners
is exactly what the engineers didn't do in this story. The company was completely blindsided.
More importantly, the EU's decision on bundling IE with Windows came down in 2007: https://en.wikipedia.org/wiki/Microsoft_Corp._v._Commission
The EU has continued pursuing antitrust cases, and the FTC has now set up a task force to investigate antitrust in tech. The main problem is that the wheels of the legal system turn very slowly, much slower than the startups rise (and fall).
If you choose Firefox or Chrome it never suggests that you should switch to Edge.
Microsoft’s lawyers are not stupid.
Imagine working at a medical or financial company and having secretive 'old timer' permissions that basically backdoored the company's engineering processes. Even if the engineers saw a good reason to do so.
I mean, who are we to say that this was doing it right? It turned out to be beneficial but there are thousands of ways that this could explode in your face if you tried it yourself.
No medical or financial company has this vibrant engineering IT culture like the web startups 20 years ago had.
Everything was new and exciting, and things moved foreward exactly due to the flexibility of all involved, the uncertainty, the flat hierarchies, and the fact that many rules either didn't exist, or only existed on paper.
This is what happens everytime a new market emerges, where structures need some time to settle.
Once they have settled down, procedures become standardized, but they also become so boring that all innovation is lost.
What happend at Youtube 12 years ago is exactly what happens right now at some Cryptocurrency start ups. The start ups that are succesful are exactly those start ups where people like this Youtube engineer do not get fired. If YT had such an authoritarian attitude in 2009, they would never have become succesful.
It's easy for the blog author to say "I have no idea why we weren't fired" but I bet back then it was all fun and games. (Getting fired isn't even that bad, what really harms people is when everyone around them starts to tell them things like "You could have ruined everything!")
And something similar to the excitement and fluidity within the Youtube of 12 years ago happened 80 years ago in pharmaceutical companies, or 200 years ago within a Bank, when such structures were still freshly formed.
> I mean, who are we to say that this was doing it right? It turned out to be beneficial but there are thousands of ways that this could explode in your face if you tried it yourself.
How would this explode in your face, exactly? What was the risk here?
Have you heard about Kerviel? A trader that cost his banks a few billions of euros? He was able to do this because he could bypass reviews by his boss and the internal compliance team.
https://en.wikipedia.org/wiki/J%C3%A9r%C3%B4me_Kerviel
Edit: to be fair, it look like he took the blame for all the failings of the company
Can't we presume that all of this setup happened because everyone involved knew perfectly well that it was just a moderately popular video hosting site? And that they would all have behaved differently if they knew that peoples' life savings and medical histories were at risk? Did it go out of style somehow to presume that people were mostly sane most of the time?
Although... the number of people involved in the "OldTubers" backdoor seems to have been pretty substantial, I don't know if they could really have fired all of them...
But who is anyone to say if doing anything is "right"? We're all equally qualified to have a moral compass.
That's really blowing it out of proportions. It's a minor button addition to the UI and they paid attention to not be sued by the European commission for it for anti competitive behavior. It's a job well done with nothing to criticize.
It's a BAU example of how works get done in a large bureaucratic organization.
https://en.wikipedia.org/wiki/Murder_on_the_Orient_Express_(...
>The next morning, Poirot discovers Ratchett was murdered during the night, having been stabbed a dozen times. [...]
>With the train back on track, Poirot concludes that justice is impossible in this case, as Cassetti deserved death; for the first time, Poirot will have to live with a lie and imbalance.
Because this is how everything on the web used to be done. Google won the internet partly because they moved away from this first and best, but product decision making by engineering and overriding approval processes in the name of a good outcome were pretty much standard back then.
Compare with ex-Firefox VP Johnathan Nightingale's recent thread about Google "amateur hour" and "oopses" that only affected Firefox:
It's sort of the reverse of "nobody ever got fired for buying IBM": nobody ever got fired for refusing to implement an accessibility measure that not even IBM (or Google, in this case) bothers to implement. Unless you're in a very specific sector (in which case you know what your obligations are), your legal duties will always be a strict subset of those of $BIGCORP; so you can use $BIGCORP's shirking of a particular duty as a heuristic to determine whether you can safely shirk that same duty.
Are you suggesting that a new browser that isn't out of beta should be less likely to weirdly break on some websites?
Now why they're still sniffing user agents instead of doing feature detection is a good question for Google, who themselves, to the best of my knowledge, push for feature detection instead of sniffing user agents as best practice. Do as we say - not as we do?
It's one of the reasons I stopped accepting lawyers and law firms as clients. Every other client would pay on time. Those in the law field would consistently slow-pay, I believe because they knew they wouldn't get sued over it.
"That overdue invoice? Oh, we never received it. Send it again." "Oh, Jenna in accounting must have it, but she's on vacation for two weeks." "Oh, that's in process." "Oh, we still haven't received it." "Oh, you have a signature on a certified mail delivery? It must be upstairs for approval." "Oh, we've already run all the checks for the month, it'll be in next month's batch."
I eventually went to a pre-paid hours, payable by credit card-only model. It was the only way to stay afloat. Plus it was delicious when someone would use up all their hours and give give some excuse.
"Oh, we really need this on the site today." "No problem. I can do that as soon as you buy more hours."
"Oh, we'll get that paid next week." "No problem, your website went offline an hour ago and will be back when you buy more hours next week." Poof! payment comes through three minutes later.
The best you can do is test the nightly or beta track all the time (if they have one). But that may have false positives due to bugs on their end. And once you notice the bug, you still have to fix it. This isn't something to drop everything for like a security bug, so it may have to wait on other things.
Meanwhile, people outside the company will assume it's intentional sabotage, because that's how the Internet thinks these days.
Google did an "oopsie" and blocked Firefox users from using Gmail
Sounds great.
It doesn't matter if they were intentional. The author mentioned there may hav been hundreds of these "oops" events. Assuming it wasn't intentional, Google knew there was a problem and chose not to fix it.
"Fool me once, shame on you. Fool me twice, shame on me."
Or in Google's case, "fool Mozilla a hundred times..."
Can you elaborate? I kind of agree on the Kubernetes part in that it's really biased towards Google-scale, but I fail to see such biases in Go, despite having used it as my primary language for 4 years now.
Web developers the world over hated dealing with IE6 lack of compliance with established standards. Microsoft even had a newer versions of IE that was better. Everyone wanted this, it's just that nobody was willing to do anything about it.
This is the world we live in. Even with a near worldwide consensus on what needs to be done, nobody is willing to say it for fear of repercussions. People would rather jump through hoops to support the status quo than risk backlash.
Another problem is that the people who would normally have to approve a change like this are not the people who had to jump through the hoops to support IE6.
We too got fed up with all the IE6-specific hacks we had to maintain. One day on the login page, we added a "IE6 might be a HIPAA violation, please upgrade your system" banner. It was technically true... the browser was well past its end-of-life support and was acquiring a running list of unpatched security holes.
Our analytics showed the remaining holdouts upgraded their systems over the next few months.
Unfortunately, that attitude in healthcare leaks to things which _are_ connected to the internet, and you get disgraceful incidents like the hacking of Britain's NHS in 2017.
Years ago, we even tried turning it into an intrusive pop-up for a percentage of users. They just clicked through the pop-up, presumably without reading it.
I wonder if it worked in this case because it started a movement?
> Between YouTube, Google Docs, and several other Google properties posting IE6 banners, Google had given permission to every other site on the web to add their own. IE6 banners suddenly started appearing everywhere. Within one month, our YouTube IE6 user base was cut in half and over 10% of global IE6 traffic had dropped off while all other browsers increased in corresponding amounts. The results were better than our web development team had ever intended.
I think this is the answer. Most people at the time weren't on IE6 out of choice. The fact that youtube (or more likely google docs) had this banner, gave disgruntled employees a better excuse to force IT to upgrade than "I don't like this browser"
I guess they can use this as an excuse to not upgrade their shitshow flash app that beings a multi-core processor to it's knees just navigating between entry fields.
It was continuing a movement and giving it visibility outside web developers. The youtube banner happened mid-2009, web developers around the world had soured on IE6 and actively trying to kill it off since 2005~2006.
What the youtube banner provided was a way to finally get the word out, and an argument to give PHBs: it became much easier to sell browser upgrades (or not supporting IE6 anymore) when you could point to Youtube / Google and go "these folks have put their foot down".
As a web developer at the time, I could put that in my proposals, saying “it’s the standard Google browser support policy”. At which point clients signed off on not needing legacy browser support.
The perspective on outdated (insecure) browsers is becoming more broadly accepted.
We installed Firefox 4 and it ran at a perfectly fine speed...
Over the last week we hit the jackpot at work, in that we had errors come into Rollbar caused by every version of Internet Explorer we don't support all the way back to 6. Yes, there was a little celebration in our dev team when we scored our first IE6 exception. :)
Ultimately, I think top-down change at enterprises is much faster than bottom up, and a national news story is just the sort of thing to spur a lot of top-down changes.
Don't underestimate the power of employees to influence employers.
iPhones didn't end up all over the enterprise space because Blackberry walked away from the market.
My guess from my time in the trenches is about 1 in 5.
http://gs.statcounter.com/browser-version-partially-combined...
That's what you train users to do when you show them intrusive popups that aren't actually important.
For example, tomorrow Google can implement a DRM that would require a plugin that works on Windows, Android (with Google Play Services) or Mac, but not on Linux. After all, Linux is not a DRM-friendly system (allowing the user to hack anything is not what copyright holders want), and almost nobody uses it on desktop, so why bother supporting it? Or Google can use it against new, not yet very popular browser, to slow its adoption.
I was reading your comment and wasn't sure if it's well hidden sarcasm or you are seriously don't know this already happened:
https://news.ycombinator.com/item?id=19553941
Basically you cannot watch high resolutions streams on Linux already even in Google Chrome because of DRM Google among others helped to push into HTML specification.
My Firefox is up to date (60.6.1esr), and Skype Web tells me it's unsupported and refused to work.
AFAIK Web Skype no longer works in any version of Firefox.
I smell an untold story... maybe one of the other teams' banners was accidentally visible to IE7 users as well? Or did IE7 sometimes spoof IE6?
IE8 was the included browser with Windows 7, which was generally well received compared to Vista which included IE7.
The timing lines up within the release window of IE 8 (March 2009) and Windows 7 (July 2009), so potentially a lot of upgrade push from other things in the air at the time.
Edit to add: people on ie7 because it came with Vista probably had a more capable/pushy update system than those on XP with ie6; but I don't remember the details on how microsoft system updates worked at the time.
the people on vista had the pushiest update system of all: vista was terrible, and most users (especially organizational users) switched to windows 7 ASAP.
It was a terrible piece of work and I was glad when it never achieved any significant market share.
IE7 didn't have tabs, IE8 did.
http://gs.statcounter.com/browser-version-partially-combined...
Corporations don't have single agendas, they don't think with one mind, they can't be simplified to a single narrative.
Rather, they're collections of 1,000's of individual each doing their own theing, and the CEO is trying (and often failing) to herd the cats in a single logical direction.
Plenty of good things (like this) can come out of it. But also plenty of bad things, like security breaches, anticompetitive behavior, and invasions of privacy.
Whenever anyone says "because Google always does <x>" or "Google is always like <x>", a story like this is a great antidote.
(And now we’re back where we started with Chrome as the new IE.)
Except for the part where you get BSoD and infinite recursion or any of the other stated IE6 nightmares...
Really, the only thing Chrome and IE have in common are market share. Chrome is magnitudes better than IE in every other way as far as being a web dev goes.
Easy to forget that now.
Also, did you know IE6 pioneered the browser web dev tools? IE6 was the first browser you could debug your JavaScript, css and html, which was copied by firebug about 12 months later. I can't even remember what it was called now, IE Dev toolbox? Something strange, you had to download it separately.
And Firebug and Chrome Dev tools copied it almost verbatim. I've never seen any sort of acknowledgement of just how much the community owes to that early tool. The announcement blogs about it were still accessible a few years ago.
The longer Chrome stays popular, the more websites will optimize for Chrome, bringing back the days of "Best viewed in Internet Explorer". And as more and more sites become accessible only in Chrome, user share tilts further in Chrome's favor, and so on in the vicious cycle we experienced in the browser dark ages.
That's the fear, at least. I don't think anyone is concerned that Chrome is going to start BSoDing Windows anytime soon.
The omens of monopoly are certainly here, though. All of Google's websites, which constitute a large percentage of web traffic, have been a "best viewed in Chrome" affair for a few years now. Microsoft recently dropped support for Firefox on the Skype web client. And I've seen my fair share of niche/corporate/etc websites that are Chrome-only.
(FYI, I'm not personally worried that we'll see the same browser dark ages as before, but it's something we should be vigilant against.)
I'm sure we'll find that a Pareto Distribution is an unavoidable market phenomenon, but I still believe its healthy for technologists to remain skeptical when one client massively outperforms others.
Soo like - you mean: built in proprietary app stores, random DRM-enforcing blobs with camera/microphone access, phone-home tracking behavior, and gobs of memory use?
side note: Firefox not IE user.
Google Meet does block me though..
Yes, it triggered a GET for /. But that generated HTML (usually the service's homepage, as was our case), which the browser would attempt to parse as an image, obviously failing. It would not trigger a recursive fetching of all the resources on the page. Even without recursion, it already inflicted major damage, because our service's homepage was dynamic, while the resources linked from it were mostly static (and thus a lot cheaper, as well as cacheable). I think I would have noticed if it multiplied other traffic, not just the homepage.
This was the bane of my existence for many months. Every few weeks I would have to fire up Dremel and try to figure what was causing the spurious page loads. I hated and still hate SQL, so that was no fun. I knew when it was time to investigate thanks to our human monitoring system: our PMs would get excited or puzzled by a sudden jump in the page view dashboards. (They lived by those graphs...)
Thank you Chris and co. for your contributions in killing the browser version from hell.
Around the same time, there was a Chrome in IE plugin that was also suggested for other applications but never got approved.
People complain about the progression/changes in JS since around 2010 (node, commonjs, es5+). But nothing is so bad as dealing with the really old browsers. IE6 was decent at release but became a boat anchor to the industry. Even then, you couldn't pay me enough to ever support IE4.x + NN4.x ever again.
Wonderful story.
This was one of the happiest days off my web development career when I could finally tell clients to drop IE6 support because "even Google is doing it."
Amazing to hear how the tail wagged the dog to achieve this :)
From '08-'10, I was in web design & development at a company that was neck-deep in IE6 dependency. The animus I harbored for IE6 was so intensely palpable that there were days where I was mere moments from getting a tattoo permanently documenting my burning hate for what I still reckon as the worst piece of software ever known, based on reach, potential for issues caused, and total net effort expended on all mitigations.
> Our boss, in on the conspiracy with us, had thoughtfully recommended that we randomize the order of the browsers listed and then cookie the random seed for each visitor so that the UI would not jump around between pages, which we had done.
It's not exactly clear whether the boss was in on the conspiracy - or whether this was a story told to satisfy the lawyers. If not, this seems a viable strategy for managing the temporary blowback of bending the rules to do the right thing: if you can, make sure your boss ends up looking good.
I mean, randomizing the order is obviously the right thing to do when you hear it. The reason companies don't want people doing this without going through the proper channels is... what if the boss hadn't happened to think of it, or hadn't been in on it?
Of course, in reality in this case, they would have just fixed it when they noticed, it in fact wouldn't have been a legal disaster. But anyway.
YouTube started a domino effect with this. I remember I was working at a web agency at the time. And when I saw YouTube's banner regarding IE6 it was my "this is it" moment. I rushed my boss trying to convince him we should stop making our client websites IE6 compatible. He considered it and started incorporating YouTube IE6 stance in every client proposal from that moment.
Oh man. At our org, we called this "being a cowboy." We have a lot of process to prevent cowboys now. Oh, the good ol' days haha.
"Shortly thereafter, the Google Docs engineers whipped up their own IE6 banner and pushed it into production, presumably under the mistaken assumption that we had done our diligence and had received all of the necessary approvals."
> In many ways what they did was unethical but the result was probably positive overall.
And that kind of thinking is the other, newer branch of ethics: consequentialism/utilitarianism.
My conclusion is that deontology and virtue ethics are necessary heuristics as actually calculating or estimating the consequences of any action are almost always effectively impossible, but, ultimately, what we care about are exactly those consequences, so we're never able to adequately 'cover' morality without, at least sometimes, explicitly considering them.
Web browsers have gotten to be MUCH more resource intensive than the IE6 days. Try loading any "modern" site on a "modern" browser on a netbook. (Or any computer with an Atom processor).
While its "compatibility" has waned, I really appreciate Opera 12 for its performance on humble machines. No modern browser seems to match its resource usage.
They didn't, the current web browsers are faster than ever. The problem is that websites are getting heavier and heavier.
There are many tricks played by modern browsers to speed up page speeds. Many of them increase resource usage in the process.
I love this so much, it's so punk-rock. It's like John Henry's signature on the Declaration of Independence.
Microsoft, that fucking browser caused so much grief YOU LITERALLY HAVE NO IDEA unless you were there. My cursewords are barely scratching the surface of the rage. Frontend dev would literally double in work if requirements dictated IE compliance. I have no doubt that it informed both the decision of many devs to head to the backend and stay there (like I did) as well as Material Design from Google which is not nearly so dependent on spacing being rendered precisely.
However, this is setting a precedent for restricting websites to certain "acceptable" clients, which is not a good direction.
The entire point of the end of the browser wars was to restrict websites to acceptable clients, i.e., those that provided reasonable support for web standards.
Microsoft was way behind in the Web era and IE6 was attempt to slow everybody else down and create web that works only with MS software.
The IE6 was prime example of the Microsoft strategy of embrace and extend using market share. They build software that included harmful features, broke standard or for no reason and had intentional inconsistencies.
Microsoft was pure "engineering evil" during the Gates era.
On a related note, I feel like most of my career has been spent preaching things that I should have just asked for forgiveness for. I can't count the number of times I've heard "we want x, but we can't do anything to achieve x". Psssst, you can - you just need to do it. Scared of change/ the unforseen. If you don't know what the ramifications are going to be, there could be positive ones you're missing too. Try it. If it's truly sinking your ship, kill it. Otherwise sail off into the promised land. Rinse and repeat. Be brave. That's my advice.
Man those were rough days.
And we did.
OldTuber granted you the ability
to completely bypass the new
Google-oriented code enforcement
policies, enabling anyone
holding it to commit code
directly to the YouTube
codebase, with only the most
glancing of code reviews from
anyone. No need for code
readability. No need for
exhaustive tests. No need for
maintaining code coverage. If
you broke the site by improperly
wielding OldTuber status, it was
on your head and you would lose
the privilege immediately, if
not your job. So you just had to
be a good citizen and never
break the site.
I don't know why I get these warm fuzzies when reading about the Internet of yesteryear.
We still see this today with the fragmentation of IE and Edge.
Sorry to disagree with the current "you're my hero" trend, but in my mind, this story just show that a bunch of irresponsible hackers can do whatever they want to ease their work - for which they're paid btw - without any regard on the impacts to users that may rely on the service.
This time, it was only showing a warning message (that may have frightened some people)... and what's the next step? Decide to allow only Chrome-users to use youtube? When a company try to reach a monopoly status, it bear a social responsability.
(anyway: I'm happy for IE6 been a thing of the past)
But MS IE 6 was significantly more entrenched: I have seen plenty of MS IE 6 - only sites and internal tools, but never MS IE 7 - only ones. They existed, probably, but were obviously were rare.
Those where indeed interesting times at both TechCrunch and YouTube, but...
“Glory days well they'll pass you by; Glory days in the wink of a young girl's eye; Glory days, glory days”
If the Feds every want to try to break up G, this will be submitted as evidence.
Edit: this is exactly what a monopolizer looks like in action. One tactical move at a time, until they have full control of an adjacent layer in the value chain. In this case browsers.
I think this was fundamentally different than a monopoly move. A monopolistic action would require a user of one Monopoly service to use another service by the supplier, such as a browser. That is absolutely not what happened here. In this case, YouTube asked people to upgrade to a number of options, including a newer version of IE. Notice that they did not require tying of one of their services with another. I think that makes it completely different.
Enjoy your new Chrome master.
I don't like the Chrome monoculture, but it's a different and unrelated issue. IE monoculture was even worse, and it had to die - I just wish the lessons from that weren't forgotten so easily.
IE6 deserved nothing but to be disposed of.
Chrome as much as it has its warts is nothing compared to the mess that was IE6
Largely it was a waste of time but it did pique the media's interest.
And a fifth of YouTube users at the time.
"IE6 users represented around 18% of our user base at that point. We understood that we could not just drop support for it"