When I worked at Google I was lucky and had coffee with Jeff Dean and Sanjay Ghemawat many mornings when my desk was near them and we had shared interests. I got the chance to quiz Jeff a bunch about the early days. When they joined, L&S had already handed indexing off to a couple programmers who had written a system that had to be run all the way through (all steps of indexing) to build a whole new index. Any failure in any step- even just one simple worker- meant you had to run all over again. That led to the development of MapReduce, GFS, and BigTable, which allowed google to scale search while also improving search prerformance (latency of a query, latency of crawling hot documents and having them appear in the index). Jeff definitely didn't have a high opinion of Larry and Sergey's programming skills.
But then, the search engine was really just phase 0 in Larry's attempt to revoluntize the world of information, sort of the things you have to do at the start of a realtime strategy game to get your tech tree up to AI.
No comments yet.
