Running the bot nowadays is hard, because a lot of sites will now block you - not just by asking nicely via robots.txt, but by checking your actual source IP. Once they see it's not Google, they send you a 403.

Cost, presumably. From the article:

> Microsoft spent roughly $100 billion over 20 years on Bing and still holds single-digit share. If Microsoft cannot close the gap, no startup can do it alone.

There's one great example of a company that did that and managed to go viral on their release, Cuil. They claimed to have a Google size of search index. Unfortunately for them their search results weren't good and so that visibility quickly disappeared.

Going further back, AlltheWeb was actually pretty decent but was eventually bought by Overture and then Yahoo and ended up in their graveyard.

For everyone else it's the longer grind trying to gain visibility.