Suddenly, Hacker News is not the first result for 'Hacker News' (opens in new tab)

Hi Paul,

A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling.

If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site.

Let me know if you need more help.

Edit Detailed instructions to set a custom crawl rate:

1. Verify the site in Webmaster Tools.

2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu.

3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead.

4. That opens up a form and choose his desired crawl rate in crawls per second.

If there is a specific problem with Googlebot, you can reach the team as follows:

1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details.

Thanks!

Pierre

Have you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages?

It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML.

There's an example of doing such with nginx here:

http://serverfault.com/questions/30705/how-to-set-up-nginx-a...

With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.

Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.

Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling?

Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

reddit had the same problem. We set up a separate server just for the google crawler with it's own copy of the database, so that the queries for old pages didn't slow down everyone else.

"This isn't HN getting some sort of preferential treatment, this is just the effect of having a userbase full of hackers"

Of course it's preferential treatment. And if you scan the last month or two of Matt's comments they are general in nature and not specific as in:

"I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly."

You don't think "pinging the right people" and "get this fixed pretty quickly" is preferential treatment?

It really is but I was personally kind of glad not to see HN have that high of a ranking on Google for the term.

I remember when I first started visiting HN I saw all these smart people and the tight community and I was amazed that something that felt so close-knit and exclusive yet was still open could still exist these days.

I was a lurker for a long time before I actually signed up and participated because I honestly felt like I swasnt entitled to be part of "the group" and I should somehow earn my wings. Then in late 2010 I signed up but didn't submit for a bit and didn't join discussions. I still felt like I didn't have enough to offer. I now feel like I've somehow earned the right to be part of this community though in hindsight I'm quite embarrassed of my first few submissions.

So this story does have a point that I'm about to get to. I first heard of HN through an article in GQ and then forgot the link. I couldn't find the site again after searching Google for "Hacker News" as easily as I thought. This frustrated me slightly back then but now I think it's a good thing.

As the size of a community gets larger the quality of comments and submissions usually decreases. Letting people join HN freely and openly is a great thing but I fear that if it became a huge sensation then we'd be inundated by garbage submissions and comments way more frequently. I know about the post on how newbies often say HN is becoming Reddit and all that so I do try to remember that.

So the point is that not everyone respects communities like this and are thoughtful about joining and how they choose to interact on communities like HN the same way I was and I feel like maybe it's okay if Google isn't giving us the best ranking for certain terms. I mean, HN is easy to find still, just not that easy to stumble over.

The question is, 'Was Matt Cutts ever THE Googleguy?'

   ( http://www.webmasterworld.com/profilev4.cgi?action=view&member=GoogleGuy )

I always wondered who it really was years back.

We do include some information in the search results for cases where our users could be harmed, e.g. if we detect malware or if we think that your site may have been hacked. We've been exploring other ways to alert site owners that there's something important affecting their site.

But people do choose to remove their sites, so we can't always tell between a mistake vs. someone who genuinely prefers not to be in Google's index.

Nope, it looks like we haven't been able to fetch even the robots.txt file for a few days, so our system is trying to play it safe by assuming that HN is unreachable/dead.

Excellent contribution... Ugh, this is why I don't mind if HN is listed on Google or not.

So which part is laughable? The part where they're able to crawl the vast expanses of the web and return relevant results for the majority of their users? Or is it the part where they came out of nowhere to dominate search because they did it better than the rest?

Come on now, you can't be all things to all people. Google is far from perfect but for a lot of us it's much closer to perfect than the competition and they're constantly trying to improve it. Why don't you go ask Matt Cutts to fix whichever parts of it you think are laughable to your liking? He's been hanging around here and he doesn't seem shy about answering people's questions and concerns. I do doubt he'd give the time of day to a one sentence remark that adds nothing of value whatsoever to the larger discussion or any of it's offshoots.

The other page isn't dead, they can access it.

yes - it looks more like google had been changing their algorithm recently - in favor of older sites , versus the content -

and it didnt turn out quite well ...

Anyone know if this is also a common issue with BingBot? Our site is doing fine in Google at the moment, but was completely deindexed by Bing, and we're totally baffled.

> I am glad Google is a company made up of humans and not uncaring robots

That's exactly the problem. Google is notorious for providing horrible customer support, so why some people get personalized help, while the rest of us are stuck with the uncaring robots?

What do you think should happen in this scenario? Matt Cutts is aware that a reasonably high-profile site is not being indexed properly. Should he just ignore that?

I think it's particularly good that this is being done publicly, too. There's billions of sites out there and Google can't provide this for all of them, but since they're doing it publicly others can learn from it and know how to address it.

If Matt was artificially boosting HN's ranking that would be disturbing, but they're fixing an indexing issue.

Are you kidding? Matt is basically a superhero who helps anyone he can find. He's solved problems for people complaining on Twitter, on Google+, he hosts regular video office hours when anyone can come to him with problems, etc. It may be the first time YOU'VE seen him rush in to help someone, but this is Matt Cutts' standard MO.

Google should simply deliver the best results, no matter what algorithm they use to do so.

Umm...Yes.

Seems like "bending over" to me:

"but I'm pinging the right people to ask whether we can get this fixed pretty quickly"

Let's be real here. Do 99.9999...% of users need or deserve this kind of support? It's easy to have sour grapes and say "oh the big shots get attention but I don't! Not fair!". But then you have to remember that people like PG worked their ass off to get a site like this to become so valuable and popular. We're not talking about some guy's blog about his cat that no one cares about, we're talking about HN here. Is it equal treatment? Of course not but is it fair? I say it is fair.

If fairness is treating people based on merit then it's totally fine. Matt isn't in here saying he's going to manually manipulate the result or change anything on Google's end (save for maybe correcting a flaw that would benefit everyone who's in a similar situation as HN). All he's doing is suggesting possible causes, trying to make a diagnosis, and just troubleshooting.

Most people who want this attention don't deserve it. They're the type of people who won't use the resources available like the google help documents or even learning how to administrate a site for best results in search engines. Plus, most people's website just aren't worthy of this attention.

I could go on but lets look at the reality here. We're all knowledgeable of how the web works here so let's just admit that HN is a damn popular site and it definitely seems like a fluke to have it rank as it was when this was submitted.

~30k employees at google .. what makes you think they hold all the keys to the millions of web developers across the globe?

Unsure. On several occasions I got the distinct impression that a lot of members aren't actually interested in "hacking" (for any of its definitions), instead you get people asking about the value of jailbreaking a phone: http://news.ycombinator.com/item?id=3169000

I suppose it is because the top hits below HN are perhaps even less about "hacker news" ?

Normally there's a lag between when a site goes dead and when we crawl the site to see that it's dead.

It's also tricky because you don't want a single transient glitch to cause a site to be removed from Google, so normally our systems try to give a little wiggle room in case individual sites are just under unusual load or the site is down only temporarily.

All these errors are reported in Webmaster Tools in the Diagnostics section. Verify your site and you'll have access to this and a wealth of more data.

Also, we do send notifications in Webmaster Tools, and you can confgure those to be delivered by email too. I'm not sure if we send messages for these kinds of serious crawl errors, so I'll need to check. If not, that's an interesting idea I can ask the team to think about.

Thanks for the feedback :)

Pierre

Here's a search that tripped me up -

I'm in Houston, TX. When I searched for 'windshield repair houston', the 1st result - wwwDOThoustonwindshieldrepairDOTnet - looked promising so I made an appointment with them to get my windshield repaired. When I went to their place of business, it was just a guy in a pickup truck in the parking lot of a strip mall, with a 'windshield repair' sign on the back of the truck.

Turns out he's running a scam where he gets ppl to file claims with their insurance, and when they pay him, he would kickback 50% to the customer. Having insurance pay for damages is common enough, plenty of businesses do it. But this guy was trying to get me to file claims for damages that I didn't even have. When I asked for a cash price just to repair the damage I did have, he refused saying that 'it wasn't enough money'.

I was pissed. Not only is this illegal, I couldn't believe he was ranked #1. When I did some digging, it turns out the guy is gaming google with a ton of paid backlinks. For example, http://www.searchpicks.com/business/automotive/patsco-windsh... (click 'suggest listing' for the price).

I'm sure plenty of other searchers ended up wasting their time with this SERP just like I did.

That's helpful, thanks. I haven't kept a record of those searches (they were mostly programming-related queries), but I'd be happy to submit reports in the future. Is there a URL or email address for this purpose?

This is awesome. Matt friggin Cutts is over here personally responding to not just the big shots (as in PG, and I don't mean that in a bad way at all) but some of us little people too. I see this stuff and it just inspires me. I really hope one day I can be in a position where all types of people, rich, poor, powerful, powerless, can all benefit from my skill or knowledge and that I would be the guy who helps the average Joe the same as the hot shot CEO. I hope I carry that with me as I build my own business. Thanks for being so cool, Matt Cutts.

I disagree. I mean, we don't want to keep it a secret but at the same time this site has always been really close knit and the people all really just "get it". If HN grows too large I fear for it. The web is full of Cretans and people who think they're thoughts are so worthy they must be heard. Lots of ignorance out there. I've seen some it here already. I learned the culture and studied the etiquette and what qualifies as a useful submission or comment before I really joined in. Not everyone is so thoughtful and it could really sour a once great community. You're obviously new around here so I'm not surprised you'd say that. I mean no disrespect - I do think you'll eventually change your mind though.