Ask HN: Bingbot sending random search queries to your site?

2 pointsmischa_u1y ago1 comments

In my access logs I noticed bingbot sending a lot of random search queries to `/search=<query>`. Keywords are in various languages, include news, product names, celebrities, conversion rates and even xxx keywords. This has been going on for months, but really kicked off since January with over 100k requests per month.

The IP addresses from the webbots look genuine, all within the msnbot-*.search.msn.com range. The user agent is "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36", so this doesn't look like LLM scraping.

As far as I can tell there are no links to these pages, and most queries won't return anything as my website is rather niche (CAIBotList). Has anyone else noticed their website being used by Bing as a proxy search engine?

1 comments

Festro1y ago

It's not a proxy search engine thing.

Bing (and Google) serve internal site search pages (your search function) in their search results if they're able to crawl your search function (it's easy to find, and as longas it's indexable).

What they don't really know is what terms people tend to search for on your site, as that's only known to you. So, they'll guess. And they'll guess terribly and in random languages.

I think Google are okay at making informed guesses now, but Bing as usual lags behind.

I wouldn't worry, this just means that your search results might appear in Bing's index and you may get traffic for terms your site hasn't been ranking for previously.

j / k navigate · click thread line to collapse