Web search on the Anthropic API (opens in new tab)

I think the most interesting thing to me is they have multi-hop search & query refinement built in based on prior context/searches. I'm curious how well this works.

I've built a lot of LLM applications with web browsing in it. Allow/block lists are easy to implement with most web search APIs, but multi-hop gets really hairy (and expensive) to do well because it usually requires context from the URLs themselves.

The thing I'm still not seeing here that makes LLM web browsing particularly difficult is the mismatch between search result relevance vs LLM relevance. Getting a diverse list of links is great when searching Google because there is less context per query, but what I really need from an out-of-the-box LLM web browsing API is reranking based on the richer context provided by a message thread/prompt.

For example, writing an article about the side effects of Accutane should err on the side of pulling in research articles first for higher quality information and not blog posts.

It's possible to do this reranking decently well with LLMs (I do it in my "agents" that I've written), but I haven't seen this highlighted from anyone thus far, including in this announcement.

Bing is shutting down API access as well and going to "Grounding with Bing Search" at $35/1k https://azure.microsoft.com/en-us/updates?id=492574

The web search functionality is also available in the backend Workbench (click the wrench Tools icon) https://console.anthropic.com/workbench/

The API request notably includes the exact text it cites from its sources (https://docs.anthropic.com/en/docs/build-with-claude/tool-us...), which is nifty.

Cost-wise it's interesting. $10/1000 queries is much cheaper for heavy use than Google's Gemini (1500 free per day then $35/1000) when you'd expect Google to be the cheaper option. https://ai.google.dev/gemini-api/docs/grounding

Is search really that costly to run? $10/1000 searches seems really pricey. I'm wondering if these costs will come down in a few years.

Good that it has an “allowed domain” list, makes it really useable. The OpenAI Responses api web search doesn’t let you limit domains currently so can’t make good use of it for client stuff.

Related: For those who want to build their own AI search for free and connect it to any model they want, I created a browser MCP that interfaces with major public search engines [0], a SERP MCP if you want, with support for multiple pages of results.

The rate limits of the upstream engines are fine for personal use, and the benefit is it uses the same browser you do, so results are customized to your search habits out-of-the-box (or you could use a blank browser profile).

0: https://herd.garden/trails/@omneity/serp

It would be nice if the search provider could be configured. I would like to use this with Kagi.

I couldn't see anything in the documentation about whether or not it's allowed to permanently store the results coming back from search.

Presumably this is using Brave under the hood, same as Claude's search feature via the Anthropic apps?

If you use your own search tool, you would have to pay for input tokens again every time the model decides to search. This would be a big discount if they only charging once for all output as output tokens but seems unclear from the blog post

Can any one answer this question: are they using custom home made web index? Or are they using bing/google api?

Also I'm quite sure that they don't use vector embeddings for web search, its purely on text space. I think the same holds for all LLM web search tools. They all seem to work well -- maybe we don't need embeddings for RAG and grepping works well enough?

I'm also interested to know if there are other limitations with this. Gemini, for example, has a built-in web search tool, but it can't be used in combination with other tools, which is a little annoying. o3/o4-mini can't use the search tool at all over the API, which is even more annoying.

I Noticed a huge amount of latency using the web search on the api , is it me or with everyone ? Generally when u see other providers like open ai's web search it takes usually 7-8 secs and same for pplx sonar api. Any comments?

Setting the allowed url to "youtube.com" results in tool usage failing (up to max_calls times).

Does this mean there are certain sites that the search tool simply can't access?

Regarding the costs, do we have a clear indication as to how much it costs to the company to perform tasks from a power consumption perspective? Or is it negligible?

It's a good reminder that AI chats won't make web searches obsolete, just embed them at deeper in the stack.

Maybe Google search revenue moves from ads to more towards B2B deals for search API use.

Now all the big 3 LLM providers provide web search grounding in their APIs, but how do they compare in ranking quality of the retrieved web search results? Anyone run benchmarks here?

Clearly web search ranking is hard after decades of content spam that's been SEO optimized (and we get to look forward to increasing AI spam dominating the web in the future). The best LLM provider in the future could be the one with just the best web search ranking, just like what allowed Google to initially win in search.