Ask HN: Has the quality of Google search declined for anyone else, or just me?

This topic is posted about once a month for the last 13 years: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

I suspect, but can’t prove that the entire internet is rotting. I get the feeling that organic user contributions are plummeting and being swamped by bots. HN is a small life raft. Many of the niche subreddits I’ve been following for years are seeing their post frequency drop, from dozens a days to a few a week. All the other forums are gone. Maybe there’s something going on in the apps, but they aren’t searchable. The internet is becoming a cold dead place. There’s not much for Google to find these days.

Case in point: I was just trying to find some information about TGV rail routes. Standard Google searched yielded nothing but spam. Appending “Reddit” game me posts more than 5 years old with out of date information.

There’s been some debate about this here on HN and someone made a point that resonated with me:

The quality of _the web_ has been in decline lately.

With ML’s capacity to paraphrase original content and to generate plausible rubbish content from scratch, it’s very difficult for Google’s pagerank (or whatever they call their algo these days) to fight back.

That been said, there does seem to be a fair bit of scraping and paste going on. I’m surprised G is not looking at published dates and lowering the the scammers ranking.

Google search has been in decline for at least the past decade, and been surpassed easily for at least the past 5 years. Try running a local instance of a meta-search engine like https://github.com/searx/searx - you'll never want to go back.

It's not just you. Google is now what Altavista and the other search engines it replaced became. The difference is that those went to crap in only a couple years where Google held out for decades and then suddenly has spiraled to where it is now. And it's the same reasons those search engines in the 90s went down the tubes. Everyone who wasn't there will tell you Google's superior technology (PageRank) is why it won. But that's not really why. It's because AltaVista, Lycos, etc. got awful to use, which cause us to look for an alternative. There was this nice clean upstart that said "Don't be evil." is their motto.

It's time for the cycle to begin again.

It's great at showing explicit ads or "content marketing" ads-that-land-in-it-without-paying-Google-directly.

On one hand you have Google, trying to keep patching an algorithm while also maintaining an ad revenue stream.

On the other hand you have the whole of humanity's ingenuity trying to make money for themselves.

Why do we think one system is going to be able to defeat the whole universe of creative hacks to try to beat that system at any reasonable rate?

(It's a wonder that it's even still adequate... but as long as the main competitors are using the same machine-and-math-driven approach, just with fewer resources to throw at it, it's unlikely that there will be a ubiquitous replacement making a major improvement any time soon.)

With all the Googlers here and this question being asked again and again including by myself I wonder if this feedback has already reached the right folks at google but there just isn't much that could be done given the business incentives at play.

I think a small variety of issues...

1. Search has to be hard because filtering spam is a cat and mouse game inherently.

2. Useful information has moved off of searchable sources. Instead of forums, we have discord, facebook groups, slack channels, twitter...

3. There is simply more shit, exponentially, than when I was young, and therefore there's more garbage lying around. The ratio of shit:not shit hasn't gotten better.

Yes, it's getting worse, but somehow no one can beat their very low bar. I always start with Brave or DDG, and a high percentage of the time I need to resort to `!g`.

Recently, I couldn't remember the domain name for the web game, https://plurality.fun. Searching "plurality game" in DDG, Brave, and even Bing, the results are quite bad¹. However, the site I wanted was the #1 result on Google.

I've been using alternative search engines like DDG since ~2015, and I'm quite disappointed at the lack of innovation so far. It feels like a $20 bill on the sidewalk, so I assume I'm missing some reason why they all suck, but for the life of me I don't have an explanation.

¹To be fair, the site I wanted is pretty crappy, and not necessarily what everyone wants with those search terms. Nevertheless, it was #14 on Brave's search, where 3 of the top 5 results are links to the same academic paper, "When are plurality rule voting games dominance-solvable?" on three separate domains.

Impossible to use. I’ve started adding “ Reddit” to my searches but even now that’s not working as well as it did.

If you want high quality materials, you need a University library or a journal, whitepapers or an academic textbook. Or if you want practical information and experience, you need to be an employee or apprentice or business owner.

The web is the lowest common denominator. The high quality information is elsewhere and Google isn't allowed to reveal it due to rights holders, such as Google Scholar and Google Books.

I tend to spend my time on Infinity Family https://o2oo.li and https://halfbakery.com a community of inventors and creators. We have an ontology to solve the world's problems and work towards goals and projects as a community. We relax and share opinionated perspectives to problems in the world.

For one, like you said, there are so many spam websites out there that it's extremely difficult for Google (or any search engine for that matter) to gather genuine results. I also think that it's a matter of incentive. As long as traffic doesn't decline, does Google really have any incentive to fix or improve things?

Also, I presume the search algorithm is not how it used to be too, what with ML models being used left and right. I wonder if anyone really knows how we get the results anymore. There are a few initiatives like kagi but I know too few about them to say anything at this point. For now, I am quite content with appending "reddit" at the end of every non-technical query nowadays... at least until they start gaming this as well.

Two things I love about DuckDuckGo:

1) I can search for a handful of specific terms I remember were on a webpage I didn't bookmark, and I can type them in and find what I'm looking for instead of them being "interpreted" by something trying to be too smart

2) Nobody bothers to send them the same DMCA takedowns that they send to Google, so I can find torrents and things I couldn't otherwise

But on most days, Google is easier to find useful stuff when I don't know specifically what I'm looking for or the exact terms to use.

In an ideal world, I'd have Google default to more of how DuckDuckGo seems to work, with a "fuzzy search" option you can use if needed

It's absolutely declined, no question. Further, the advanced search tools have been all but eviscerated, so it's even more difficult to find what you want even if you know what you're looking for.

I don't think Google Search got worse, as in, I don't think changes they made directly made search a worse experience than not adding the change. I think that spam posters, scammers, et al., have gotten much better at gaming SEO, that Google got stuck with two bad options: (1) do nothing and have all common search terms loaded with spam, scams, and viruses, or (2) change how their search works to lessen the amount of bad actors, but also reduce the good search outcomes showing up.

Maybe I'm wrong. This is just a hunch based on looking at how big online communities fail, and how services that rely on user generated content fail. At a certain scale, you can no longer rely on people being good actors with light moderation. You need to tighten up moderation, even if it means making the community overall worse than it's peak. As, the alternative is letting the bad actors make it even worse than that.

More and more lately I've been coming across spam, reposted content from sites like StackOverflow, w3schools, Reddit, etc., but posted on different domains, on a page plastered with ads. To me, this is a peak into the arms race between Google's Search trying to return good results, and bad actors trying to get their spam to the front results.

I get amazing results while programming if I use DuckDuckGo.

I think google dorking make the suggestions better. Also, using bangs on duckduckgo has vastly improved my search results. If you know where to find stuff, you will get good results. For example if I am looking for review or recommendations, I use site:"reddit.com" .

Meanwhile I have the same question about DDG. Lately I've had to explicitly use google to get sensible results for direct by-name queries that ddg doesn't show even after 20 pages.

I'm talking innocuous stuff not political or porn or warez etc.

I know the domain name of a vintage computer wiki site. I search the name by name, I don't even need a search engine to find me the name, I did the main work for them already and showed up already knowing a domain name, and yet even so, all I get are 20 pages of links to mail list posts archived by narkive, and never the actual site. Google, of course, first link and most links on the first and all other pages of results.

It's bizarre and makes you wonder about all those unknown unknowns.

They want to make as much money as possible so they can go into space, or fund their extropian fantasies, and can afford to pay lots of unimaginative people like lawyers, marketing and bean-counting people lots of money to achieve it. Advertising is the crutch.

Ask a technical query and you will invariably get a bunch of sales offers for the product in question.

Just use more than one search engine. SearX mix results from multiple search engines. Kagi.com (now payed) has "lenses" features that do similar things.

Or just search on other search providers like you.com, qwant.com, startpage.com or DDG.

Isn’t it mostly that the structure of content on the web has changed? It's mostly centralized social media now, so that's where the most relevant results are going to be. Most of the attention is there, anyway.

1. You are the product, not the customer. The customer is advertisers. 2. What you see is what advertisers want, not what you want. 3. All ad-supported media converges on this no matter where it started.

In the last couple of months Google's results have improved for me, I'm no longer seeing stack overflow clones, there are fewer content farm results too.

I know that's the big money-maker for them, but it irritates me that almost the entire first page of results is adverts made to look like search results.

I agree - stopped using it more than a year go. certain topics it does well on but overall its become a terrible search engine.

No, it is not just you, the quality of Google results has gotten dramatically worse. BTW, you can ask Google 'is Google in decline' and see for yourself (DISCLAIMER: my answer on Quora is highly ranked where you can go into more details).

In short, Internet as we have known it has fallen apart because there are hardly any true organic links left. They used to be the lifeblood of the Web, the links based SOLELY on the relevance and quality of the linked page. Such a notion is quaint these days.

The Web used to grow exponentially in the early days, from say 1994-2008. Google launched with 100M webpages in their index, then they grew it to 1B in 2000 but they were NOT the first to reach that milestone, they were beaten to it by a Norwegian search company FAST that launched alltheweb.com with 1B before Google in 2000.

FAST was acquired by Yahoo in 2003 and you can guess what happened to them. For a few years Google and Yahoo were playing off against each other in terms of the size of their indexes, with Google always in the lead. But around 2003 the game of numbers stopped as they both announced they would stop publishing numbers. The Web continued to grow, still basically exponentially, and the next big milestone was announcement of Cuil in 2008. Cuil was a competing search engine created by a top team of Anna Patterson and Tom Costello, also with Luis Monier from Altavista. Their claim was that they would launch with an index of 120B pages, with an index bigger than Google.

That was widely considered an outrageous claim, as the notion that Google knows practically everything was already firmly entrenched. But they did manage to stir things up a bit, to the point of Google issuing a vague release in their official blog claiming they knew about 1 trillion urls. Of course they did not mention anything about indexing all that but the damage was done.

Shortly upon launch, the quality of Cuil results turned out to be far worse than expected which is really a shame as their basic premises were spot on, apart from index quality. Cuil then promptly fizzled out.

Note that projecting exponential growth (doubling every 18 mo i.e. quadrupling every three years) since 2008 we would expect 4^4*120B or more than 3 quadrillion(!) for the size of the Web index, with Google knowing 8 times that.

Such an expectation is plain silly, especially having simple queries such as e.g. 'Novak Djokovic', or 'Roger Federer' on Google returning less than 100 results.

But all this is only a (smaller) part of the story. Indexing is now a LEGACY technology, more than 20 years old. Users expect much more than returning bunch of blue links with matched keywords in response to their queries. They want much of the time direct answers to their questions.

The technology to do it has been known for 10 years now, in terms of dense vectors also known as word, sentence and other types of embeddings. Direct answers would be then found by nearest neighbor search. The scale of the system would of course have to be in the billions. BTW, it is a very interesting open question how many direct answers Google can provide now, in terms of infoboxes/featured snippets. Google has been coy about the issue but in my professional opinion, as a founder of multiple search engines, the answer is no more than around 20B. Feel free to shed more light on the subject and challenge this number.

In summary, the time has come to have a system based on vectors and nearest neighbor search with billions of vectors, giving direct answers to queries, with no ads nor tracking,and hopefully with API too.

One more DISCLAIMER, such systems are online for all to try and play with, at https://qaagi.com (for causal queries about causes and effects of things with billions of ranked answers) and https://yottaanswers.com (for factoid and and general questions what/how/where etc. with billions of answers). Both of the projects are led and principally funded by me, Borislav Agapiev.

The same thing has been happening to me, for the last few months the average search results have been of very low quality.

It's a fine balance between accuracy and rewarding your highest ad spenders.

No, Google search remains awesome for me. I’d switch if there was something that gave better results, but that doesn’t exist for me. I gave DDG a fair shake (about 3 months) and it was horrid.

I’m also not concerned with privacy/whatever so I won’t switch to something equally as good or worse to make a statement.

If I search for "walmart 24 hours", I get a list of closed bakeries.

Is your personalisation for google search turned on?

What can be done about it?

I don't think it has declined

It serve its users, you are probably in the niche category that's declining, or rather the casual group is growing in number and as a result your category is being diluted even more, tech stuff usually are old stuff, users prefer new and up to date topics

If you are looking for something specific, i suggest trying few tricks to get better specific results: https://betterprogramming.pub/11-tricks-to-master-the-art-of...

My vision for the future of search:

Text search on the web will slowly die. No one trusts the results of random text. Google is in the adversarial position of wanting to censor certain answers as well as present answers that maximize their own revenue.

People will search video based content instead, and use the fact that a human spoke the information, as well as comments/upvotes to vet it as trustworthy material (like on TikTok).

Google search as we know it will slowly die, and then will decline like Facebook. TikTok will steal search marketshare as their video clips span all of human life.