Ask HN: Is there a search engine which excludes the world's biggest websites?

578 pointscJ0th5y ago228 comments

Discovering unknown paths of the web seems almost impossible with google et al..

Are there any earch engines which exclude or at least penalize results from, say, top 500 websites?

228 comments

noad5y ago

This is a great question, I also want a way to search the internet but exclude all major media domains as well as any company over a certain size. So I just want to search through old blogs, SO, non-corporate social media, weird forums, etc.

There are so many cool things I remember reading on the web like 10-20 years ago that still exist that are so buried now on Google they might as well not exist. Nowadays searching any topic seems to always lead you to CNN and Microsoft and Facebook and other huge corporations. Search results are just becoming more sanitized and beige and meaningless every day.

dorkwood5y ago

For years, my trick for finding interesting content was to go to, say, the 7th page of Google results, and start there. This doesn't work anymore -- it's SEO-optimised listicle blog posts all the way down.

My trick now is to use Twitter to discover interesting people, and follow them there. Granted, it's not a search engine, but it's at least given me the ability to discover weird things again.

gbin5y ago

I agree on the original issue but social media tailored to people's own bubble is probably not a good source for enlightenment either...

1 more reply

tomaskafka5y ago

Many times I have thought about what could happen if Twitter asked you to recommend up to 3-5 people you value, and write a tweet-sized (or shorted) recommendation.

Over time you would get a 'pagerank for people' and could do awesome stuff with that, like 'You don't know XYZ, but 3 people you trust trust her, and this is what they tell about her:' ...

1 more reply

edeion5y ago

I was about to ask about dmoz.org. But apparently it's dead. We could probably do something with bookmark sharing à la Delicious. Good dead things for a better future.

Scoundreller5y ago

Heh, I was trying to do research on coronaviruses (of which COVID-19 is one of many coronaviruses), but Google sanitized the result and only showed me "official" COVID-19 resources and buried the broader coronavirus resources.

https://www.google.com/search?q=coronavirus

stolenmerch5y ago

COVID-19 isn't a coronavirus, it's the disease caused by the SARS-CoV-2 virus.

4 more replies

rczhang5y ago

You can just search for 'Coronaviridae' instead. The overwhelming majority of people searching for coronavirus want pandemic news.

pwdisswordfish25y ago

"Is there a search engine which excludes the world's biggest websites?"

There was "rebranded" web search that someone created a number of years ago and posted on HN that aimed to exclude the top websites from results. I cannot remember the name he gave to the project.

One way to exclude the world's biggest websites when using Google is to restrict the search to TLDs other than .com, .net and .org. The root zone is full of silly new TLDs that no one uses for large websites. There are hundreds to choose from.

https://www.google.com/search?q=coronavirus+site:edu

Looks like Google Scholar is including a number of "coronavirus links" on the main page but thankfully not in the results.

https://scholar.google.com

https://scholar.google.com/search?q=coronavirus

Why not skip Google and "web search" and use a database that does not include all the crap one finds on the www

Something like

https://pubmed.ncbi.nlm.nih.gov?term=coronavirus

https://search.crossref.org/?q=coronavirus

ikeyany5y ago

In that case, I would omit webpages from the past year.

drran5y ago

Try http://biomed-sanity.com/ .

nojito5y ago

Why would Google showcase non official sources?

40four5y ago

It is kind of funny that we talk about SARS-COV-2 as if it is the only coronavirus. Coronavirus, singular. If I’m not mistaken the common cold is in the corona virus family.

1 more reply

chris_f5y ago

If you are looking for a forum search, check out https://boardreader.com/.

I have a theory that web crawling alone is not the best way forward to find the most relevant results because of the volume of content continually being created, much of which is niche and sometimes dynamic.

Instead I believe linking together vertical search sources that have targeted information based on search intent will provide better results.

I created Runnaroo [0] for that purpose. If you search a programing question, it will pull traditional organic results from Google, but it will also directly query Stack Overflow for a deeper search.

[0] https://www.runnaroo.com

scottlocklin5y ago

This is really good!

https://www.runnaroo.com/search?term=J+apply+verb

versus google's completely useless:

https://www.google.com/search?safe=active&q=J+apply+verb&oq=...

Everyone else, get in here: this is top notch stuff.

lazyjones5y ago

This is somewhat ironic because 20 years ago, hobbyists would frequently put their obscure personal pages on Geocities and other large corporation's web space.

wolco5y ago

Memory can be foggy but the most useful were hosted on university pages or random folders off a random domains or you get a subdomain. I picked the username 'search' which gave me search.batcave.net which worked great until one day they just took over the subdomain for a site wide search. They were confused when I complained.

Sure people hosted on geocities and tripod and they were the biggest and easiest to remember. But quality of a geocities page compared to a mit student page was much lower.

CalRobert5y ago

20 years ago, it was extremely common for your ISP to give you 5MB or whatever of space to use. users.ispname.com/~yourname or whatever. It was great, tbh, since anyone with cuteftp and notepad could publish to the world.

1 more reply

fedede5y ago

Hey! I actually liked this idea and I'm considering starting a learning project on it. I've seen a lot of interest and ideas in the comments, and decided to create a very short Google form to start gathering all the interested people so we can organise something interesing. Is anybody in? :)

https://forms.gle/5KuTYVdYaMzRD2n78

ryandrake5y ago

If there was a way to simply exclude from searches: shopping, news, images, videos, and “listicles”, I think that would get us most of the way.

Especially shopping. The endless stores are the worst part of search results. If I search for anything that remotely looks like a product, the results are just choked with store after store trying to sell me the thing. Awful.

type05y ago

If you do your own search engine like that, implementing rules that block affiliated commerce blogs from appearing would help alot.

sanqui5y ago

There is a search engine with this exact goal: https://millionshort.com/.

I haven't had that great results with it myself though.

smackay5y ago

I tried it with "waders" which are either the things that you to put on your feet to go fishing or a category of birds (shorebirds or herons). The results after going for all the options were still exclusively stores wanting to sell me the former.

Garbage in, garbage out. I guess. Still I like the idea of something to side-step the SEO perhaps with more effort they can make it work but relying on Google or any major search engine for the base results is the wrong way to go.

glenstein5y ago

I tested your search term and had similar experience. I have, however, had positive experience with other categories, such as philosophy. Searching Wittgenstein with the top million sites removed, I found some gems: a play, a disney character that was a supercomputer named after Wittgenstein on a direct to video movie which I learned was later partly inspiration for Wall-E, Wittgenstein-oriented societies, awards, and general philosophy references I had never heard of.

I suppose it depends on the category.

puranjay5y ago

I've found that you get far better results for many queries on YouTube now since it's tougher to mass spam YouTube with low quality content (compared to churning out a $5 article).

Nevada-Smith5y ago

Google Scholar "waders" [1]

[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C29&q=wad...

behnamoh5y ago

Man it's super fast! I wish Google were like this.

hopesthoughts5y ago

Oh, I've had epic results with Millionshort!! I mean, as far as stripping most everything away, and just surfing.

erikbye5y ago

For google you can use this https://addons.mozilla.org/en-US/firefox/addon/g-search-filt..., just drop in your list of those 500 URLs, once you've decided on what the top 500 is.

For other engines you can use https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/ with this script https://greasyfork.org/en/scripts/1682-google-hit-hider-by-d...

atrudeau5y ago

For Chrome: https://chrome.google.com/webstore/detail/google-search-filt...

terrycody5y ago

really nice, thx for share!

thekyle5y ago

There is Million Short which allows you to search without the top 100, 1k, 10k, 100k, or 1m sites. Personally what I'd like to see is a search engine that only indexes webpages without ads since that should eliminate lots of the SEOd garbage. It would also be nice to use the text to code ratio to derank JS heavy sites.

https://millionshort.com/

b0ner_t0ner5y ago

> like to see is a search engine that only indexes webpages without ads since that should eliminate lots of the SEOd garbage

Not just ads, but also ranked by the number of third-party cookies/tracking scripts a site has.

joshspankit5y ago

And blacklist anything with those outbrain blocks. “One weird trick to _____.”

Very surprised where I see those these days, and they always make me run away.

dorkwood5y ago

The fact that news websites happily adopted those is disgusting. They're literally tricking people into thinking it's news content.

1 more reply

tlarkworthy5y ago

I made a script on ObservableHQ to surf YouTube psuedo-randomly https://observablehq.com/@tomlarkworthy/random-place-on-yout...

I do a random city + documentary as the search term, it's taken me all over the world and seen some very strange things.

One of my favourites was Aarhus, which had a Danish language rapper proclaiming he was putting Aarhus on the global map (I have never heard of the city of Aarhus). https://youtu.be/WSZxuzgImLo They dis Copenhagen a lot too, lol. You get a more intimate YouTube experience with the low view videos

But I also seen amazing religious rituals. An excellent documentary on Karachi.

Because it's observable hq you can fork it and figure out your own algorithm for biasing the random.

totemandtoken5y ago

Reminds me of this classic pg essay: http://www.paulgraham.com/ambitious.html

Specifically this quote: "The way to win here is to build the search engine all the hackers use. A search engine whose users consisted of the top 10,000 hackers and no one else would be in a very powerful position despite its small size, just as Google was when it was that search engine."

There has been a lot of grumblings about the state of search these days. Maybe the time is nigh for a new search engine?

tetris115y ago

I feel that we should go down the adblock hosts list approach, where people download website lists from individuals they trust who have curated or scraped links of websites complete with keywords, and its up to the user to refresh their lists and then perform a search on their website.txt file

It will be limited, but still quite powerful, similar to the way that we can pick and choose different host file sources from the web.

rthomas65y ago

Does anyone else remember StumbleUpon? It's not exactly the same as a search engine, but that worked really well for finding interesting content back in the day.

ublaze5y ago

I've wondered if this can be built with a limited budget, in terms of cloud costs. At least it feels like you'd need a lot of hosts to satisfactorily index the web, and store the results so that you can return results instantaneously.

joshspankit5y ago

With limited budget and the right results, I think people would pay with “non-instant” results. Even 5-10 seconds might be perfectly fine, and that’s “easy” on a limited budget.

_xnmw5y ago

DEVONagent is a highly configurable search utility which can be used to combine and de-duplicate results from multiple search engines at once, exclude sites or keywords from a blacklist, follow deep links within search pages, and perform some filtering logic on the text of results.

Before I knew about DEVONagent I would often just search multiple engines and sources trying to find something particular (e.g. a particular PDF) or unique results.

https://www.devontechnologies.com/apps/devonagent

lemonberry5y ago

Thank you for the link. This looks really cool. I used DEVONthink years ago. It seemed like a great piece of software but I didn't have a great use case for it. Looking forward to checking out DEVONagent.

petra5y ago

It looks really interesting. Sadly, it's Mac only.

Does anybody knows of something similar for Windows or Linux ?

brightball5y ago

This does look really interesting. Thank you.

pavelmark5y ago

Simply removing Pinterest would be a huge step in the right direction.

joshuaissac5y ago

I use an add-on called Unpinterested! to remove Pinterest results from my Google search results:

https://github.com/sellomkantjwa/unpinterested

All it does is add -site:pinterest.com to the search bar for image results (can be configured to also do it for Web results), but it gets the job done.

pier255y ago

And quora

stock_toaster5y ago

ye gods, yes.

user94294505y ago

Any insight with regard to how Pinterest was able to do what they've done? Did they simply dump more money into it?

chaos_a5y ago

https://wiby.me/ exists to solve this exact problem. I've found some pretty neat/odd websites on it in the past.

lucb1e5y ago

The "About" information is, counter-intuitively, under "Settings":

> In the early days of the web, pages were made primarily by hobbyists, academics, and computer savvy people about subjects they were interested in. Later on, the web became saturated with commercial pages that overcrowded everything else. All the personalized websites are hidden among a pile of commercial pages. [...] The Wiby search engine is building a web of pages as it was in the earlier days of the internet.

btrettel5y ago

That makes me think: Is there a search engine which removes pages with any ads or affiliate links? That might be the easiest way to remove the commercial pages.

1 more reply

generalpass5y ago

It is a carefully curated directory, which is problematic.

For example, I submitted Pizza Hut's archived original web page [1], but it wasn't added.

Even for a search engine exposing niches, updating a directory manually will likely be too slow, unless the directory is maintaining a single nich (e.g., unladen airspeed of every species of swallow), but then we end up with some insane number of search engines and how to select which one?

[1] http://www.pizzahut.com/assets/pizzanet/home.html

kd5bjo5y ago

Especially if you’re focussing on evergreen information, there’s no reason why people can’t have their own personalized crawler and index— I’ve occasionally thought about rolling my own with a browser extension that lets me add seeds at the click of a button.

1 more reply

ehonda5y ago

I clicked your link, but I don't see an archive, its redirecting me to their main website.

Wiby is based around two main things:

Non commercial content (1) that does not rely heavily on excessive javascript and CSS (2).

http://wiby.me/submit contains the submission criteria.

bsanr25y ago

I tried this yesterday. It seems biased towards the interests of the curators, and, like Google in regards to "some ideal average consumer", is therefore useless if you fall outside a certain level of similarity to the target demo.

For example, I enjoy weightlifting and strength sports. I did a search for "muscle", and every result but one was using the word "muscle" as a figurative metaphor. Barely anything about actual muscles. Searching "funk" was just as bad. One page about Motown and a LOT of midis.

keenmaster5y ago

What if Google Advanced Search produced a visual network map which showed you the salient clusters of terms related to your search? You’d then be able to click on a cluster and the search results would change to adjust to what you’re really searching for.

Ex: The network map for “weightlifting” would include many clusters, but 2 big ones would be the hypertrophic cluster (surrounded by a bunch of related terms) and toning cluster (calisthenics would be under this cluster for example). Click on either and the results will change accordingly.

This would actually work even better for subjects you don’t know much about, because Google will teach you about the salient clusters in that field. The clusters could be enhanced with popular images associated with each term. Popular clusters would display as larger than others.

1 more reply

ehonda5y ago

Hi, just wanted to clear something up. Wiby is biased towards the interests of those who submit websites to it.

EmilioMartinez5y ago

Just added wiby.me/surprise to the bookmarks bar. Amusingly, the icon keeps changing every time I use it.

Aeolun5y ago

But to add my site I have to add every page individually. Nobody ain’t gonna use thst.

I’d have to submit every blog post?

ngold5y ago

Can't wait to try this search engine. Thank's for the link.

mikekchar5y ago

Here is an idea that I've always wanted to do, but will never have time for: A curated search engine.

Basically the idea is to have people band together and "recommend" links. You then do your normal spidering of the websites to create a search engine (or even just call through to a number of existing search engines). However, the ranking of the results is based on the weighting of the recommendations.

It's essentially a white list based on your own personal bubble. Of course this won't work in general because you will always get SEO creeps spamming recommendations. However, it gives you tools for working around those creeps. The average person probably won't be able to manage it, but power users probably will.

By not trying to solve the problem for everybody, it makes it easier to solve to problem for some people. Or at least that's my thesis :-) I might be wrong.

netsectoday5y ago

You can boot up your own custom search engine in a few minutes with YaCy (Ya See!) an open-source, P2P, Dockerized crawler and search engine built on top of Solr.

https://yacy.net/

If you're generous; you can make your index available to other P2P instances.

I wanted to run an API search the other week and was blown away with how quickly I could prop-up my own custom search portal (I didn't want to pay for API access to other search engines, and YaCy comes with a JSON and Solr endpoints).

I ran it locally to test my crawl filters, then pushed a private instance out to Digital Ocean to turn up the heat with the crawling. The only issue I had was the crawler would hit the max memory threshold on long crawls and the container would restart, but that was fixed by scaling up the box.

l725y ago

I have my own yacy search engines running internally (non-peered) for similar reasons. One crawls some key code documentation sites that I need for work, and another crawls a whole bunch of music blogs.

While I typically still use RSS for reading music blogs, I find having the search engine is a great way to go back and find something or discover something new! Every time I find a new blog, I just add it as an index to yacy to crawl.

I think it'd be great to see people spinning up larger instances that are highly specialized. For example, maybe a search engine that is dedicated solely to sci-fi and only crawls high quality boards, personal sites and blogs, and skips all the spammy, seo-optimized sites.

crawlcrawler5y ago

I built a search engine for this and other, similar purposes. With Crawl Crawler you start out by searching the meta data of a Common Crawl ("CC") crawl. Then you define a sub section of that data collection by designing a query which search result includes your favorite sites. Then you enrich that sub section by linking those meta data documents (that come from CC's WAT repo) to full text extracts or HTML from CC's WET repo or the WWW. Then you set it to recurringly refresh that section. Voila! You have created a search index that includes your preferred sites. https://crawlcrawler.com

chris_f5y ago

This is pretty cool. I always wondered why there wasn't a user interface search somewhere for the CommonCrawl data.

allwynpfr5y ago

You should try million short. As the name suggests, it takes our the first 100 / 1k / or a million results so you're left with those that aren't all that popular. That seems to be what you're looking for. https://millionshort.com/

nic-waller5y ago

My hobby project is https://random.surf (works better on desktop than mobile).

I share that same desire to visit the web less travelled. I want to discover interesting sites that deserve to be bookmarked because they will never show up in a search engine.

77ko5y ago

Love it! Discovered something interesting very quickly. Bookmarked for future use.

hopesthoughts5y ago

That's definitely spinning the web, which is a whole lot of fun.

severine5y ago

Thank you!

True "Interdimensional Cable" vibes.

tetris115y ago

This is excellent

dangoljames5y ago

There used to be java applet embedded in altavista.com's website that could be run against search results. It would do semantic processing on the results and present a list of generated terms, each with a checkbox. Checking a box would pull any returns which contained the topic from the remaining search results.

This was fire. If a topic were being discussed on the web, you could find it with this tool. Unfortunately, it did not fit the vision of the parasitic overlords who bred us to produce and consume for their benefit.

visarga5y ago

Altavista itself was a junk search engine though, especially after they sold out and the new owner stuffed it with ads.

dennisy5y ago

I think you could get good results if you just penalise sites for the number of third party JS. Which shows by proxy a more established site/corp.

You could add a bunch of heuristics such as size, number of links etc.

Maybe even train a classifier to select the “smaller” part of the web.

inopinatus5y ago

I would pay real subscription money for a search engine that focused on knowledge-oriented results rather than retail and commercial results.

When I type “shoes”, it would give me: links for the functional and creative history of footwear, the taxonomy of shoes, methods of construction, current and historical footwear industry data, synonyms and antonyms, related terms and professions, the dictionary definition, and similar links related to secondary meanings (such as any protective covering at the base of an object, horseshoes etc). I’d also hope for a comedy link to a biography of Cordwainer Smith.

What I actually get, which I don’t want at all: pages and pages of shoe shopping.

The various means to exclude “top X sites” are the roughest possible heuristic in that direction, and throw out the baby with the bathwater (for example, a long-established manufacturer may well have an informational online exhibit)

Google has essentially failed me in its primary mission. Bing at least has the grace to admit they are here to “connect you to brands”. And sadly, right now, every other option is an also-ran.

In practice I use DDG, directed by !bangs towards known encyclopaedic or domain-specific sources. I am certain that I’m missing out.

seektable5y ago

What you describe sounds like a mix of personal knowledge base that seats on the top of existing search engines, public and maybe even private databases. Major difference is here:

* when you make a query to this knowledge base, it has a history of your prev searches / preferences (not google)

* it can propose variants of suggestions on what is your intent in this particular query - and make much more detailed queries (auto include/exclude keywords websites etc) to multiple sources (not only google, maybe anonymously)

* it can parse results from these sources and re-arrange them (use own rank system) according to the your preferences. In this system, you can explicitly say - I hate that, and I like that, and this will affect the behavior. Yes this is 'information bubble' but it is controlled by you and not by google!

* finally, this system may work in background and handle 'research' search queries. What I mean here: currently, Google is about instant search - it gives you results in milliseconds, and that's all. It cannot spend much computations for more precise, more intellectual check of content in links from the search results - it cannot do reasoning - and you have to do that by yourself: open links from 1-st page, and close most of them immediately b/c they are not relevant for you, go to 2-nd page and so on. It would be cool if most of this could be automated - with modern natural language processing approaches and old-school prolog-like reasoning this is real and not a fantastic from sci-fi.

My vision that this kind of search assistant cannot be SaaS / closed source. It is about the freedom - and thus this should be open source / self-hosted app that can be deployed on PC or on cloud VM - but hosting should be controlled by end-users, not companies.

I don't know if something like this ever exist. If not, maybe its time to create it.

atlantique5y ago

That sounds like the job of an encyclopedia. Maybe some sort of collaborative encyclopedia where people can edit pages and add references.

inopinatus5y ago

I know you’re only half joking. In practice many, perhaps most of my DDG queries do end in !w - but there’s a wealth of information that is relevant, interesting, and useful, but wouldn’t be considered encyclopaedic, or that is merely summarised in Wikipedia; in addition, their references are included as supporting citations, very far from a comprehensive index of currently accessible information.

text_exch5y ago

I've long wanted to build a search engine of only personal blogs. I am less familiar with the field of information retrieval so I haven't gotten started yet, but it's always been a dream of mine and if anyone is interested please contact me at threemillionthflower [at] the world's largest email provider.

Discovering unknown parts and blogs on the internet is one of the enduring goals of a newsletter that I run [1], which provides a single link to an interesting article every day, usually by lesser-known authors and blogs across the internet.

[1] www.thinking-about-things.com

hopesthoughts5y ago

I'm now a subscriber.

011-video5y ago

You are your best search engine !

On a daily basis your brain use shortcut to get to the point. Open Firefox (of course) ALT+B. Then add a new bookmark for instance :

Name : Stack Overflow

Location : https://stackoverflow.com/search?q=%s

Tags :

Keyword : st

Now if you want to search "javascript timer", just type : st javascript timer

Add "%s" to all your favorites website search url.

Example : https://en.wikipedia.org/wiki/%s

To discover some new website content, apply the same trick to Hacker news, Reddit or any RSS River.

Voila, bye bye GG.

NateEag5y ago

For Google, you can ignore specific sites by adding "-example.com".

See this example of filtering Stack Overflow out of search results:

https://www.google.com/search?q=loop+over+array+items+in+jav...

1f60c5y ago

More specifically, -site:example.com, although there have been reports of Google breaking this time-honored functionality.

brentis5y ago

Imagine if sort results had table filters and sort.

Popularity, Relevance, Age, Type, etc. type could be blog, forum, site, or video. Or like it used to be.

busymom05y ago

I have been finding recently that Google has been breaking their existing "sort", "time" filters. Try searching for something with site:reddit.com prefix for example and set the time filter to be lets say "Last year". Google still shows you results from 4-5 years ago.

BostonFern5y ago

I also discovered that recently. It's gone the way of the verbatim constraint.

Control is being forfeited to steer users back to more profitable content in order to capitalize on a captive market.

I wonder if being open about it would be so bad for business, instead of the attempt to manipulate users into enjoying the ratcheting-up of their impotence.

Now, Youtube truncates search results and loads the recommendation stream instead, long before hits are exhausted.

At least it's been a while since Silicon Valley was keeping the mythical personalized advertising spiel in active circulation.

sneeuwpopsneeuw5y ago

I personally use Google Chrome with the duckduckgo search engine. Duckduckgo is not perfect, very in depth searches (such as gameboy advance memory layout only return junk, while google knows you are searching for a nich) but on your average search it is as good as google, somethimes better because it is more factual and will promote less webstores. When it does not give me what im looking for I can add !g anywhere in the question and the same search is done using google.

Then I use Violentmonkey an open source js/css injector to inject this user script: https://greasyfork.org/nl/scripts/1682-google-hit-hider-by-d... This will block specific domains for you in google, yahoo, duckduckgo etc. I use this to block domains like Quora, sourceforge, cnet and softonic.

The nice thing about this script is that you can permaban domain you know are junk and they will completely be removed or you can ban a domain like commercial websites. When you ban something it is not removed from google or duckduckgo but it only shows the title in light gray, Im currently experimenting with this on some mayor webstores so I can not really say if this may help you but It can be a good start.

(edit) I saw some people say why this was not possible before. Google allowed you to block domains and website a few years ago, but they removed this feature. Duckduckgo never allowed you to do that because that would mean that you will have a cookie that remembers your preferences and that is against there principles.

1f60c5y ago

> I can add !g anywhere in the question

I knew about !bangs, but I didn't know you could put them anywhere in the query (e.g. "hello !g world" searches Google for "hello world"). This is going to save me a lot of time on mobile. Thanks!

greglindahl5y ago

If the question is "Is there a commercially-viable search engine that supports this feature", then the answer is "probably not".

Implementing this properly involves having your own search index. And that's pretty expensive.

bamboozled5y ago

I think on DDG you can do !mil which excludes the first million top ranking sites.

Edit: Maybe it’s the first million results? I use it to find obscure things sometimes.

mmsimanga5y ago

When researching a topic I have had great success searching HN and reading through the comments. If I want to find alternative software tools for a tool I am using the comments on HN are best. Searching through subreddits also yields better results than Google.

DavidPiper5y ago

It feels like there could be a (partial) meta solution here:

A search engine that returns results whose pages weigh in under a certain size.

From the comments it seems most of the "cruft" filling up Google results are newer web apps, generally JS-heavy and advertising-heavy, etc.

If you had a filter for pages with (e.g.) < ABC kb of JS, < XYZ external links (excluding img tags), I feel like there'd be a good chance that the "old" web and the "unknown" web would bubble to the top.

There are plenty of false positives (particularly for "small" forums build with modern JS apps, etc), but it could be one of many filtering tools to achieve better search results.

ngold5y ago

Great idea. Google seems to do nothing but remove search options. At least they still have a time filter. Ddg only does a year old I believe.

turnipla5y ago

Google used to let you blacklist websites many moons ago, that would go a long way already.

Now there are a few extensions that do that, but obviously they only hide the results from each page, so sometimes you will see pages with 2 results, if any at all.

rozab5y ago

Would be easy to just inject a negative site clause into the query, e.g. `-site:fandom.com`

chrisfrantz5y ago

Looks like google limits search query length to 2048 characters. That’s probably enough room to exclude a majority of the biggest names.

dublin5y ago

It would be nice if there were a way to make the exclusion list de the default for all your queries. For instance, I never want to see results from WikiHow again. Ever. Or the New York Times or any of the other paywalled sites...

1 more reply

methou5y ago

I used a Google Custom Search Engine (CSE) to remove results from Softonic and alikes, it works well, but still very Google.

petra5y ago

Google CSE is a great idea.Tried it in the past.

But i find the search is at a much lower quality than Google.

dexen5y ago

There is a similar problem where Youtube's recommendations and auto-play are mostly big name brands, to the exclusion of individual reporters, commentators, and other content producers. Since recently, a "De-Mainstream Youtube" plugin[1] is available for Firefox and Chrome, fixing that to some extent.

[1] https://demainstream.com/

bmd39915y ago

What I’d like to see is a search that excludes any page with ads, and any page with affiliate links. That alone would get rid of 90% of the garbage

chongli5y ago

I've expressed this wish on HN a few times in the past as well. One downside to it that I can think of (off the top of my head) is that there may be a lot of small, personal websites that are hosted on a free blogging platform which injects ads into every page. These websites may be very valuable (and not SEO garbage link farms) but would get blocked nevertheless.

dddddaviddddd5y ago

Sort of a server-side, page-level adblocker.

bmd39915y ago

It could make for an interesting Firefox extension I think

peel405y ago

I think there's a simple google way. Just add `-bigwebsite.com` to your query.

[search term] -google -youtube -facebook ... -top100website and it should work.

I found a list of the top 1m alexa websites here:

http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

An add-on with that list should do the work.

beachy5y ago

That's pretty clunky.

- there's probably a pretty low limit for size of Google queries, you'll likely hit it quickly

- you won't be able to search for e.g a story about YouTube censoring some content

peel405y ago

I don't know about the current query size limit but I think it's pretty likely to get hit quickly as you correctly pointed out. But, it's useful to use the wildcard "-site" ex: "-site:bigwebsite.com" for excluding just the site, and not the very word being mentioned.

ex:

facebook censorship -site:facebook.com

https://www.google.com/search?q=facebook+censorship+-site%3A...

Mandatum5y ago

Filtered out regional sites that Google wouldn't cater my search for (eg Taobao, Baidu, etc).. Only possible with 31 negative sites.

-site:google.com -site:youtube.com -site:facebook.com -site:jd.com -site:yahoo.com -site:wikipedia.org -site:amazon.com -site:netflix.com -site:reddit.com -site:live.com -site:zoom.us -site:okezone.com -site:alipay.com -site:instagram.com -site:twitch.tv -site:csdn.net -site:blogspot.com -site:microsoft.com -site:bing.com -site:github.com -site:tribunnews.com -site:myshopify.com -site:office.com -site:panda.tv -site:stackoverflow.com -site:ebay.com -site:bongacams.com -site:livejasmin.com -site:babytree.com -site:naver.com -site:apple.com <search query>

coronadisaster5y ago

your query would obviously be too long for Google

peel405y ago

yeah, that's sad. Google limits its queries to just 32 words.

bhartzer5y ago

There is a custom search engine called Newgle.xyz that only shows results from the 1000 or so new gTLDs (new top level domains).

It’s custom google search results, but since it’s excluding .com, .net, .org etc then you probably won’t see any of the large sites there.

It’s also interesting to see which sites have been built in the last few years, as the new gTLDS haven’t been around that long.

rkagerer5y ago

I would like one that punishes sites with too much ad to content ratio.

loosetypes5y ago

What are folks’ non-commoditized heuristics for finding new things online?

I was intrigued by how dorkweed’s approach has changed over time, as described in a reply to a sibling comment.

As general search results get watered down and rotten tomato inflation maybe trends towards reflecting company interests rather than my interest-level, maybe it’s worth re-evaluating the vetting avenues we take as users.

Here’s mine: for games and shows I’ve recently found myself using quantity of fan-videos on YouTube as a proxy for quality. So far it’s been a decent means to find cult followings for something I otherwise wouldn’t necessarily hear about.

Obviously this approach has its flaws - and is subject to financial perversions to an extent - but I figure if enough people genuinely want to pay tribute to a work, it might be worth checking out.

bluishgreen5y ago

How'd you find the quantity of fan videos?

Personal trick: I follow reaction video blogs, and if they are reacting to something then it is usually worth watching. But reaction blogs are only for short videos and other short form content.

ChrisMarshallNY5y ago

Remember sites like stumbleupon?

I find that the YouTube sidebar is useful for me to find interesting music. I have eclectic tastes, and Google seems to have figured that out. I don't mind.

I suspect that it would be possible to create a custom API query to Google that would have a "blacklist."

smsm425y ago

There's Million Short: https://millionshort.com/

I think they try to do exactly what you ask, but I haven't used them extensively so don't know how good are they.

abarrettwilsdon5y ago

For more queries, you can add modifiers to a Google Search to get the results you want

Seeing folks mention the NOT operator (-). It's quite powerful! For example, you can do:

intext:"Powered by intercom" -site:intercom.com will find all the sites that use the Intercom widget

or ~blog bread baking -inurl:checkout -intext:checkout will find bread blogs (or similar) without commercial intent

I put together a list of the two dozen or so most useful templates of this, for folks who are interested: https://www.alec.fyi/dorking-how-to-find-anything-on-the-int...

dhbradshaw5y ago

I've wondered too about something similar to that. Basically, I'd like sessions for searching.

Each session would have an updatable list of sites that are favored, whitelisted or blacklisted for a particular class of search.

maayank5y ago

I'm intrigued by actual use-cases for it except exploring, i.e. where it would give better result for a query than the common search engines.

Anyone reading this, please post if you find any

crocodiletears5y ago

Big ones:

1. Looking for niche domain or institutional/social knowledge produced by experts or insiders for an informed audience that isn't necessarily available in a scientific journal.

Especially with respect to the social sciences and literary analysis, there's a wealth of intelligent commentators that don't surface well on Google without very specific search terms, and the willful subtraction of domains like quora, medium, and tumblr.

They're usually contained on poorly maintained WordPress sites that the author has long-since forgotten about, or as invalid, handcoded html docs hidden in the personal subdomains of university professors and students.

2. Finding online communities that aren't a part of Reddit or a similarly prominent platform

chasd005y ago

in the same vein, it would be awesome to search for a product to buy with the results being ecomm websites owned by people in my area. A way to "shop local" online.

technotarek5y ago

ATTIC: A visual search and discovery engine to help you find the latest products from small, unique businesses near you.

https://attic.city/

Currently for three product tiers (furniture, home decor, and fashion/clothing) in 14 major US markets, where stores within ~100 miles or a ~2 hour drive are considered as part of the market.

Disclaimer: I'm one of the founders.

derision5y ago

How do you curate the stores?

1 more reply

fomine35y ago

I'm curious how English speakers (especially not in US?) finding national shopping websites because I live in Japan so never find foreign website written in Japanese

alibaba_x5y ago

Shopify released a mobile app recently for this exact purpose.

technotarek5y ago

...but only for stores using Shopify as their ecom platform. That misses A LOT of what's out there, probably 75% or more.

amelius5y ago

If only Google allowed us to omit websites from search results.

Google says they need our information to "improve our experience", but we can't tell them what to omit ...

rsoto5y ago

They used to allow that, it was very useful. But as with almost everything Google does, they killed it.

fedede5y ago

https://forms.gle/5KuTYVdYaMzRD2n78

jsgo5y ago

I don't know that I'd want a search engine to specifically exclude or limit the results of specific sites of their choosing (even if top 500 as the example is fairly unbiased), but I think I'd really like the ability to say "move these specific domains a few pages back. Don't eliminate them outright, but I have felt dumber having read them previously."

pengstrom5y ago

What I want is to filter out commercial results. When I'm searching for a product I don't want shills, I want real opinions.

third_I5y ago

And then came undisclosed sponsorship and that difference blurred more than ever...

21xhipster5y ago

https://cyber.page

Its kinda new so it excludes kinda everything :-) But you can make it work better :-)

https://ipfs.io/ipfs/QmQ1Vong13MDNxixDyUdjniqqEj8sjuNEBYMyhQ...

Aeolun5y ago

Can you explain to me what this it is (or is meant to be)? It doesn’t appear to have a search field, though the lightning effects are impressive.

freefriedrice5y ago

Why exclude the biggest websites?

The problem I see on DDG & Google is having to scroll 5-10 pages of utter SEO nonsense.

"Do you have a question about ____? Many ask about _______. ____ is a common question, here the are we some answer. [sic]".

Just utter garbage pages.

It used to be just with recipes or medical questions, but now it feels like most everything that is a general query.

dredmorbius5y ago

https://news.ycombinator.com/item?id=22792243

piusp5y ago

I have used copernik in the past this was a collection of search engines, listing more than 140 search engines. It combined the search results and sorted them by the key word % matched. It also had a lot of tools inbuilt for validating he links, coping the selection/ sorting and sharing the results. Simply amazing results.

wyck5y ago

Google search is so sad these days, all results are media conglomerates, it's completely counter to the core reason why the internet existed. I really hope by catering to these mega corps that they are completely undermining their brand and someone else comes along and pulls the rug out from underneath them.

If anyone noticed during the first couple days of covid, google search was free from large media results, the algorithm reverted back to how it was years ago and it was such a breath of fresh air. Of course they fixed the algo immediately, it went back to only showing curated media results..there was an anon google employee who posted why this occurred.

fermienrico5y ago

I think we are gonna see aversion from going public. Companies like Stripe and SpaceX are gonna stay private for a long time.

When SEC laws, shareholder interest, quarterly performance and stock volatility comes into play, corporations become this mindless soulless monster that will devour everything in its way and fuck consumers in every which way.

Democratization of funds from central authority to public creates disincentives and the shareholders don’t give a shit about many auxiliary things such as environmental concerns. Bottom line always matters.

It’s not just google but any public corporation. Can you imagine SpaceX being able to operate with the same passion with shareholder interests?

texasbigdata5y ago

It's not, I would argue, those things.

It is, potentially, the compensation plans. If you go to the proxy document and look at how comp plans are set, they usually hire a consultant, and "best practices" drivers are cash + big bonus based on typically some TSR (total shareholder return metric).

So for google, "don't be evil" is what's written down, but for the top execs "sell ads" is what gets they paid out before they retire. And those senior level "lifers" are what 40 now?

Don't really have proof to support these claims though.

1 more reply

koheripbal5y ago

I find the only way to get useful results from google is to use the "site:xyz.com" or "related:" options.

om425y ago

Do you have a link to that thread where an anon google employee posted on why it occurred? Interested in what happened since I didn't realize it did.

wyck5y ago

The reason given: Legit covid information was not getting through during the fist few days of the crisis, basically made an emergency move to disable the whitelist/blacklist functionality of the algo which reverted to how it worked years ago.

chrischen5y ago

It’s a hard problem because the lower down the pyramid you go the more content spam there is.

kian5y ago

Do you have a source for the claim in the last paragraph? Would be very interested to read it.

pkamb5y ago

I'd love a search engine that mainly searched Stack Exchange, (old.)Reddit, and some subset of blogs or single-author websites.

Especially removing Quora, Pinterest, and aggregation/reposting/SEO/affiliate blogs.

And all "product" images with a white background. Only show real photographs.

dredmorbius5y ago

Setting specific site restrictions (only one applies at a time, e.g., "site:example.com"), or utilising DDG bang search, gets close.

Cyclone_5y ago

Seems like a browser plugin might be a quick and dirty way of just filtering results to achieve the goal.

social_quotient5y ago

A mainstream search engines kinda like a big marker equity ETF or index? There are a ton of benefits but as a negative they make price discovery difficult and give monetary allocation to companies that probably shouldn’t have it.

Just a thought experiment, curious what others think.

wmnwmn5y ago

Maybe what we need is a return to the very beginning, namely human curated web catalogs, aka Yahoo

dluan5y ago

I mainly use google as a reddit search engine these days. "tiki cocktail pineapple juice reddit" gives me way more than google algorithm, and plus it's kind of like human powered SEO where genuinely useful links will likely have some discussion.

rdtwo5y ago

So I figured I’d try a few of these with “Seattle vegetable garden blog” as the keyword. Either there aren’t a lot of blogs on the topic or most search engines miss them because results are sparse and they really shouldn’t be.

ErikAugust5y ago

A curated, searchable web directory might be a concept that could come to be these days. It would share some of its DNA with the old school web directory but also share some with a search engine.

tokyokawasemi5y ago

I sometimes use "inurl:wordpress" when searching for travel info. This ensures more first-person blog accounts, rather than all the tripadvisor junk that's at the top.

known5y ago

https://twitter.com/search?q=twitter&src=typed_query

moreWeed5y ago

Man you read my mind, just starting thinking about this. From a search censorship perspective, the BBS's we were building in 93 would be better than what we have now.

Nevada-Smith5y ago

Depending on what you're looking for, try Google Scholar [1]

[1] https://scholar.google.com/

blondin5y ago

omg yes please.

can google allow us to exclude certain sites? i was surprised to see w3school showing up above official documentations for pandas and numpy. this is simply ridiculous!!

jotm5y ago

the "-" operator still works. E.g. "weird stuff I found interesting -reddit.com -youtube.com -wired.com -w3schools.com"

badrabbit5y ago

It wouldn't be hard to remove such results using a browser extension,but you will be scrolling a lot. Maybe duckduckgo should support it,feature request?

saadalem5y ago

Ok here is an additional idea for fun :

A search engine that shows only urls that are not indexed b google / another one that gives you the websites with lower pagerank

bmwracer5y ago

Not sure it would yield any useful results, but have been thinking about a no-index search engine for a while to help find obscure or otherwise hidden information. If one exists or you build one let me know.

hopesthoughts5y ago

Showing all the lower ranked sites would be pretty cool. A lot of blogs would probably end up there, especially if they don't care about pagerank like me.

jungletime5y ago

Is there an option to filter out news articles?

"If you don’t read the newspaper you are uninformed; if you do read the newspaper you are misinformed." Mark Twain

lihaciudaniel5y ago

That's why I watch TV instead.

dangoljames5y ago

Is there a search engine that actually combines selective search logic with reductive logic, so that can be used to actually search topically?

corndoge5y ago

Similarly, I have always wanted a YouTube search filter for "least views", since that content is invariably way higher quality

thoughtstheseus5y ago

I think one of the underlying problems in search is that most search engines are more like recommendation systems.

runawaybottle5y ago

Filter google against Alexa rankings?

egberts15y ago

What we all really need is a long-tail Bloom filter search engine on the search engines themselves.

coronadisaster5y ago

if google would switch to the google's engine code that was used right before they modified the "+" operator for google-plus, it would be a lot better.... ie: bring back the + operator please (the quotes dont work the same way)

citizenpaul5y ago

RSS used to be this. Google has done its best to kill it.

aiisjustanif5y ago

RIP StumbleUpon. The randomized search engine I want back.

starfallg5y ago

The elephant in the room is Baidu.

dpau5y ago

Ask HN: Is there a search engine which includes only what the Chinese government wants me to see?

starfallg5y ago

That was tongue-in-cheek, but I guess some people are legitimately offended enough to downvote.

jotm5y ago

What are they blocking anyway? Anything important?

There's so many Chinese forums for hardware/firmware hacking/mods, a shame translators are still very bad...

1 more reply

wojtczyk5y ago

That’s what hacker news is for ;)

Upvoter335y ago

google: search terms -site:cnn.com -site:wikipedia.org -site:...

aiisjustanif5y ago

RIP StumbleUpon.

martin-adams5y ago

I like this question. I’ve often wanted a search engine which gives you the choice to find sites that don’t contain a paywall, tracking or advertising.