# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
User-agent: *
Disallow: /
You can see it here: https://search.google.com/test/rich-results/result?id=_mYogl... (click on "View Tested Page")
Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.
so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
https://old.reddit.com/r/redditdev/comments/1doc3pt/updating...
Of course, that became unsustainable so now I have everything behind a login wall.
I will never take a statement given by a company that blatantly lies like this at face value going forward. What a bunch of clowns.
This is a dangerous precedent for the internet. Business conglomerates have been controlling most of the web, but refusing basic interoperability is even worse.
If reddit had exclusive agreement, it would be anti-competive.
This is classic HN anti-Google tirade (and downvoting facts, logic and concepts of free market)
Yes, actually, there is - having $60m to throw around.
"Barriers to entry often cause or aid the existence of monopolies and oligopolies" [0]. Monopolies and oligopolies are definitionally the opposite of free market forces. This is quite literally Econ 101.
How many other sites might have leverage to charge to be indexed?
I don't want to live in a world where you have to use X search engine to get answers from Y site - but this seems like the beginning of that world.
From an efficiency perspective - it's obviously better for websites to just lease their data to search engines then both sides paying tons of bandwidth and compute to get that data onto search engines.
Realistically, there are only 2 search engines now.
This seems very bad for Kagi - but possibly could lead the old, cool, hobbiest & un-monetized web being reinvented?
edit:
> Realistically, there are only 2 search engines now.
https://seirdy.one/posts/2021/03/10/search-engines-with-own-...
From the article:
Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results;
(they could crawl but never ever return those results == more odd).Scraper engine->validation/processing/cleanup->object storage->index + torrent serving is rough pipeline sketch.
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... ("HN Search: annas archive")
[2] https://academictorrents.com/details/9c263fc85366c1ef8f5bb9d... ("AcademicTorrents: Reddit comments/submissions 2005-06 to 2023-12 [2.52TB]")
It's not the beginning, it's mere continuation.
Walled gardens have existed since the AOL days. They deteriorate over time but it doesn't prevent companies from trying (each time, in bigger attempts).
It still exists. It just isn't that popular.
There's an established player with institutional protections, then a scrappy upstart takes a bunch of VC money, converts it into runway, gives away the product for free, gradually replaces and becomes the standard, then puts out an s-1 document saying "we don't make money and we never have, want to invest?" and then they start to enjoy all the institutional protections. Or they don't. Either way you pay yourself handsomely from the runway money so who cares.
The upstart gets indexed and has an API, the established player doesn't.
The upstart is more easily found and modular but the institutional player can refuse to be indexed to own their data and they can block their API to prevent ai slop from getting in and dominating their content.
https://www.ilga.gov/legislation/ilcs/ilcs4.asp?DocName=0720...
But I can understand why they made the change they did. The data was being abused.
My guess is that this was an oversight -- that they will do an audit and reopen it for search engines after those engines agree not to use the data for training, because let's face it, reddit is a for profit business and they have to protect their income streams.
Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it).
If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all.
From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).
Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2].
[1] https://www.bloomberg.com/news/articles/2024-05-09/reddit-la... | https://archive.today/wQuKM
We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.
you can always buy a competitor's or make your own vacuum cleaner if you hate buying at Walmart
maybe what you are really mad about is Reddit monopolising content
Being forced into using google services, because they are paying information companies to deal only with them seems like a disaster for the web.
I'm not sure what to make of that.
It all depends of course what the market is. If one looks as reddit not as a whole but as a collection of niches then one could imho find niches where reddit has a dominant knowledge position.
An enterprise sales team with only 1 customer happens (eg, Mozilla 's search bar), but... That's surprising here, and scary as a sustainable & scalable business. Ignoring 5-6 figure/yr inquiries says a lot to me. In contrast, we did that same-day with Twitter without talking to anyone.
I wonder if this might affect redis, as in slowly kill it's user base especially when it comes to user providing (and often also looking for) high quality content, because who of such users would want to use google search?
I don't understand what you're saying. That's exactly why people append `site:reddit.com` to their searches in the first place, because those search results typically aren't like that.
The veracity of this statement is questionable.
I found at least four web search engines not using Google's index that produced results from the last week.
Example: Recent eruption at Yellowstone Black Diamond Pool
https://www.ecosia.org/search?method=index&q=site:reddit.com...
https://search.brave.com/search?q=reddit.com+black+diamond+p...
https://api.yep.com/fs/2/search?client=web&gl=all&no_correct...
POST /sp/search HTTP/1.0
host: www.startpage.com
content-length: 74
content-type: application/x-www-form-urlencoded
query=site:reddit.com black diamond pool&abp=-1&t=&lui=english&sc=&cat=web
https://old.reddit.com/search/?q=black+diamond+pool
If anyone has some good examples of search queries that I can test showing why a search engine must be used, please share.
Google hasn't been a search engine in a long while, it's just an advertisement engine now.
it started when youtube removed the ability to search for videos older than 5 years, if I had to guess? cost saving, have every old video in cheaper storage... but it sort of fragments youtube, every couple of years you only get newer content.
It’s not like everyone wasn’t already pulling the same grift, but quantity really does have a quality all its own.
Capitalism seems to work ok for the common good until you remove all the protections. LLMs provide a defacto monopoly for the owner which must already be a near monopoly: they take vast resources to train; only a giant corp can afford to buy all the content and provision enough resources to train one.
LLM did not enshittify what's left of the internet, greed did it.
This is a very good point IMO. If we're going to chastise LLM's we may as well give servers, switches, routers, fiber-optic cable, and silicone a bollocking as well since that's ultimately what's facilitating all this.
And to me, forgetting to log in to each of them feels similar, too. For what that's worth. (I hate both of them when not logged in.)
AI companies like Google, Microsoft and OpenAI have deep pockets to 'unprotect' themselves from anything. The barrier to entry is for small AI companies and those aren't really making an impact currently.
This has greatly enhanced my Google experience - easy to ban content farms, AI-generator websites from appearing in Google Images etc.
Kagi shill here. Are they finally applying filters and operands to image searches?
Asking because it was a tough year seeing Pinterest as top filter choice and top result in images (when set as filter=block).
(edit: I just tried searching->image: beautiful quilt patterns. I didn't spot any Pinterest results!)
I have never understood why DDG, etc steadfastly refuse to obey operands in image searches. Most days. Every blue moon operands seem to work. I think.
sidebar: Yesterday I saw Yandex obey quotes in a web search. It was the 1st time I've seen that.
Honestly, that makes those other engines way less valuable because for many topics, telling the engine to specifically narrow the results down to reddit comments is the only way to get a decent answer to what you're looking for. I'd definitely support blocking Quora from everything though.
I don't consider the discussions there to be "real" in any meaningful way, thanks to the extensive moderation.
From what I've seen, there typically ends up being a small handful of moderator-enforced narratives that are deemed "acceptable" for a given subreddit, and any commenters deviating from those narratives get banned, or their comments end up as "[removed]" by "[deleted]", or the comments get obscured with the "comment score below threshold" notice.
It's generally some of the most one-sided and blandest discussion around. Given that there's often no meaningful back-and-forth involving differing perspectives of any sort, I'm not even sure if it should be considered "discussion". It's more like regurgitation and repetition.
I've found the situation to be particularly bad on the Canadian locale-specific subreddits, for example, but a enough of the tech-oriented ones I've seen seem to end up like that, too.
Obviously, people do see value in it, or they wouldn't keep saying so! I would happily exclude Reddit links from search results though.
Not exactly always a reliable source of info outside uncontroversial niche topics or places like /r/AskHistorians that actually moderate. And even there I've seen the occasional humdinger.
Google really should blacklist reddit entirely for this practice, but sadly as bad as reddit is it's still a much higher quality result than average for google.
[0]https://www.reddit.com/r/ChatGPT/comments/133xgb5/gpt2_was_p...
Remember, the only reason Reddit "won" was because Digg destroyed itself with a radical upgrade that everyone hated.
Reddit would have to do something similarly self-inflicted, and I can't even guess where people would go. Reddit was already an alternative to Digg -- what's the alternative to Reddit? I mean, it's certainly not Quora.
The main thing I see Reddit being useful for are discussions about entertainment.
There's probably a subreddit for your favorite sports team, twitch steamer, TV show, book series video game, politics (which is entertainment for some people).
Reddit has seriously degraded the experience of a lot of these communities with things like restricting custom CSS.
It seems to me that the way you'd disrupt Reddit as a startup is to pick a vertical and laser focus on becoming the best discussion board for that community. If it's sports than have integrations for live stats, scores, etc.
In general you could attract users by offering profit sharing on ads the same way Youtube does for creators.
Have the best moderation tools in the world, a constant painpoint with Reddit. Give admins more flexibility over the appearance of the board, all things Reddit took away.
The other path for disruption would be if an established company with those communities tackled the problem. Lots of communities already us Discord, but they tend to also have a subreddit because chat and forums are different communication methods. Discord could easily offer a forum product as an extension of their chat services. If they do it well they'd drive a lot of users away from the subreddits.
>what's the alternative to Reddit? I mean, it's certainly not Quora.
If it was deliberate I certainly can't tell, but one of the characteristics of Reddit is that it caused so many other little tiny internet forums to just wither away. Most were visually unappealing, running some ancient phpbb software or whatever, but there were so many like stars in the night sky. Now, if they're even still running, you look for the newest post, and it will say "November 2023". Hell, the only reason they are still running is that the credit card number on file paying for hosting doesn't expire until next year somehow. Reddit is a red tide algae choking out all life in the ocean, nothing else gets to exist anymore.
This site is essentially 'orange reddit', they just need to add sub-HNs or tagging or something and it'd be ready for an influx of reddit refugees. Not that any of really want it, but it's possible.
You don't want to end up banned from a movies forum because you also participate in a political forum. Federation solves that problem because you can use separate accounts without either forum knowing that you also use the other.
Honestly, it's strange to me how hard people are trying to make distributed anything happen. Federation mostly solves a problem that real people don't have or care about.
I would like to see a wikipedia-style system for Twitter/Reddit: open access data, non-profit.
They could monetize it much better while being less annoying.
Ultimately - Google is getting everything they want from Reddit with this deal without having to buy it outright.
Short of Reddit transforming to an entirely different product (difficult) - I'm not sure where the major growth opportunity is for it.
Does this mean there will be a future where everyone is running their own crawler? I suppose.
As soon as someone shows me a search engine that restores quality of searxh, im getting a subscription for work.
It really cany be hard to whitelist sources and index appropiately.
Get goimg nerds , google has fallen.
I remember seeing an unhelpful hyperlink for the first time. It was a random word in the body of a random tech site that redirected to a list of articles from that site tagged with that term.
I remember being stunned, my expectation was that the link would lead me to another website, one that would be an authoritative source on that term and freely accessible.
20 years later we get a paywalled article about fragmented web – and we’re not slowing down.
https://cc.bingj.com/cache.aspx?d=5070227914243&w=ljIRk8yx42...
For example, when I search for product reviews, I always specify reddit. Otherwise the search results are inundated with SEO spam.
Reddit's justification for this is profoundly wrong. Their "public content policy" is absurd doublespeak, and counter to everything the open internet is and hopes to be. You cannot simultaneously call yourself "open" and "public" while refusing access to automated clients. Every client is automated. They even go so far as to say that "crawling" (also known as "downloading") is an "abuse" and violates user privacy.
This is absurd, and not justified. I would love to see legislation that restricted server operators' ability to prohibit automated access in this way, but I suppose it will never happen. Some people in this thread have attempted to justify the policy by saying "they have to protect their income streams". No they don't. You don't have a right to an income stream, and you certainly don't have a right to lie in order to get all the benefits of an open internet with none of the downsides. Noting of course that the "downsides" are in this case actually just "competitors".
Or are you somehow suggesting that it’s google’s fault that Reddit took this step? I don’t see any indication that’s the case.
google is using their power to prevent others from competing.
the problem here is of course that if reddit would be in financial trouble (i don't know if they are but let's imagine they need this money), they'd be between a rock and a hard place.
google should not be allowed to make exclusive deals, and reddit could not survive without the deal, then what would be left? google buys reddit, or the relevant authority approves of the deal?
i thought about the same problem with firefox. let's assume firefox is forced to allow people to make a choice of the default search engine (just like microsoft was forced to allow a choice of default browser on windows) then google might stop paying mozilla, and they could end up in financial trouble.
ideally no company ever depends on a single other company that much, but that only works if we don't allow companies to grow that much in the first place.
https://help.kagi.com/kagi/search-details/search-sources.htm...
>Our search results also include anonymized API calls to all major search result providers worldwide
Incorrect: https://www.mojeek.com/about/why-mojeek
> Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google.
Also things like the API fiasco, and also small annoyances like the fact that when you click on an image on reddit, it now goes to a wrapper html page instead of just the actual image (this was one of the reasons reddit was better than most social media...).
Part of the blame for the redirect-to-wrapper page lies with browsers. If browsers didn't let servers reliably differentiated between a direct request and an <img> embed then this practice would not be as widespread.
The most recent 10-K financial results 2024-03-31 (filed 2024-05-08) shows they actually lost money: https://www.sec.gov/edgar/browse/?CIK=1713445
(For 2024-Q1, Reddit lost -$575 million on revenue of $242 M.)
If the quoted "$60 million deal"[1] from Feb 2024 is accurate, that small amount from Google may not be enough for Reddit to turn a profit. It remains to be seen what the Q2 or Q3 financials will show.
Then again most of what that site does is just blend and regurgitate the information that's currently on it anyway.
For a while, the internet had an end-run play that made that guy less useful. You can just go on the internet for obscure movie information, buddy.
But now it seems like knowing a movie guy is going to be the only way to get a real person's opinion on movies. The internet is about to forget everything without a profit motive and just start telling you that the latest product from a monolith corp like disney is the only movie worth watching. If someone scrapes all the useful movie opinions off of reddit and spends their time crafting it into a usable format, that guy's probably got a company. But not Bill. Bill's just a guy you can know or not know. You can't monetize knowing Bill. Sidenote that's probably why it irked me so bad when some bozo coined the phrase "social capital".
Startpage, Kagi and Lukol are 3 that source from Google. I imagine there are others.
Blame Reddit, not Google.
When Apple strikes an exclusive deal with suppliers for parts, it is sound business practice.
When Google strikes an exclusive deal with Reddit, it is ..
Some of you have no idea how businesses work, and it shows.
It's because reddit is selling content created by users, base on promises that reddit supports open internet, open data, etc, without their consent and sharing revenue, which maybe legal but likely not ethical.
The users hold the copyright (reddit claim that they made the meme) but reddit has the non-exclusive right to redistribute and license the content.
Two different things.