undefined | Better HN

0 pointsmccorrinall3y ago0 comments

If they don’t want to be spidered maybe just use robots.txt

They can’t have both: exposure AND content only available on their platform

0 comments

Can you expand on that? Should Google have rights to use any of your content if you allow spidering? Including for commercial gain?

Yes and yes. It’s okay to crawl a site and show the meta tags like short description, title, author etc. Why should it be illegal?

j / k navigate · click thread line to collapse

0 pointsmccorrinall3y ago0 comments

If they don’t want to be spidered maybe just use robots.txt

They can’t have both: exposure AND content only available on their platform

Can you expand on that? Should Google have rights to use any of your content if you allow spidering? Including for commercial gain?

Yes and yes. It’s okay to crawl a site and show the meta tags like short description, title, author etc. Why should it be illegal?

j / k navigate · click thread line to collapse