Ask HN: Fastest Crawl of HN Articles

8 pointsagencies3y ago13 comments

HN links to over 6 million urls in stories and comments. Many domains have expired or content is no longer available. Internet archive has much of the content but throttles requests. What's the fastest way to get the historical content?

13 comments

arinlen3y ago

HN does have a REST API which is quite easy to use.

https://github.com/HackerNews/API

I'm not sure what rate limiting policy is in place, but in theory you can start with a request for maxitem and from that point on just GET all items down to zero until you hit some sort of blocker.

krapp3y ago

HN's API doesn't have rate limiting. The Algolia HN API[0] however does.

[0]https://hn.algolia.com/api

agenciesOP3y ago

To clarify I'm not asking about HN itself but articles linked from HN.

As you said the HN api is great and there are at least 2 existing published crawls of it that help a lot.

arinlen3y ago

> To clarify I'm not asking about HN itself but articles linked from HN.

I might not have a clear picture of what you're looking for, but items of type "story" returned by the HN API do have a URL field, which I believe correspond to submitted links.

You can scrape the text field of comment items, but that takes a bit more work.

1 more reply

krapp3y ago

The fastest way to get that would probably still be through HN's API, you just have to take the URL field for stories and ignore everything else.

2 more replies

jpcapdevila3y ago

The best way to do it is from Google BigQuery.

There's a dataset containing everything: bigquery-public-data.hacker_news.full

You can write SQL and is super fast. Sample:

SELECT * FROM bigquery-public-data.hacker_news.full LIMIT 1

python2733y ago

maybe https://commoncrawl.org/

j / k navigate · click thread line to collapse

13 comments

arinlen3y ago

HN does have a REST API which is quite easy to use.

https://github.com/HackerNews/API

I'm not sure what rate limiting policy is in place, but in theory you can start with a request for maxitem and from that point on just GET all items down to zero until you hit some sort of blocker.

krapp3y ago

HN's API doesn't have rate limiting. The Algolia HN API[0] however does.

[0]https://hn.algolia.com/api

agenciesOP3y ago

To clarify I'm not asking about HN itself but articles linked from HN.

As you said the HN api is great and there are at least 2 existing published crawls of it that help a lot.

arinlen3y ago

> To clarify I'm not asking about HN itself but articles linked from HN.

I might not have a clear picture of what you're looking for, but items of type "story" returned by the HN API do have a URL field, which I believe correspond to submitted links.

You can scrape the text field of comment items, but that takes a bit more work.

1 more reply

krapp3y ago

The fastest way to get that would probably still be through HN's API, you just have to take the URL field for stories and ignore everything else.

2 more replies

jpcapdevila3y ago

The best way to do it is from Google BigQuery.

There's a dataset containing everything: bigquery-public-data.hacker_news.full

You can write SQL and is super fast. Sample:

SELECT * FROM bigquery-public-data.hacker_news.full LIMIT 1

python2733y ago

maybe https://commoncrawl.org/

j / k navigate · click thread line to collapse