Show HN: Top PDFs and Papers Submitted to Hacker News (opens in new tab)

(hackernewspapers.com)

250 pointscloudkj6y ago21 comments

21 comments

Hmm. Reverse-engineering this page with BigQuery is surfacing a lot more results than the page itself (for 2019 atleast).

https://docs.google.com/spreadsheets/d/1he6ca0BBYbj2ZEOEpu8L...

    #standardSQL
    SELECT id, title, url, score
    FROM `bigquery-public-data.hacker_news.full`
    WHERE timestamp > '2019-01-01'
    AND REGEXP_CONTAINS(url, '.pdf|arxiv.org')
    ORDER BY score desc

seedie6y ago

Thanks to your post I just learned about the big query public datasets https://www.reddit.com/r/bigquery/wiki/datasets

sytelus6y ago

One of the huge issue on HN is lack of good algorithms for ranking. Currently all posts suffer from insufficient initial exposure to receive enough votes and thus false negatives are extremely high on HN. Just go to new page right now, take a look at most recent posting and you will find that majority of good genuine posts would never make it beyond their initial single upvote. Very small subset gets re-posted few times and only randomly some will get attention. Unfortunately papers are probably the biggest victim so you could have as well have collected all the links, did kind-of-random sort and you might still get fairly decent results. The bottom line is that never rely on HN's flawed ranking. Use that as weak signal, combine with Reddit, citations, tweet counts etc if you want to work around vast sea of false negatives that HN currently is.

gmiller1234566y ago

I think this is what most social news sites get wrong about their points systems. Someone who submits an article gets points for each vote it gets. But the few heroes that waded through the submissions to initially upvote them get nothing. Even though they likely did a lot more work than the submitter.

I'd think a better algorithm would be that if you upvote an article in the "new" section that eventually makes it to the main page, you'd get at least as many points as the submitter.

smitop6y ago

That incentivize upvoting everything, since there is no punishment for upvoting posts that don't make it to the frontpage. That could be solved by making votes a limited resource, but that approach has many downsides.

ignoramous6y ago

Back when pg used to run the community [0], he'd deeply think about problems [1] and then later solve them [2] but not without controversy [3]. news.yc was better for it with such a strong personality at the helm who'd withstand the backlash and hold on to their convictions. I guess, we need someone of that caliber, someone to step up to usher in to the post-pg era [4], as it were.

What I do to find my share of interesting links is to search on https://hn.algolia.com with different keywords and sift through those discussions, follow links in the comments, follow the commentators previous submissions and posts, follow other submissions from the website, gather new keywords and search again... it is kind of a rabbit hole once you start. Though you'd quickly begin to realise that it is great folks continue to submit links that don't end up hitting the front-page, as the aggregation of links is valuable even though the discussion is missing [5].

That said, I have stumbled upon a lot of interesting things aren't simply shared at all on news.yc, though, I guess this is by design. I just hope folks don't stop sharing links just because their previous submissions didn't generate discussion or upvotes on news.yc.

[0] https://news.ycombinator.com/item?id=7494708

[1] https://news.ycombinator.com/item?id=2403696

[2] http://paulgraham.com/hackernews.html

[3] https://news.ycombinator.com/item?id=3122233

[4] https://hn.algolia.com/?query=%22Ask%20PG:%22&sort=byPopular...

[5] Some would argue that few news.yc comments are worth their weight in upvotes anyway: https://danluu.com/hn-comments/

richardw6y ago

Team is doing a fine job. It's much harder than the early days. Many want to game the system so it's a SEO vs. anti-spam job to keep this place running smoothly.

Getting to the front page isn't predictable, it's like creating a hit song. Tons of great music gets unnoticed. There's bias and well known names have an advantage.

burtonator6y ago

We did one for Polar at the end of 2018:

https://getpolarized.io/2019/01/08/top-pdfs-of-2018-hackerne...

If you want an awesome PDF reader to read these you should check out Polar :)

Yes. There's a Linux version! ;)

https://getpolarized.io/

Yetanfou6y ago

Could you add links to the relevant threads on HN to this listing e.g. by linking the HN score there? This would give access to both the PDF for those who just want to read the thing as well as to the discussion for those who wonder what made it reach such dazzling heights on HN.

cloudkjOP6y ago

Thanks for the suggestion. I went ahead and made the change to link the score to the original discussion. It will take a few moments for the CDNs to propagate the changes.

One caveat is that some stories have multiple submissions; I just linked to the one with the highest score for now, but will need to iterate a bit to better handle multiple submissions.

kbenson6y ago

A good suggestion. For those that want to look up the conversion before that suggestion is (hopefully) implemented, you can use the search of HN to search by submission URLs.

danielecook6y ago

I put together something similar but for articles people are linking to on Arxiv here:

Bolderman6y ago

Seems like it hasn't been updated for a few weeks. But it's a cool site!

ZoomStop6y ago

A link at the top for 2019 would be helpful, maybe change text color for year being displayed? When navigating years returning to 2019 isn't intuitive.

cloudkjOP6y ago

Thanks for the suggestion. I added a link for 2019, and the style for the active year is now bolded. I didn't want to deviate too much from the default color scheme of the theme I'm using, but that may change in the future.

epicwhaleburger6y ago

Related: https://fermatslibrary.com/

itdog6y ago

It will be convenient if there's a search box on the page

baby6y ago

Now if we could categorize this so that I can get the top crypto :)

karmakaze6y ago

It would be great if these could be tagged into a few categories.

cloudkjOP6y ago

That sounds like a fun and useful follow up. My initial thoughts would be to do some unsupervised topic modeling with LDA. Happy to consider other suggestions.

mathnmusic6y ago

Why not add these research papers in learn-awesome which is building a topic graph? https://github.com/learn-awesome/learn-awesome

j / k navigate · click thread line to collapse

21 comments

minimaxir6y ago

Hmm. Reverse-engineering this page with BigQuery is surfacing a lot more results than the page itself (for 2019 atleast).

https://docs.google.com/spreadsheets/d/1he6ca0BBYbj2ZEOEpu8L...

    #standardSQL
    SELECT id, title, url, score
    FROM `bigquery-public-data.hacker_news.full`
    WHERE timestamp > '2019-01-01'
    AND REGEXP_CONTAINS(url, '.pdf|arxiv.org')
    ORDER BY score desc

seedie6y ago

Thanks to your post I just learned about the big query public datasets https://www.reddit.com/r/bigquery/wiki/datasets

sytelus6y ago

gmiller1234566y ago

I'd think a better algorithm would be that if you upvote an article in the "new" section that eventually makes it to the main page, you'd get at least as many points as the submitter.

smitop6y ago

ignoramous6y ago

[0] https://news.ycombinator.com/item?id=7494708

[1] https://news.ycombinator.com/item?id=2403696

[2] http://paulgraham.com/hackernews.html

[3] https://news.ycombinator.com/item?id=3122233

[4] https://hn.algolia.com/?query=%22Ask%20PG:%22&sort=byPopular...

[5] Some would argue that few news.yc comments are worth their weight in upvotes anyway: https://danluu.com/hn-comments/

richardw6y ago

Team is doing a fine job. It's much harder than the early days. Many want to game the system so it's a SEO vs. anti-spam job to keep this place running smoothly.

Getting to the front page isn't predictable, it's like creating a hit song. Tons of great music gets unnoticed. There's bias and well known names have an advantage.

burtonator6y ago

We did one for Polar at the end of 2018:

https://getpolarized.io/2019/01/08/top-pdfs-of-2018-hackerne...

If you want an awesome PDF reader to read these you should check out Polar :)

Yes. There's a Linux version! ;)

https://getpolarized.io/

Yetanfou6y ago

cloudkjOP6y ago

Thanks for the suggestion. I went ahead and made the change to link the score to the original discussion. It will take a few moments for the CDNs to propagate the changes.

One caveat is that some stories have multiple submissions; I just linked to the one with the highest score for now, but will need to iterate a bit to better handle multiple submissions.

kbenson6y ago

A good suggestion. For those that want to look up the conversion before that suggestion is (hopefully) implemented, you can use the search of HN to search by submission URLs.

danielecook6y ago

I put together something similar but for articles people are linking to on Arxiv here:

Bolderman6y ago

Seems like it hasn't been updated for a few weeks. But it's a cool site!

ZoomStop6y ago

A link at the top for 2019 would be helpful, maybe change text color for year being displayed? When navigating years returning to 2019 isn't intuitive.

cloudkjOP6y ago

epicwhaleburger6y ago

Related: https://fermatslibrary.com/

itdog6y ago

It will be convenient if there's a search box on the page

baby6y ago

Now if we could categorize this so that I can get the top crypto :)

karmakaze6y ago

It would be great if these could be tagged into a few categories.

cloudkjOP6y ago

That sounds like a fun and useful follow up. My initial thoughts would be to do some unsupervised topic modeling with LDA. Happy to consider other suggestions.

mathnmusic6y ago

Why not add these research papers in learn-awesome which is building a topic graph? https://github.com/learn-awesome/learn-awesome

j / k navigate · click thread line to collapse