There are websites that try to do that (e.g. scirate) but the problem is that people doesn't participate much. The problem is not the website, but convincing academics to comment and vote (academics are typically starved of time, and reading a paper and writing a good coment is not easy...)

It would be nice if someone solved the problem and managed to create a working one, though.

What about collecting tweets from verified researchers? It could help getting to critical mass. You could even consider papers tweeted by researchers that are followed by your known researchers, and so on, with the right weighting (something resembling pagerank).

With the right weighting this could really boost the size and quality of your dataset.

Can I have a beta when it's ready? I've been thinking about this for a while and I like to think that I could provide useful input.

Also very much something is be interested in, if invites are available, thanks.