How Should We Critique Research? (2019) (opens in new tab)

Its too bad research papers can't be organized like a git history. We'd see many forks that never end up as pull requests that are merged back to main. And probably forks of forks that stray too far from the founding paper's intent. It would be nice to more easily identify original versus derivative research. Maybe that solves a different problem. I like their suggestion though:

"I offer a prag­matic cri­te­rion: what makes a crit­i­cism im­por­tant is how much it could change a re­sult if cor­rected and how much that would then change our de­ci­sions or ac­tions: to what ex­tent it is a “dif­fer­ence which makes a dif­fer­ence”. This is why is­sues of re­search fraud, causal in­fer­ence, or bi­ases yield­ing over­es­ti­mates are uni­ver­sally im­por­tant: be­cause a ‘causal’ ef­fect turn­ing out to be zero ef­fect or grossly over­es­ti­mated will change al­most all de­ci­sions based on such re­search; while on the other hand, other is­sues like mea­sure­ment error or dis­tri­b­u­tional as­sump­tions, which are equally com­mon, are often not im­por­tant: be­cause they typ­i­cally yield much smaller changes in con­clu­sions, and hence de­ci­sions."

So, 2 papers, both with data and claims.

The first is critiqued on its claim because the data, while correct with quality methodology, doesn't support the extent made in the claim. This critique is more meaningful because it changes the outcome of the paper and any decisions following its publication.

The second's claim is within the bounds of the data but there is a discrepancy in the data collection which is the source its its critique. Fixing that doesn't change the claim but may indicate more research is needed. This critique could change decisions made from publishing, but if the claim is still within the reason of the data, then likely not.

I had to think through that and I think I like it.

Related:

How Should We Critique Research? - https://news.ycombinator.com/item?id=26834499 - April 2021 (51 comments)

How should we critique research? - https://news.ycombinator.com/item?id=19981774 - May 2019 (20 comments)

It seems to me that the most important factor is not being mentioned here. That is money - who funds the research.

Its a simple enough problem - eg if you wanted papers to show anything - eg 'that koalas cause forest fires in Australia' (I know that's ridiculous!) - then you simply fund a bunch of papers. If you have 10 papers, and 2 are supportive, 2 are against and the remaining are ambiguous - you have a start! You take the supportive ones, and fund similar studies. Soon you have a lot of data that seems to say something in support of the thesis you like, but this is nothing to do with uncovering some underlying principle.

If you have a big enough pocket, you get the science you pay for.

https://www.threads.net/@tmurrayhimself/post/C56uXtKM0y0

"studies show that all studies can be traced back to the guy with the most money"

With data quality standards, research ethics, and random audits.

https://mchankins.wordpress.com/2013/04/21/still-not-signifi...

The primary issue is circular references in citation, and non-cascaded retractions for known errata.

Typically thesis work tends to be reproducible most of the time, but around 12% to 17% of the hundreds of papers I read every month were nongeneralizable or worse outright BS.

It can be really disillusioning for many students...

Now I eat cheese goldfish crackers, and no longer care either way. Have a wonderful day =3

I am reviewing papers for my substack regularly and there is a structure I believe works well that focuses on potential impact, structure, reading comprehension, github availability (where applicable), and problem relevance.