undefined | Better HN

0 pointscloudbonsai1y ago0 comments

But how do you explain those results?

- Brian Nosek's team examined 100 studies from high-ranking psychology journals in 2015, and could only reproduce 1/3 of them.

- Tim Errington did the same for cancer papers, and could not reproduce most of them either (he spent 8 years for this efforts btw)

- When you aggregate the reported p-value in scientific publications, it often reveals a "funny" distribution (Leggett 2013, Ookubo 2016)

They are not picking up rare misconducts by low-profile researchers. Fraudant research (from p-hacking to data rigging) is very common and a very serious issue.

0 comments

tptacek1y ago

I don't have much to say about psychology. But Tim Errington himself pushes back on the notion you're trying to sell, that his failure to reproduce research in his own replication projects creates a "yes this research is real" and "no this research isn't" result. Reproduction is hard, effect sizes can be small, reproduction studies can themselves be flawed (that's just how science goes).

The biggest thing though is just this idea that a non-reproducing paper is a failure of science. Journal articles are the beginnings of conversation in a discipline, not the last word on it.

You can see what I mean, though: people who probably couldn't name 3 important researchers in a field see people working on replications in those fields (Nosek, Errington) as celebrities. Because reported failures to replicate are newsworthy, and the day-by-day grind of incremental findings and negative results aren't.

j / k navigate · click thread line to collapse